61 critical pipe payload deadlock

Part VII: Ecosystem, Runtime & Agent-Specific | Challenge §61

61. Bidirectional Pipe Payload Deadlock

Source: Antigravity 01_io_and_formatting.md (RA)

Severity: Critical | Frequency: Situational | Detectability: Hard | Token Spend: High | Time: Critical | Context: Low

The Problem

UNIX pipes have a finite kernel buffer (typically 64KB on Linux). If a CLI tool simultaneously reads a large payload from stdin AND writes a large response to stdout, both sides can block waiting for the other to drain — a classic deadlock. The tool waits for the agent to read its stdout output so it can write more; the agent waits for the tool to finish reading stdin so it can process the response. Neither side unblocks; the process hangs forever.

# Agent sends 200KB JSON payload to tool and expects 200KB response:
$ echo "$large_json" | my-tool transform > result.json
# Timeline:
# 1. Agent starts writing 200KB to tool's stdin
# 2. Tool starts reading stdin, starts writing response to stdout
# 3. Tool's stdout fills the 64KB pipe buffer — tool blocks waiting for reader
# 4. Agent is still writing stdin — hasn't started reading stdout yet
# 5. Tool is blocked on stdout write; agent is blocked on stdin write
# → DEADLOCK. Both processes hang forever.

This manifests silently: both processes are "running" (non-zero CPU is possible), no timeout detection by either side, no error message.

Impact

• Full timeout budget burned with no useful output
• Process requires external kill signal; any partial output is lost
• No error message produced by either side
• Harder to detect than §50 (stdin deadlock): both sides are actively writing, not just blocking on read

Solutions

Use temporary files for large payloads instead of pipes:

# Avoid: pipe large data
echo "$large_json" | my-tool transform

# Good: use file reference
echo "$large_json" > /tmp/input.json
my-tool transform --input-file /tmp/input.json > result.json

Schema declares maximum stdin payload size:

{
  "stdin_input": {
    "max_bytes": 65536,
    "overflow_flag": "--input-file",
    "overflow_hint": "For payloads >64KB, use --input-file <path> instead of stdin"
  }
}

Framework enforces size limit on stdin reads:

# Framework reads stdin with size limit:
data = sys.stdin.buffer.read(MAX_STDIN_BYTES)
if len(data) >= MAX_STDIN_BYTES:
    exit_with_error("STDIN_TOO_LARGE", "Payload exceeds 64KB. Use --input-file instead.")

For framework design: - Framework MUST enforce a maximum stdin payload size (default: 64KB) and fail with exit 2 if exceeded, directing the caller to use --input-file instead - The --input-file flag MUST be auto-generated by the framework for any command that accepts stdin input - Framework MUST document the pipe buffer limit prominently in the agent integration guide

Evaluation

Score	Condition
0	Bidirectional pipe with large payloads deadlocks silently; no --input-file alternative; no stdin size limit
1	--input-file flag available on some commands; no stdin size enforcement; overflow not detected
2	Framework enforces stdin size limit; STDIN_TOO_LARGE error (exit 2) with --input-file hint when exceeded
3	stdin_input.max_bytes declared in schema; --input-file auto-generated for all stdin-accepting commands; limit documented

Check: Pipe a payload larger than 64KB to any command that also produces large output — verify the tool exits with STDIN_TOO_LARGE rather than deadlocking.

---

Agent Workaround

Never use bidirectional pipes with large payloads; always use --input-file for payloads over the safe threshold:

import subprocess, json, tempfile, os

PIPE_SAFE_BYTES = 32 * 1024  # conservative: 32KB, well under 64KB pipe buffer

def run_with_payload(
    cmd: list[str],
    payload: dict | str,
) -> dict:
    payload_str = json.dumps(payload) if isinstance(payload, dict) else payload
    payload_bytes = payload_str.encode()

    if len(payload_bytes) > PIPE_SAFE_BYTES:
        # Payload too large for safe piping — use a temp file
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".json", delete=False
        ) as f:
            f.write(payload_str)
            tmp_path = f.name

        try:
            result = subprocess.run(
                [*cmd, "--input-file", tmp_path],
                capture_output=True, text=True,
                stdin=subprocess.DEVNULL,
            )
        finally:
            os.unlink(tmp_path)
    else:
        # Small payload: safe to use stdin pipe
        result = subprocess.run(
            cmd,
            input=payload_str,
            capture_output=True, text=True,
        )

    return json.loads(result.stdout)

Limitation: If the tool has no --input-file flag and requires stdin for large payloads, the only safe option is to split the payload into chunks below the pipe buffer size — this is only possible for array-type payloads; for single large objects there is no workaround other than asking the tool author to add --input-file support