Repository navigation
fix(minimald): keep exec output relays live while a stdin write is blocked - #2148
Merged
norrietaylor merged 3 commits intoOct 10, 2026
Merged
Conversation
…ocked bridge_one wrote each stdin chunk to the child inside the read branch's handler, parked on a nested select that raced only client loss. While that write was blocked on a full child stdin pipe, the outer select never polled the child's stdout or stderr, so a child that produced output before consuming stdin (for example a large cat) deadlocked the bridge: the session stalled until the client's keepalive timed out. Queue the chunk instead, and drive the child-stdin write from a dedicated branch of the outer select. The branch uses the cancel-safe write and resumes from a saved offset, so the loop keeps polling stdout, stderr, wait, and client loss while a write is parked, and no bytes are lost when another branch wins the race. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
…n queued Add a real-process regression test that pipes 4 MiB through cat in both directions at once, and a mock test that a child exiting with a stdin chunk still queued reports its own status. Seed child_stdin_open from whether the step has a child stdin at all.
…ectly select! evaluates every precondition before it builds any branch future, so the gates can read child_stdin and pending themselves; the mirror flags only had to be kept in step by hand. Note at the loop exit that stdin still queued when the loop ends is dropped, and make the queued- stdin exit test keep every pipe end open so the wait branch is the only way out, asserting the child saw exactly its pipe's worth of stdin.
Member
|
/review |
Contributor
norrietaylor
approved these changes
Oct 10, 2026
norrietaylor
deleted the
inbox-patch/exec-bridge-stdin-parking-a26b2d69531fe2bc
branch
October 10, 2026 02:38
bryan-minimal
added a commit
that referenced
this pull request
Oct 13, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains: the channel is read as data arrives, a writer of its own feeds the command from what is held, and the held amount is capped at 256 MiB, past which the exec ends with a notice on its stderr rather than hanging (an active reject). A command that closed its stdin or exited is not charged for input it never reads. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
added a commit
that referenced
this pull request
Oct 13, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains: the channel is read as data arrives, a writer of its own feeds the command from what is held, and the held amount is bounded twice, 256 MiB per exec and 512 MiB across the daemon's execs, past which the exec that would cross the bound ends with a notice on its stderr rather than hanging (an active reject). A command that closed its stdin or exited is not charged for input it never reads. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
added a commit
that referenced
this pull request
Oct 13, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains: the channel is read as data arrives, a writer of its own feeds the command from what is held, and the held amount is bounded twice, 256 MiB per exec and 512 MiB across the daemon's execs, past which the exec that would cross the bound ends with a notice on its stderr rather than hanging (an active reject). A command that closed its stdin or exited is not charged for input it never reads. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
added a commit
that referenced
this pull request
Oct 14, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains. The pump takes each packet off the channel into a queue the bridge reads as the command's stdin; a byte is charged to the exec's and the daemon's held counts from push to read, as a plain Vec (not russh's power-of-two locked buffer). The counts are bounded, 256 MiB per exec and 512 MiB across the daemon's execs: past either, the queue is cleared and released at once and the exec ends under its own reason (not as a lost client), with a notice on its stderr naming the bound. For a single program, input arriving after the command closed its stdin is read and dropped, never held against it; a sequence keeps it for the next command. At a bound the pump stops taking from the channel instead of ending the exec at once: before the command starts that is the wait for it (a fast client puts the cap in flight before a command spawns on a loaded host), and once it runs a bound with no byte read for 10 s is the stall that ends it. The connection is parked for at most that long per bound; a reading command is never ended. The nsenter shim inherited the exec's stdin pipe as its own fd 0 and held it while waiting on the command, so a command that closed its stdin never closed the pipe: the bridge saw no EPIPE and took it for a command that had stopped reading. The shim never reads fd 0; after the spawn it points fd 0 at /dev/null. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
added a commit
that referenced
this pull request
Oct 14, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains. The pump takes each packet off the channel into a queue the bridge reads as the command's stdin; a byte is charged to the exec's and the daemon's held counts from push to read, as a plain Vec (not russh's power-of-two locked buffer). The counts are bounded, 256 MiB per exec and 512 MiB across the daemon's execs: past either, the queue is cleared and released at once and the exec ends under its own reason (not as a lost client), with a notice on its stderr naming the bound. For a single program, input arriving after the command closed its stdin is read and dropped, never held against it; a sequence keeps it for the next command. At a bound the pump stops taking from the channel instead of ending the exec at once: before the command starts that is the wait for it (a fast client puts the cap in flight before a command spawns on a loaded host), and once it runs a bound with no byte read for 10 s is the stall that ends it. The connection is parked for at most that long per bound; a reading command is never ended. The nsenter shim inherited the exec's stdin pipe as its own fd 0 and held it while waiting on the command, so a command that closed its stdin never closed the pipe: the bridge saw no EPIPE and took it for a command that had stopped reading. The shim never reads fd 0; after the spawn it points fd 0 at /dev/null. What the daemon holds is never overstated: a push that races the reader's drop is dropped uncharged (clear discards), the pump is awaited after its abort, and an exec releases what it still counts when it is dropped; a client that goes releases its held input at once. At the daemon bound every exec's releases wake the waiting pumps, and an exec holding nothing of its own waits for the others rather than being ended for their hoarding. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
added a commit
that referenced
this pull request
Oct 15, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains. The pump takes each packet off the channel into a queue the bridge reads as the command's stdin; a byte is charged to the exec's and the daemon's held counts from push to read, as a plain Vec (not russh's power-of-two locked buffer). The counts are bounded, 256 MiB per exec and 512 MiB across the daemon's execs: past either, the queue is cleared and released at once and the exec ends under its own reason (not as a lost client), with a notice on its stderr naming the bound. For a single program, input arriving after the command closed its stdin is read and dropped, never held against it; a sequence keeps it for the next command. At a bound the pump stops taking from the channel instead of ending the exec at once: before the command starts that is the wait for it (a fast client puts the cap in flight before a command spawns on a loaded host), and once it runs a bound with no byte read for 10 s is the stall that ends it. The connection is parked for at most that long per bound; a reading command is never ended. The nsenter shim inherited the exec's stdin pipe as its own fd 0 and held it while waiting on the command, so a command that closed its stdin never closed the pipe: the bridge saw no EPIPE and took it for a command that had stopped reading. The shim never reads fd 0; after the spawn it points fd 0 at /dev/null. What the daemon holds is never overstated: a push that races the reader's drop is dropped uncharged (clear discards), the pump is awaited after its abort, and an exec releases what it still counts when it is dropped; a client that goes releases its held input at once. At the daemon bound every exec's releases wake the waiting pumps, and an exec holding nothing of its own waits for the others rather than being ended for their hoarding. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bryan-minimal
added a commit
that referenced
this pull request
Oct 17, 2026
…and cannot park the connection The second half of gominimal/inbox#866: with #2148 in, an exec that streams enough data both ways still hangs, and the whole connection with it. minimald's input pump wrote each CHANNEL_DATA into the command's stdin pipe before reading the next; a command blocked on its own stdout left the pump waiting, russh's session loop parked on that channel's bounded queue, and a parked loop delivers nothing to any channel, including the command's output, whose draining is what would let it read its stdin again. russh grants the client's send window on receipt and gives the server no way to hold it back (Eugeny/russh#730 was declined upstream: "the application must always drain"), so the daemon drains. The pump takes each packet off the channel into a queue the bridge reads as the command's stdin; a byte is charged to the exec's and the daemon's held counts from push to read, as a plain Vec (not russh's power-of-two locked buffer). The counts are bounded, 256 MiB per exec and 512 MiB across the daemon's execs: past either, the queue is cleared and released at once and the exec ends under its own reason (not as a lost client), with a notice on its stderr naming the bound. For a single program, input arriving after the command closed its stdin is read and dropped, never held against it; a sequence keeps it for the next command. At a bound the pump stops taking from the channel instead of ending the exec at once: before the command starts that is the wait for it (a fast client puts the cap in flight before a command spawns on a loaded host), and once it runs a bound with no byte read for 10 s is the stall that ends it. The connection is parked for at most that long per bound; a reading command is never ended. The nsenter shim inherited the exec's stdin pipe as its own fd 0 and held it while waiting on the command, so a command that closed its stdin never closed the pipe: the bridge saw no EPIPE and took it for a command that had stopped reading. The shim never reads fd 0; after the spawn it points fd 0 at /dev/null. What the daemon holds is never overstated: a push that races the reader's drop is dropped uncharged (clear discards), the pump is awaited after its abort, and an exec releases what it still counts when it is dropped; a client that goes releases its held input at once. At the daemon bound every exec's releases wake the waiting pumps, and an exec holding nothing of its own waits for the others rather than being ended for their hoarding. Refs: gominimal/inbox#866, Eugeny/russh#730 Fixes: gominimal/inbox#1089 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABU46TDw
In the exec bridge, each stdin chunk was written to the child inside the
read branch's handler, parked on a nested select that raced only client
loss. While that write was blocked on a full child stdin pipe, the outer
select never polled the child's stdout or stderr, so a child that
produced output before consuming its stdin (a large
cat, say)deadlocked the bridge: the session stalled until the client's keepalive
timed out.
Queue the chunk instead, and drive the child-stdin write from its own
branch of the outer select. That branch uses the cancel-safe
writeandresumes from a saved offset, so the output relays,
wait, and clientloss keep being polled while a write is parked, and no bytes are lost
when another branch wins the race.
Verification
cargo fmt --all --check --manifest-path target/Cargo.toml— cleancargo clippy --workspace --all-targets --locked --manifest-path target/Cargo.toml -- -D warnings— no warningscargo test --locked --manifest-path target/Cargo.toml -p minimald— 915 passed; the 11 failures (net::switch, connection, rpc) are pre-existing, environment-dependent, and fail identically on pristine mainbridge_relays_output_while_a_stdin_write_is_blockedproves child output still flows while a stdin write is blocked, and times out on the old code; all 39 exec tests passbridge_relays_output_while_a_stdin_write_is_blocked(256 KiB each way through mock pipes),bridge_streams_megabytes_through_cat(4 MiB through a realcatchild on the Linux CI lanes),and
bridge_reports_exit_when_child_exits_with_stdin_queued(child reaped with a chunk queued:exit status kept, queued chunk dropped, no kill).
Summary by CodeRabbit