Repository navigation
fix(common,minimald): fail a build fast when a source cannot connect (0.7 backport) - #3181
Conversation
A package build for an own-address box attaches to the box's network after its process starts. A short build step, such as claude-code's chmod and mv, can end before the attach finishes. Its namespace goes with it, the attach fails on `nsenter`, and the launch killed a process that had already ended and failed a build that had worked. An attach that fails after its process ended now leaves that process's exit status as the result. The plan's release still runs. A failed attach of a process that still runs stays a failure. Port to the 0.7 line: the change lands in sandbox2, the line's name for the sandbox crate. main's other call site, the run of an inherit_cwd task in minimald's env.rs, does not exist on the line, so minimald is unchanged here.
On a VM host a source download that leaves as the daemon's own traffic is dropped with no answer, so an activation hung until its RPC timed out at 240 s. See [#3058](#3058). A download now fails within 15 s. A web download bounds its connect across all of a host's addresses, on the placed relay too. The Google Cloud Storage client has no connect bound and retries for minutes, so a `gs://` source bounds the wait for its first answer. That bound ends with the first answer, so a long download is not limited. It also means a `gs://` read gives up retrying a failing first answer after 15 s, not 300 s. The failure names the host, and the build reports the package, the host and the reason. The reason is what the download saw, such as `no answer after 15s`, because no answer can have several causes and the download cannot tell them apart. Port to the 0.7 line: main's box_ops.rs is session_sop.rs here, and main's BoxChannel is SessionChannel in env.rs.
A build the egress rules refused named the package, the host and the rules,
and stopped there. The person who read it had no command to run. It now adds:
To allow it, activate a new session with --network own_ip
--allow-dns-hosts <host>
The step names `--network own_ip` because on the default network the flag
alone is refused at create. A destination's port is dropped from the flag's
value, because a host allowance covers every port of the addresses it pins.
The step appears only on a connection the build opened itself. It is left
out for a literal address, which that flag does not take. It is left out for
a web source too: an own-address box has no classifier leaf, so its build
fetches the source as the daemon's own traffic, and in a VM no rule of the
box lets that out. It names no `minimal.toml` key, because none reaches a box
on this line.
Refs [#3058](#3058)
Port to the 0.7 line: main's boxes crate is sessions here, so
`is_valid_dns_host` is made public there. main's box_actor/tests.rs is
minimald's session/tests.rs.
A step that ended during its network attach left the plan's release to `Drop`, which only spawns it. The next step's plan could run first and replace the reservation, so the release gave back the new one and the old one stayed held. The attach now finishes the release before it returns. `Drop` still covers a cancelled launch.
…resses The relay split its 15 s bound equally over a host's addresses, after the lookup. An address that refused at once wasted its share, so a later address that needed longer could fail with time left. The lookup was outside the bound, so name servers that never answer could exceed it. One deadline now covers the lookup and every address. Each address gets the time left, divided by the addresses left.
The requirement now says where the step appears, that it is for a new session, and when it is left out. The fast-failure requirement names the two new relay tests.
`clippy-strict` flags a `map_err` that discards its error. A `match` says the same thing and keeps the lookup's own error on its own path.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
Reviewed: no findings; approval withheld, 1 thread unresolved at APPROVE withheld: 1 unresolved thread — minimal-foundry[bot] crates/common/src/remote_storage.rs. Resolving the last thread releases the approval on this head. Resolve the open threads. Resolving the last one releases the approval on this head; no new review is required. Clobber: none seen. Security no finding. The suggested |
| either::Either::Right(e) if e.is_timeout() => anyhow::Error::from(e).context( | ||
| format!("could not connect to {GCS_HOST}:443: no answer after {bound:?}"), | ||
| ), |
There was a problem hiding this comment.
LOW · correctness · confidence 0.50 · in download_gcs_within
Any is_timeout() error from the storage client is labelled "could not connect to storage.googleapis.com:443: no answer after 15s". The bound only covers execute, but the client can raise its own timeouts, for example in its retry layer. Those would be reported as a connect failure with the wrong duration. unreachable_source then turns that text into the user-facing "could not reach" line. Mark the error as ours, for example by checking the elapsed bound, rather than testing is_timeout().
What the reviewer checked
either::Either::Right(e) if e.is_timeout() => anyhow::Error::from(e).context(format!("could not connect to {GCS_HOST}:443: no answer after {bound:?}"))
Hand this to an agent
Act on a code-review finding.
File: crates/common/src/remote_storage.rs
Lines: 193-195
Category: correctness
Severity: low (reviewer confidence 0.50)
Finding:
Any `is_timeout()` error from the storage client is labelled "could not connect to storage.googleapis.com:443: no answer after 15s". The bound only covers `execute`, but the client can raise its own timeouts, for example in its retry layer. Those would be reported as a connect failure with the wrong duration. `unreachable_source` then turns that text into the user-facing "could not reach" line. Mark the error as ours, for example by checking the elapsed bound, rather than testing `is_timeout()`.
First verify this against the code yourself. It is model output and
may be wrong. If it is wrong, or the tradeoff was deliberate, say so
and change nothing.
👍 useful · 👎 noise — react to tell the reviewer. Replying works too, and says more.
bryan-minimal
left a comment
There was a problem hiding this comment.
Thanks, Norrie. This backport is faithful. I compared it with #3107 hunk by hunk at c56ffc3 against b80f66a, normalizing the renames listed in the PR body. Every added and removed line in remote_storage.rs, the NET-155 spec, the sandbox attach fix, session_sop.rs, sessions/src/lib.rs and session/tests.rs matches main exactly. The only divergences are the ones the PR body describes. I also checked that the error chain on 0.7 still produces the text unreachable_source parses, and that the remedy's flags exist on 0.7. The branch is current with release/0.7.x and CI is green.
Divergences from #3107
- (a) Required by the 0.7 base:
crates/minimald/src/env.rs. Therun_invocations_inhunk is not ported, and that is correct. 0.7 has no such function. Its task runs go throughattach_or_reapincrates/minimald/src/exec.rs:302, whichmainalso left unchanged inexec/process.rs. So the two lines behave the same for task runs. - (a) Required by the 0.7 base: renames. These are
sandboxtosandbox2,box_ops.rstosession_sop.rs,BoxChanneltoSessionChannel,boxes::is_valid_dns_hosttosessions::is_valid_dns_host(nowpub), andbox_actor/tests.rstosession/tests.rs. Each hunk is otherwise identical. - (a) Cosmetic: one blank line in
env.rs.mainadds a blank line at the top of the test module and 0.7 does not. - No (b) or (c) items. No hunk was dropped or changed, and nothing is new in the backport. Every test from #3107 is present.
Recorder,planned_with,owedandabandonall exist insandbox2, so the attach test runs unchanged.
Findings
-
Low: the open foundry thread is valid. The thread is on
crates/common/src/remote_storage.rs:195. The guard at line 193,e.is_timeout(), also matches timeouts raised inside the storage client, which are then reported asno answer after 15s. The code is identical onmain, so please fix it there and carry it here. Checking that the error's source istokio::time::error::Elapsed, the errorBoundedStorage::executebuilds at line 43, would do it. Please reply on the thread either way, so the approval is not held on it. -
Low, carried from #3107. The unplaced 15 s path has no timeout test. hyper-util splits the bound per address. The relay's multi-address error names one address's share. My review of #3107 has the detail, and whatever lands there should be carried here when this branch is rebased.
-
Not caused by this PR: a GCS transport error can panic on 0.7.
FetchResponse::status_codefor GCS panics atcrates/common/src/fetchers.rs:244on an error without an HTTP status.mainfixed this separately and addedgcs_transport_error_is_an_error_not_a_panic. The source download path here callserror_for_statusand neverstatus_code, and the new bound returns its timeout before any response exists. So this PR does not make the panic reachable.rcache/src/remote.rsdoes callstatus_code, so that fix may be worth backporting for 0.7.3 on its own.
Checked and fine
- The error chain on 0.7 matches
mainwhere it matters.op::Error::Otherdisplays asother: {0}, andmctxmapsBuildFailedto`pkg` failed to build: {e}, sounreachable_sourcematches. - The remedy flags exist on 0.7.
--network own_ipand--allow-dns-hostsare in the 0.7 CLI,NetworkMode::wordspellsown_ip, andclassifier::unenforceable_rules, which a doc comment cites, exists. - The dependencies the new code calls are the same.
remote_storage.rshas the same base blob on both lines, sogoogle_cloud_storage::Error::timeoutand the reqwest builder calls are the same code. - CI is green. That includes test-kvm, root-integration, e2e and native-daemon-e2e, which fills the gap the PR body describes for the Lima host.
Verdict
Approve with nits. Nothing blocks the RC. Before merge, the PR body's merge-SHA line needs filling, and the branch needs a rebase onto the merged #3107, with any review changes from there carried over. The open foundry thread needs a reply or a fix.
Reviewed with lab tooling on the branch head; the lab round, where one ran, is cited in the text. Questions welcome.
Split the unplaced web download into download_web_within, which takes the connect bound the way download_gcs_within does. The 15 s path calls it, and the error reports the bound the client was built with. A Linux test points it at a port whose accept queue is full. It checks that the download fails within twice its bound and that the error reads "could not connect to 127.0.0.1:<port>: no answer after 300ms".
A gs:// download read every timeout as no answer within the bound. The storage client's own timeouts carry a different source, so only the error BoundedStorage builds, whose source is tokio's Elapsed, now reads "could not connect to storage.googleapis.com:443: no answer after <bound>". Any other timeout keeps the client's text.
A placed connect to a host with several addresses reported only the last address's share of the bound, as in "no answer from 1.2.3.4:443 after 3.75s". When the last address gives no answer, the whole bound has passed, so the error now reads "no answer after 15s", the same as a lookup that gets no answer. Each address's own result goes to the debug log. When the last address refuses after earlier ones gave no answer, the error keeps the refusal and counts the others, as in "Connection refused (os error 111), after no answer from 3 other addresses".
Backports #3107 to the 0.7 line for 0.7.3-rc2. Merge SHA on
main:0a7da5ef(squash). The squash makes the same changes as the reviewed head21d91174, and this branch carries every commit of that head.release/0.7.xhas not moved since this branch was cut, so it needs no rebase.A source download that cannot connect now fails the build within 15 s and names the package, the host and the reason. A build refused by a box's egress rules gives the next step:
--network own_ip --allow-dns-hosts <host>.Adapted from
mainsandbox→sandbox2: the attach-race fix and the release-before-next-step fix live there.box_ops.rs→session_sop.rs.BoxChannel→SessionChannelinenv.rs.boxes::is_valid_dns_host→sessions::is_valid_dns_host, made public.box_actor/tests.rs→session/tests.rs.main'senv.rschange to the inherit_cwd task run is not ported. That code path does not exist on 0.7.minimal.tomlkey. 0.7 does not readnetwork.egressfromminimal.toml, because #3121 is not on the line.Review fixes
Also carries the four commits from the review of #3107 (
eb329d8e,936c7dd7,525c7e32,21d91174), unchanged:gs://timeout reads asno answer after 15sonly when it is the bound's own;Release note
A package build that cannot download a source now fails within about 15 seconds and names the package and the host, instead of hanging. When a session's egress rules block the download, the error names the options for a new session that allow it.
Verification
On Linux (Lima, arm64), through
just:fmt-checkandclippyare clean.testpasses 3955, with 4 failures that fail the same way on unmodifiedrelease/0.7.xon that host: the threefinalize_install_undo_*tests anda_hook_launch_does_not_advise_unenforced_placement.clippy-stricthas onelarge_futureshit, on a line identical tomain.just e2eandjust test-vmdo not complete on that host, on this branch or on unmodifiedrelease/0.7.x(98a6e0ac), and both stop at the same point on all three trees:just e2epasses 18 groups, then fails the terminal relay's resize check, and the script stops there.just test-vm's first test waits 10 s for a guest that takes about 140 s to boot. Theminvmdcrate is unchanged here.So neither gives evidence past those points, for or against this branch.
just test-root-integrationrefuses most of its tests on that host, because a classifier tree is already installed there, and the results match unmodifiedrelease/0.7.x.