Skip to content

fix(common,minimald): fail a build fast when a source cannot connect (0.7 backport) - #3181

Merged
norrietaylor merged 12 commits into
release/0.7.xfrom
fix/backport-3107-0.7.x
Nov 9, 2026
Merged

norrietaylor merged 12 commits into
release/0.7.xfrom
fix/backport-3107-0.7.x

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Backports #3107 to the 0.7 line for 0.7.3-rc2. Merge SHA on main: 0a7da5ef (squash). The squash makes the same changes as the reviewed head 21d91174, and this branch carries every commit of that head. release/0.7.x has not moved since this branch was cut, so it needs no rebase.

A source download that cannot connect now fails the build within 15 s and names the package, the host and the reason. A build refused by a box's egress rules gives the next step: --network own_ip --allow-dns-hosts <host>.

Adapted from main

  • sandbox → sandbox2: the attach-race fix and the release-before-next-step fix live there.
  • box_ops.rs → session_sop.rs. BoxChannel → SessionChannel in env.rs. boxes::is_valid_dns_host → sessions::is_valid_dns_host, made public. box_actor/tests.rs → session/tests.rs.
  • main's env.rs change to the inherit_cwd task run is not ported. That code path does not exist on 0.7.
  • The refusal names no minimal.toml key. 0.7 does not read network.egress from minimal.toml, because #3121 is not on the line.

Review fixes

Also carries the four commits from the review of #3107 (eb329d8e, 936c7dd7, 525c7e32, 21d91174), unchanged:

  • a test that pins the unplaced 15 s bound and its text;
  • a gs:// timeout reads as no answer after 15s only when it is the bound's own;
  • when every address of a host gets no answer, the error reports the whole bound;
  • a host whose last address refuses after the others drop says how many gave no answer.

Release note

A package build that cannot download a source now fails within about 15 seconds and names the package and the host, instead of hanging. When a session's egress rules block the download, the error names the options for a new session that allow it.

Verification

On Linux (Lima, arm64), through just: fmt-check and clippy are clean. test passes 3955, with 4 failures that fail the same way on unmodified release/0.7.x on that host: the three finalize_install_undo_* tests and a_hook_launch_does_not_advise_unenforced_placement. clippy-strict has one large_futures hit, on a line identical to main.

just e2e and just test-vm do not complete on that host, on this branch or on unmodified release/0.7.x (98a6e0ac), and both stop at the same point on all three trees:

  • just e2e passes 18 groups, then fails the terminal relay's resize check, and the script stops there.
  • just test-vm's first test waits 10 s for a guest that takes about 140 s to boot. The minvmd crate is unchanged here.

So neither gives evidence past those points, for or against this branch. just test-root-integration refuses most of its tests on that host, because a classifier tree is already installed there, and the results match unmodified release/0.7.x.

A package build for an own-address box attaches to the box's network
after its process starts. A short build step, such as claude-code's
chmod and mv, can end before the attach finishes. Its namespace goes
with it, the attach fails on `nsenter`, and the launch killed a process
that had already ended and failed a build that had worked.

An attach that fails after its process ended now leaves that process's
exit status as the result. The plan's release still runs. A failed
attach of a process that still runs stays a failure.

Port to the 0.7 line: the change lands in sandbox2, the line's name for
the sandbox crate. main's other call site, the run of an inherit_cwd
task in minimald's env.rs, does not exist on the line, so minimald is
unchanged here.
On a VM host a source download that leaves as the daemon's own traffic is
dropped with no answer, so an activation hung until its RPC timed out at
240 s. See [#3058](#3058).

A download now fails within 15 s. A web download bounds its connect across
all of a host's addresses, on the placed relay too. The Google Cloud Storage
client has no connect bound and retries for minutes, so a `gs://` source
bounds the wait for its first answer. That bound ends with the first answer,
so a long download is not limited. It also means a `gs://` read gives up
retrying a failing first answer after 15 s, not 300 s.

The failure names the host, and the build reports the package, the host and
the reason. The reason is what the download saw, such as `no answer after
15s`, because no answer can have several causes and the download cannot tell
them apart.

Port to the 0.7 line: main's box_ops.rs is session_sop.rs here, and main's
BoxChannel is SessionChannel in env.rs.
A build the egress rules refused named the package, the host and the rules,
and stopped there. The person who read it had no command to run. It now adds:

    To allow it, activate a new session with --network own_ip
    --allow-dns-hosts <host>

The step names `--network own_ip` because on the default network the flag
alone is refused at create. A destination's port is dropped from the flag's
value, because a host allowance covers every port of the addresses it pins.

The step appears only on a connection the build opened itself. It is left
out for a literal address, which that flag does not take. It is left out for
a web source too: an own-address box has no classifier leaf, so its build
fetches the source as the daemon's own traffic, and in a VM no rule of the
box lets that out. It names no `minimal.toml` key, because none reaches a box
on this line.

Refs [#3058](#3058)

Port to the 0.7 line: main's boxes crate is sessions here, so
`is_valid_dns_host` is made public there. main's box_actor/tests.rs is
minimald's session/tests.rs.
A source download that cannot connect within 15 seconds fails the build and
names the package, the host and the reason. A refusal of a host name that a
build's own connection named gives the command options that allow it.

Refs [#3058](#3058)
A step that ended during its network attach left the plan's release to
`Drop`, which only spawns it. The next step's plan could run first and
replace the reservation, so the release gave back the new one and the old
one stayed held.

The attach now finishes the release before it returns. `Drop` still covers a
cancelled launch.
…resses

The relay split its 15 s bound equally over a host's addresses, after the
lookup. An address that refused at once wasted its share, so a later address
that needed longer could fail with time left. The lookup was outside the
bound, so name servers that never answer could exceed it.

One deadline now covers the lookup and every address. Each address gets the
time left, divided by the addresses left.
The requirement now says where the step appears, that it is for a new
session, and when it is left out. The fast-failure requirement names the two
new relay tests.
`clippy-strict` flags a `map_err` that discards its error. A `match` says the
same thing and keeps the lookup's own error on its own path.
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: a09a63f4-8b7e-46bc-a925-0ac08b1f880d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@minimal-foundry

minimal-foundry Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Reviewed: no findings; approval withheld, 1 thread unresolved at ca4492fa7811.

APPROVE withheld: 1 unresolved thread — minimal-foundry[bot] crates/common/src/remote_storage.rs. Resolving the last thread releases the approval on this head.

Resolve the open threads. Resolving the last one releases the approval on this head; no new review is required.

Clobber: none seen. Security no finding. The suggested --allow-dns-hosts host passes sessions::is_valid_dns_host, which allows only alphanumerics, -, _ and .. Build output cannot inject shell text into the next step. The connect bound only adds timeouts and does not widen egress. Not checked: runtime behaviour, which needs a build.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — advisory review

Verdict COMMENT — 1 finding, 1 inline.

Panel: 1 of 3 personas ran (correctness); did not complete: security, supply-chain.

Comment thread crates/common/src/remote_storage.rs Outdated
Comment on lines +193 to +195
either::Either::Right(e) if e.is_timeout() => anyhow::Error::from(e).context(
format!("could not connect to {GCS_HOST}:443: no answer after {bound:?}"),
),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW · correctness · confidence 0.50 · in download_gcs_within

Any is_timeout() error from the storage client is labelled "could not connect to storage.googleapis.com:443: no answer after 15s". The bound only covers execute, but the client can raise its own timeouts, for example in its retry layer. Those would be reported as a connect failure with the wrong duration. unreachable_source then turns that text into the user-facing "could not reach" line. Mark the error as ours, for example by checking the elapsed bound, rather than testing is_timeout().

What the reviewer checked

either::Either::Right(e) if e.is_timeout() => anyhow::Error::from(e).context(format!("could not connect to {GCS_HOST}:443: no answer after {bound:?}"))

Hand this to an agent
Act on a code-review finding.

File:       crates/common/src/remote_storage.rs
Lines:      193-195
Category:   correctness
Severity:   low (reviewer confidence 0.50)

Finding:
Any `is_timeout()` error from the storage client is labelled "could not connect to storage.googleapis.com:443: no answer after 15s". The bound only covers `execute`, but the client can raise its own timeouts, for example in its retry layer. Those would be reported as a connect failure with the wrong duration. `unreachable_source` then turns that text into the user-facing "could not reach" line. Mark the error as ours, for example by checking the elapsed bound, rather than testing `is_timeout()`.

First verify this against the code yourself. It is model output and
may be wrong. If it is wrong, or the tradeoff was deliberate, say so
and change nothing.

👍 useful · 👎 noise — react to tell the reviewer. Replying works too, and says more.

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Norrie. This backport is faithful. I compared it with #3107 hunk by hunk at c56ffc3 against b80f66a, normalizing the renames listed in the PR body. Every added and removed line in remote_storage.rs, the NET-155 spec, the sandbox attach fix, session_sop.rs, sessions/src/lib.rs and session/tests.rs matches main exactly. The only divergences are the ones the PR body describes. I also checked that the error chain on 0.7 still produces the text unreachable_source parses, and that the remedy's flags exist on 0.7. The branch is current with release/0.7.x and CI is green.

Divergences from #3107

  • (a) Required by the 0.7 base: crates/minimald/src/env.rs. The run_invocations_in hunk is not ported, and that is correct. 0.7 has no such function. Its task runs go through attach_or_reap in crates/minimald/src/exec.rs:302, which main also left unchanged in exec/process.rs. So the two lines behave the same for task runs.
  • (a) Required by the 0.7 base: renames. These are sandbox to sandbox2, box_ops.rs to session_sop.rs, BoxChannel to SessionChannel, boxes::is_valid_dns_host to sessions::is_valid_dns_host (now pub), and box_actor/tests.rs to session/tests.rs. Each hunk is otherwise identical.
  • (a) Cosmetic: one blank line in env.rs. main adds a blank line at the top of the test module and 0.7 does not.
  • No (b) or (c) items. No hunk was dropped or changed, and nothing is new in the backport. Every test from #3107 is present. Recorder, planned_with, owed and abandon all exist in sandbox2, so the attach test runs unchanged.

Findings

  1. Low: the open foundry thread is valid. The thread is on crates/common/src/remote_storage.rs:195. The guard at line 193, e.is_timeout(), also matches timeouts raised inside the storage client, which are then reported as no answer after 15s. The code is identical on main, so please fix it there and carry it here. Checking that the error's source is tokio::time::error::Elapsed, the error BoundedStorage::execute builds at line 43, would do it. Please reply on the thread either way, so the approval is not held on it.

  2. Low, carried from #3107. The unplaced 15 s path has no timeout test. hyper-util splits the bound per address. The relay's multi-address error names one address's share. My review of #3107 has the detail, and whatever lands there should be carried here when this branch is rebased.

  3. Not caused by this PR: a GCS transport error can panic on 0.7. FetchResponse::status_code for GCS panics at crates/common/src/fetchers.rs:244 on an error without an HTTP status. main fixed this separately and added gcs_transport_error_is_an_error_not_a_panic. The source download path here calls error_for_status and never status_code, and the new bound returns its timeout before any response exists. So this PR does not make the panic reachable. rcache/src/remote.rs does call status_code, so that fix may be worth backporting for 0.7.3 on its own.

Checked and fine

  • The error chain on 0.7 matches main where it matters. op::Error::Other displays as other: {0}, and mctx maps BuildFailed to `pkg` failed to build: {e}, so unreachable_source matches.
  • The remedy flags exist on 0.7. --network own_ip and --allow-dns-hosts are in the 0.7 CLI, NetworkMode::word spells own_ip, and classifier::unenforceable_rules, which a doc comment cites, exists.
  • The dependencies the new code calls are the same. remote_storage.rs has the same base blob on both lines, so google_cloud_storage::Error::timeout and the reqwest builder calls are the same code.
  • CI is green. That includes test-kvm, root-integration, e2e and native-daemon-e2e, which fills the gap the PR body describes for the Lima host.

Verdict

Approve with nits. Nothing blocks the RC. Before merge, the PR body's merge-SHA line needs filling, and the branch needs a rebase onto the merged #3107, with any review changes from there carried over. The open foundry thread needs a reply or a fix.


Reviewed with lab tooling on the branch head; the lab round, where one ran, is cited in the text. Questions welcome.

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: hunk for hunk the same as #3107 after the 0.7 renames. Fill in the merge sha and rebase once #3107 lands; the open bot thread on the GCS timeout label is the same point as on #3107.

Split the unplaced web download into download_web_within, which takes
the connect bound the way download_gcs_within does. The 15 s path calls
it, and the error reports the bound the client was built with.

A Linux test points it at a port whose accept queue is full. It checks
that the download fails within twice its bound and that the error reads
"could not connect to 127.0.0.1:<port>: no answer after 300ms".
A gs:// download read every timeout as no answer within the bound. The
storage client's own timeouts carry a different source, so only the
error BoundedStorage builds, whose source is tokio's Elapsed, now reads
"could not connect to storage.googleapis.com:443: no answer after
<bound>". Any other timeout keeps the client's text.
A placed connect to a host with several addresses reported only the last
address's share of the bound, as in "no answer from 1.2.3.4:443 after
3.75s". When the last address gives no answer, the whole bound has
passed, so the error now reads "no answer after 15s", the same as a
lookup that gets no answer. Each address's own result goes to the debug
log.

When the last address refuses after earlier ones gave no answer, the
error keeps the refusal and counts the others, as in "Connection refused
(os error 111), after no answer from 3 other addresses".

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE withheld: 1 unresolved thread — minimal-foundry[bot] crates/common/src/remote_storage.rs.

minimal-review — advisory review

Verdict COMMENT — No findings.

Panel: 2 of 3 personas ran (security, supply-chain); did not complete: correctness.

@norrietaylor
norrietaylor merged commit 412b9d2 into release/0.7.x Nov 9, 2026
33 checks passed
@norrietaylor
norrietaylor deleted the fix/backport-3107-0.7.x branch November 9, 2026 23:04
norrietaylor added a commit that referenced this pull request Nov 10, 2026
#3181 landed on the line as one squash commit (412b9d2) with the same tree
as the reviewed head ca4492f that this branch carries, so every conflict
keeps this branch's side. Also brings in #3215.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants