Skip to content

fix(minimald): run every process the daemon injects into a box in the box's leaf - #3593

Open
norrietaylor wants to merge 2 commits into
mainfrom
fix/932-forked-processes-box-leaf
Open

norrietaylor wants to merge 2 commits into
mainfrom
fix/932-forked-processes-box-leaf

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

What changes

  • The one hook that ran in the daemon's leaf, the attach hook (run inside the host loop through InjectedCommands), now joins the box's classifier leaf before the box's namespaces, as an exec does. The headless hooks (activate at finalize, the re-run on resume or restart, detach, destroy) already went through command_in_session and were placed. Exec and hooks now share one placed_injection_command, which also runs the host_ip cohort's UDP 443 check (BEP-018).
  • The shim's leaf join is fatal for every leaf, not only a deny-all box's. An injected process that cannot join its box's leaf stops before its program runs, so it never runs in the daemon's own leaf.
  • A failed join is a refusal, not a failed command. A leaf that is gone, because the box stopped first, prints "The box stopped before the command started." and exits 2. Any other failure prints "Did not run the command: the host could not apply the box's network rules to it ()." and exits 5. min box exec reports these as " stopped before the command started." (exit 2) and "Did not run the command in : the host could not apply the box's network rules to it." (exit 5).
  • The daemon logs how the in-box dial for a forwarded port (direct-tcpip) ended. A dial that never ran is a warning that gives its exit code.

Package builds are out of scope here: main runs them in the box's leaf (NET-155), and #3256 moves them into build boxes. A task's own process in a host-address box is covered by #3610.

Spec

NET-079 gains two sub-requirements. First, a process the daemon starts inside a host-address box that has a classifier leaf, by joining its namespaces (an exec, a lifecycle hook, the dial of a forwarded port), runs in that leaf and never in the daemon's. The attach shell is the box's launch, not an injection. Second, a process that cannot join the leaf is stopped before its program runs, whatever the box's declaration, and ends with an exit code of its own that min box exec reports as a refusal.

Tests

  • nsenter_integration::an_injection_that_cannot_join_its_box_leaf_is_refused_for_every_verdict (drives the real shim)
  • nsenter_integration::an_injection_whose_box_leaf_is_gone_says_the_box_stopped
  • nsenter::tests::a_failed_leaf_join_is_one_line_and_an_exit_code_of_its_own
  • nsenter::tests::an_injection_joins_its_box_leaf_by_writing_its_pid
  • box_runtime::tests::an_exec_and_a_hook_into_a_placed_box_join_its_leaf
  • hooks::tests::a_hook_injection_joins_the_box_leaf
  • hooks::tests::a_hook_into_a_host_address_box_takes_the_udp_443_check
  • connection::tests::an_in_box_dial_that_did_not_run_is_logged_with_its_exit_code
  • minimal tests::box_exec_reports_an_injection_that_never_started_as_a_refusal

Verification

Linux (umask 022): just fix and just clippy clean; -p minimald 1536 passed, -p minimal 1195 passed, -p minimald-rpc 91 passed, no failures. macOS just test: 1721 passed. Not run locally: just test-cross, just e2e, just test-root-integration; nothing has run against a real cgroup yet.

… box's leaf

A lifecycle hook now joins the box's classifier leaf before it joins the
box's namespaces, as an exec does: both build their command through one
placed_injection_command, which also carries the UDP 443 cohort check.

The shim's leaf join is now fatal for every leaf, not only a deny-all
box's, so an injected process that cannot join its box's leaf stops
before its program runs rather than run in the daemon's own leaf.

NET-079 gains the two sub-requirements this binds.
@norrietaylor
norrietaylor requested a review from a team as a code owner November 18, 2026 22:13
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 13 billable files and costs up to $3.25.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 6 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available. Your 36 included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 6d6a831b-06ec-4430-be6c-67e54ac2e096
Commits

Reviewing files that changed from the base of the PR and between fa65ffa and 7eda83f.

Files selected for processing: 13

  • crates/minimal/src/cli.rs
  • crates/minimal/src/cmd/boxes/exec.rs
  • crates/minimal/src/main.rs
  • crates/minimal/src/tests.rs
  • crates/minimald-rpc/src/exec.rs
  • crates/minimald/src/box_runtime.rs
  • crates/minimald/src/box_runtime/tests.rs
  • crates/minimald/src/connection.rs
  • crates/minimald/src/hooks.rs
  • crates/minimald/src/net/classifier.rs
  • crates/minimald/src/nsenter.rs
  • crates/minimald/tests/nsenter_integration.rs
  • docs/specs/18-spec-box-networking/18-spec-box-networking.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Spec lint

docs/specs/18-spec-box-networking/18-spec-box-networking.md — 10 finding(s)

docs/specs/18-spec-box-networking/18-spec-box-networking.md:171: P004 NET-009 names test 'native_resolution_without_proxy_env', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:378: P004 NET-040 names test 'fresh_install_own_ip_ingress_publishes_loopback', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:388: P004 NET-041 names test 'installer_switch_binary_executable', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:435: P004 NET-048 names test 'linux_amd64_manifest_ships_vm_stack', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:440: P004 NET-049 names test 'fresh_linux_kvm_activate_local_minvmd', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:445: P004 NET-050 names test 'linux_arm64_manifest_ships_vm_stack', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:450: P004 NET-051 names test 'fresh_arm64_kvm_activate_local_minvmd', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:958: P004 NET-132 names test 'proxy_sees_each_vm_box_by_its_switch_address', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:984: P004 NET-134 names test 'deny_all_box_reaches_proxy_and_no_other_host_port', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:1008: P004 NET-107 names test 'session_outbound_request', which was not found in the repository

Advisory. None of this blocks the merge; the review decides.

Advisory, and never a merge gate. Linter: gominimal/foundry@3a8fa378, spec format: How Minimal Writes Specs.

@minimal-foundry

minimal-foundry Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Approved: no findings at 7eda83f5b421.

Nothing to do.

Clobber: none seen.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Nov 18, 2026
@norrietaylor

Copy link
Copy Markdown
Member Author

/arch-review

@minimal-arch minimal-arch Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arch-review: PASS · the full ruling

The change conforms to §4.1 line 54. A process outside the boxes cgroup subtree takes the node-plane identity, and minimald must keep a box's placement inescapable. Exec and hook injections now take one path, placed_injection_command, and the shim refuses any failed leaf join before setns. A box with no leaf joins nothing, so the §7.4 exception for a host without enforcement still holds. One small fix: the comment at spec line 717 says the exception is "above", but it is below.

Ruled at 5fefdd0 against gominimal/arch@327e1b57d16b by claude-opus-5-5.

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read at 5fefdd0, by tracing the code; no build and no run on a host with the classifier. The placement is sound: every injection (exec, detached exec, the direct-tcpip dial, all four hook events) now builds its command in placed_injection_command, the shim writes its own pid to the leaf first thing in shim_main (single-threaded, before setns and the fork), a failed write exits before setns, and a box with no leaf behaves as before. Nothing blocking. Four should-fix points, mostly about what a failure looks like and what the tests can catch.

1. A failed join reads as the command failing (should-fix). The shim prints minimald: joining the box's classifier leaf <cgroup path>: <errno> (join_classifier_leaf in nsenter.rs), then main returns the error, so a second line Error: Other("joining the box's classifier leaf …") follows and the process exits 1.

  • min box exec b -- make on an allow-list box whose leaf write is refused: two lines in the daemon's terms, no next step, and exit 1, which a script cannot tell from make failing.
  • An exec that races min box stop: the leaf is gone, so the person reads "No such file or directory" about a cgroup path where "the box stopped" is what happened.

This was already so for deny-all leaves. The PR makes it so for every host-address box. Exec propagates the command's exit code, so a refusal by Minimal wants its own code (as the exit 5 refusals have) and one plain sentence.

2. The in-box dial for direct-tcpip now fails without a trace (should-fix). dial_in_box in connection.rs sends the shim's stderr to null. Before, a failed join on a box that is not deny-all was advisory and the dial ran. Now the shim exits 1, the channel opens and closes at once, and the daemon logs nothing because spawn() succeeded. The forward looks dead with no cause. Logging the shim's exit status in the daemon would be enough.

3. The "every verdict" test cannot catch what it is named for (should-fix). an_injection_that_cannot_join_its_box_leaf_is_refused_for_every_verdict calls join_classifier_leaf directly, and that helper no longer looks at the verdict, so the loop over deny and allow adds nothing. Put an is_deny() advisory branch, or a let _ =, back at the call in shim_main and all three new tests still pass. No test goes through shim_main.

4. No test covers the exec half of the new spec line (should-fix).

  • Host::command_in_session exists only outside test builds. Replace its placed_injection_command(…) call with injection.command() and every test passes.
  • The spec cites the_classifier_leaf_travels_on_the_shims_argv_when_one_is_set for exec. That test checks how the Injection command is put together, not that exec uses it.
  • a_hook_injection_joins_the_box_leaf covers InjectedCommands, not hook_plan passing self.leaf, and it runs with host_ip_enforced: false, so the UDP 443 check never runs for a hook.
  • Nothing runs against a real cgroup. I can run the host-address scenarios on a classifier host against this head, as offered on #3588, if that helps.

5. The body overstates the gap (nit). Before this PR the headless hooks (activate at finalize, the re-run on resume or restart, detach, destroy) already went through HostHandle and command_in_session, and were placed. The one hook that ran in the daemon's leaf was the attach hook, which runs inside the host loop (run_hooks_for(Attach) → InjectedCommands). Also "attach" in the new spec line has no injection of its own: the attach shell is the launch, which #3588 covers.

6. A question, not about this diff. min task run in a host-address box: the task gets its own sandbox and task_network passes no leaf, while a package build gets one (NET-155). I could not confirm which leaf the task's own process runs in. Does NET-079 mean to cover it?

Also checked and right: a shim that dies on a failed join leaves the box sound (settle_injection gets ESRCH and skips it), and a blocked leaf's directory is never removed, so an exec cannot race a lifted block into ENOENT. I agree with the arch bot's note about where the exception sits in the spec.

Not checked: the real-cgroup refusals (a daemon under systemd Delegate=, for one), the exact text a tty exec prints, and the two test failures the body says also fail on main.

🤖 Generated with Claude Code

@norrietaylor

Copy link
Copy Markdown
Member Author

@bryan-minimal yes please, the same here. A fix for your review is in progress and will push a new head shortly. Running against that head would cover your points 1 and 2 on a real cgroup: an exec whose leaf join is refused, an exec racing min box stop, and a forward whose in-box dial cannot join. I'll say here when it lands.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@norrietaylor

Copy link
Copy Markdown
Member Author

@bryan-minimal thanks, addressed in 7eda83f:

  1. A failed join is now one line and a code of its own: a gone leaf (the stop race) prints "The box stopped before the command started." and exits 2; any other failure exits 5. min box exec reports both as refusals. A -t exec still shows the shim's line, with the same code.
  2. The direct-tcpip dial's child is reaped and its end logged; a dial that never ran is a warning with its exit code.
  3. The every-verdict case now runs the real shim (tests/nsenter_integration.rs), and goes red if the join in shim_main is made advisory.
  4. Exec placement moved into Host::injected_command and hook commands into Host::hook_commands, both reachable from tests; new tests go red if either stops using placed_injection_command or the leaf, and the UDP 443 check now runs for a hook with host_ip_enforced: true.
  5. The spec line drops "attach" (the attach shell is the launch), and the body now says only the attach hook ran unplaced.
  6. The task case is fix(mctx,minimald,minvmd): a task on a VM-backed host_ip box runs in the classifier leaf of its box #3610.

This head is ready for the classifier-host run you offered: an exec whose leaf join is refused, an exec racing min box stop, and a forward whose in-box dial cannot join.

@norrietaylor

Copy link
Copy Markdown
Member Author

/review

@norrietaylor

Copy link
Copy Markdown
Member Author

/arch-review

@minimal-arch minimal-arch Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arch-review: PASS · the full ruling

The change conforms to §4.1 of the egress design. Every injected process (exec, hook, forwarded-port dial) joins the box's classifier leaf before it joins the namespaces. A failed join stops the program before it runs, for every verdict. The new exit codes 2 and 5 agree with the exit-code table in architecture.md. Hooks now take the UDP 443 cohort check, as an exec does.

Ruled at 7eda83f against gominimal/arch@8b824707cde6 by claude-opus-5-5.

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second round, read at 7eda83f by tracing the code; no build and no run. All four points from the first round are answered. Nothing blocking. Two small should-fix points and three nits.

The first round's points

  1. The failed join is one line with a code of its own, and min box exec reports it as a refusal only when the code, the first stderr chunk and the absence of earlier output all agree (exec_refusal, reads_as_refusal), so a command that exits 2 with its own stderr stays the command's result. A script that reads only $? still cannot tell the two apart, as with the existing exit-5 refusals; I accept that.
  2. watch_in_box_dial reaps the dial's child and logs exits 2 and 5 at WARN with the code.
  3. The every-verdict case runs the real shim (tests/nsenter_integration.rs): make the join advisory and echo ran exits 0, so it goes red.
  4. an_exec_and_a_hook_into_a_placed_box_join_its_leaf goes red if injected_command or hook_commands drop the leaf, and the CLI test goes red if the mapping goes.

New

  • The cause behind exit 5 is lost for a plain exec (should-fix). The shim prints "Did not run the command: the host could not apply the box's network rules to it ()." min box exec replaces it with its own line, which has no cause and no next step; relayed_status returns the refusal and discards the held line, and the CLI test asserts an empty stderr. The daemon logs only exec ended exit_code=5. So for an exec without -t nobody can see whether the write was refused for permission, a busy group or something else. Keep the cause in the CLI's line and name min bug as the next step, as #3588's refusal does.
  • A stop race may read as a rules failure (should-fix, one line; the errno is from memory, please check it). ENOENT is mapped to "The box stopped before the command started." If the shim has cgroup.procs open and the stop's removal of the leaf lands before the write, I believe the write fails with ENODEV, not ENOENT, and the person gets the exit 5 "network rules" text for what was a stop. If that is right, treat ENODEV as ENOENT in join_refusal. I will try to catch it in the run on a classifier host.
  • Nit: the dial's stderr still goes to null, so the WARN has the code but not the shim's one line. Piping it and logging the line would give the cause.
  • Nit: a refused --detach exec shows only as exit 5 from box wait, not as a refusal.
  • Nit: two one-line seams are still untested: the call from command_in_session (built only outside tests) to injected_command, and ChildStdio::new calling watch_in_box_dial (the dial test runs the watcher by itself).

Checked and right: ENOENT on the leaf means the box stopped in every path I traced (the leaf is removed only in SandboxBackend::drop, after the box is reaped, and on an abandoned launch; place_pid never creates it); the two new texts say what happened in the person's words.

The run on a host with the classifier installed is being set up against this head and 4435a72 on #3588: an exec whose join is refused, an exec racing min box stop, a forward whose dial cannot join, and a launch whose supervisor placement fails. I will post what it shows on both PRs.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants