Skip to content

fix(minimald): let a host-address grant box on a VM host reach the proxy - #3214

Merged
norrietaylor merged 2 commits into
mainfrom
fix/guest-root-daemon-host-ip-lane
Nov 10, 2026
Merged

norrietaylor merged 2 commits into
mainfrom
fix/guest-root-daemon-host-ip-lane

Conversation

@norrietaylor

Copy link
Copy Markdown
Member

Summary

After #3119, a host_ip box with a GitHub grant starts on a VM host. However, it still cannot reach the Box Egress Proxy, for two separate reasons. This PR fixes both. Live runs of the T122 cases on macOS found them; see #2500.

1. The guest's root eth0 has a random MAC address

  • bring_up_root_egress (crates/minimald/src/guest.rs) never sets a MAC address on the daemon's eth0 tap, so the kernel picks a random one.
  • On the switch, the proxy leg accepts an ARP claim (ArpClaim::holds) and a SYN (prescreen) only from the address's switch-derived MAC, MacAddr::for_switch_ip. A box's tap already carries that MAC (tap_netns_commands).
  • A host-address box shares the daemon's namespace, so its traffic leaves through eth0. The switch dropped every ARP request it sent for 100.64.255.252 and logged dropped an ARP claim whose sender pair is neither pinned nor the address's switch-derived MAC.
  • In the box, git failed with Failed to connect to github.com:443 over proxy 100.64.255.252 after 3069 ms, and bep never saw a connection.

Fix: a new set_interface_mac function uses the SIOCSIFHWADDR ioctl to give eth0 the daemon address's derived MAC before the interface comes up. The guest rootfs has no ip binary, so this follows configure_interface_v4.

2. The UDP 443 block needs sudo in the guest

  • HostMoves::add_blocked_leaf and remove_blocked_leaf always run sudo -n /usr/local/lib/minimal/quic443-leaf.
  • The guest daemon runs as root and has neither sudo nor the helper. So a box whose quic443 setting resolves to block failed at launch. Example: auto with a GitHub grant and DNS steering. The error was the session was stopped because QUIC (UDP to port 443) could not be blocked.
  • The proxy-lane set already handles a root daemon (BEP-083, change_lane_set).

Fix: both sets now go through one root-aware path. change_leaf_set and run_leaf_set_nft map the helper's add and remove verbs to quic443_blocked_leaves, and lane-add and lane-remove to bep_lane_leaves. As root they call nft directly; otherwise they go through sudo, as before.

Verification

  • The three live T122 cases in scripts/session-e2e.sh pass on macOS against a minvmd VM, with the initramfs built from this branch: bep_host_ip_grant_box_reaches_the_proxy_lane, bep_host_ip_git_and_gh_under_proxy_env and bep_host_ip_git_and_gh_under_dns_steering. Before this change, the first and third failed with the UDP 443 error and the second with the connect timeout.
  • root_leaf_set_change_runs_nft_on_the_named_set (formerly root_lane_change_runs_nft_on_the_lane_set) now also checks that add and remove reach quic443_blocked_leaves.
  • just fmt-check passes.
  • Not run locally: just test-cross (clippy and tests for minimald). The Linux CI lanes cover it.

No unit test covers the MAC fix: setting it needs a real tap device. The live cases above check it.

Two gaps kept a host_ip box with a GitHub grant from working on a VM host:

- The guest daemon's root eth0 kept a kernel-random MAC. The switch's Box
  Egress Proxy leg admits ARP claims and connections only from an
  address's switch-derived MAC, so every host-address box, which shares
  that interface, lost its ARP for the proxy's address and timed out.
  The tap now takes the daemon address's derived MAC before it comes up,
  as a box's tap does.
- The UDP 443 block always ran its helper through sudo. The guest daemon
  runs as root with neither, so a box whose quic443 stance blocks failed
  its launch. The blocked-leaves set now changes through the same
  root-aware path as the proxy-lane set: nft directly as root, the helper
  through sudo otherwise.
@norrietaylor
norrietaylor requested a review from a team as a code owner November 9, 2026 23:00
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 2 billable files and costs up to $0.50.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 1 minute for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 5 included reviews currently available. Your 21 included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: b2fe7629-ac4a-4db4-ab2f-6fb99f0b0d5f
Commits

Reviewing files that changed from the base of the PR and between 93952a9 and 739697a.

Files selected for processing: 2

  • crates/minimald/src/box_actor/host.rs
  • crates/minimald/src/guest.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@minimal-foundry

minimal-foundry Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Approved: no findings at 739697ad675e.

Nothing to do.

Clobber: none seen.

@norrietaylor

Copy link
Copy Markdown
Member Author

/arch-review

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Nov 9, 2026

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Norrie. Both halves check out against the code. The switch's proxy leg admits an ARP claim and a SYN only from an address's switch-derived MAC (ArpClaim::holds, crates/switch/src/bep_host.rs:1303), a box's tap carries that MAC, and the daemon's eth0 did not, so every host-address box's traffic was dropped at the leg; set_interface_mac gives eth0 MacAddr::for_switch_ip(daemon_ip) before configure_interface_v4 brings it up, and the ifreq layout (16-byte name, 16-byte sockaddr, 8 bytes of padding) is the kernel's 40 bytes. For the UDP 443 block, change_leaf_set and run_leaf_set_nft are the lane path of BEP-083 widened to the blocked-leaves set with the right nft verb per helper verb, and the guest table declares quic443_blocked_leaves (it is the installer's rendered ruleset, guest_table_is_the_installers_rendered_ruleset), so the root nft add element lands in an existing set. The renamed test now pins all four verbs and both refusals.

Findings

  1. Medium: a failed MAC change now takes the daemon's own egress down with it. crates/minimald/src/guest.rs:1391. set_interface_mac(...)? returns from bring_up_root_egress before the address, the route and the relay exist, so a guest whose SIOCSIFHWADDR fails (a tap driver or kernel that refuses it, or a future rootfs change) boots with no network at all, where before this change it had egress and only its host-address grant boxes could not reach the proxy. The MAC is needed by those boxes alone; the daemon's own fetches, pkgs clone and DNS never depended on it. Suggested: log the failure (tracing::warn! with the error) and go on with the random MAC, so the degradation stays confined to the boxes that need the derived MAC, and they still fail loudly at the switch as they did before (the dropped-claim warning and the 3 s connect timeout).

  2. Low: the ifreq layout has no compile-time check. guest.rs:1468-1473. configure_interface_v4 gets away without one, but the hardware-address variant is the one most likely to be miscounted (a sockaddr is 16 bytes, so the pad is 8). A const _: () = assert!(std::mem::size_of::<IfReqHwAddr>() == 40); next to the struct costs nothing and catches a padding slip on a new target, the way FstrimRange has its size pinned at guest.rs:1812.

  3. Low: the root path is looser than the helper it stands in for. crates/minimald/src/box_actor/host.rs:143 (run_leaf_set_nft). The installed helper refuses an add whose leaf is a symlink or not a directory, and a leaf that is not a single safe name under the allow root. The root path only requires the leaf to sit below the cgroup2 mount. Root needs no delegation boundary, so this is not a hole, but one leaf.is_dir() check on add and lane-add would keep the two paths honest to each other and turn a typo in a caller into the same InvalidInput the helper gives.

Checked and fine

  • addr.sa_family = libc::ARPHRD_ETHER is the kernel's expectation for SIOCSIFHWADDR (the family field carries the ARP hardware type), and the MAC bytes go into sa_data[0..6].
  • The change happens while the interface is down: set_interface_mac runs before configure_interface_v4 raises IFF_UP, which is where SIOCSIFHWADDR is allowed on a tap.
  • The relay's NET-084 check is on the IPv4 source, not the MAC, so the daemon address lease is unaffected; host-address boxes share the namespace and so the address and now the MAC.
  • for_switch_ip is deterministic from the low three octets, so the daemon's MAC cannot collide with a box's (distinct addresses on the same /16).
  • The verb map in run_leaf_set_nft is exhaustive over the helper's four verbs and refuses anything else before running nft; the test covers the four and the two refusals.
  • The non-root path is unchanged byte for byte: run_quic443_leaf_helper(sudo, helper, verb, leaf) with the same verbs as before.

Observation (outside this PR)

The installed helper (scripts/install-host-classifier.sh:705-709) knows only add and remove, while its header comment at line 126 and the non-root arm of change_leaf_set send it lane-add and lane-remove. On a native host, a non-root daemon's add_lane_leaf is therefore refused by the helper and logged with "its connections there stay refused" (classifier.rs:1127). If the lane set matters on native hosts, the helper needs the two verbs; if it is VM-only by design, a comment at the call site would save the next reader the trip. Not this PR's bug.

Verdict

Approve with nits. Finding 1 is the one I would change before merge, since it turns a boxes-only problem into a whole-guest one on the failure path; 2 and 3 are quick. The lab can give this a pre-push (CI's exact commands, including the renamed test) and the native host_ip egress scenario on two distros today if that helps the RC; the VM-host grant cases themselves need a macOS minvmd lane, which the lab does not have.


Reviewed with lab tooling on the branch head; the lab round, where one ran, is cited in the text. Questions welcome.

@minimal-arch minimal-arch Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arch-review: PASS · the full ruling

The change conforms. A root guest daemon now changes quic443_blocked_leaves with nft, the same way it changes the lane set. A non-root daemon still uses sudo -n and the helper, so the §7.4 privilege boundary does not move. A refusal still stops the box, so the §5.3 block fails closed. The guest eth0 now carries the switch-derived MAC of its own address. The §3 anti-spoof check on the switch does not change.

Ruled at 780b1b7 against gominimal/arch@9e2ff817adba by claude-opus-5-5.

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with nits (in the review above). The one I'd change before merge: a failed SIOCSIFHWADDR now fails the whole root egress bring-up, where only the grant boxes need the derived MAC; warn and continue would keep the daemon's own egress.

…e adding a missing leaf

Address review: a refused SIOCSIFHWADDR now warns and keeps the kernel's
MAC instead of failing the guest's root egress, the hardware-address
ifreq has its 40-byte size pinned at compile time, and the root nft path
refuses an add or lane-add of a leaf that is not a directory, as the
helper does.
@norrietaylor

Copy link
Copy Markdown
Member Author

Thanks, Bryan. All three are in 739697ad:

  1. The MAC change no longer takes egress down. A refused SIOCSIFHWADDR now logs a warning and keeps the kernel's MAC. The address, route and relay come up as before, so only host-address grant boxes lose the proxy.
  2. Size check added. const _: () = assert!(std::mem::size_of::<IfReqHwAddr>() == 40); sits next to the struct.
  3. The root path now refuses an add or lane-add of a leaf that is not a real directory, with InvalidInput, before it runs nft. A remove still runs for a leaf that is gone. The test now creates the leaf and checks that both adds of a missing leaf are refused without an nft call.

On the observation: the helper on main does take lane-add and lane-remove (scripts/install-host-classifier.sh:744-750, mapping them to $LANE_SET). The non-root lane path reaches a helper that knows the verbs.

I rebuilt the guest and host binaries from this head and reran the three live T122 cases on macOS against a minvmd VM. All three pass: proxy lane, proxy_env steering and DNS steering.

Generated by Claude Code

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — advisory review

Verdict COMMENT — No findings.

Panel: 1 of 3 personas ran (correctness); did not complete: security, supply-chain.

@norrietaylor
norrietaylor added this pull request to the merge queue Nov 10, 2026
@bryan-minimal

Copy link
Copy Markdown
Member

Lab round on 780b1b7, the head the review read:

  • Pre-push (CI's exact commands: fmt, clippy -D warnings, nextest workspace, doctests, the native daemon e2e, cargo-deny and the workflow lints): green, with the renamed root_leaf_set_change_runs_nft_on_the_named_set in it.
  • Scenario 727 (a host_ip box's declared egress enforced on a native host) and the baseline, once per distro on Ubuntu 24.04 and Fedora, a release build of main against this branch: pass on both sides, no verdict change.

739697a I checked by read: the MAC refusal now warns and keeps the kernel's MAC with the address, route and relay coming up after it; the 40-byte assert sits on the struct; run_leaf_set_nft refuses add and lane-add of a leaf that is not a directory with InvalidInput before any nft call, and the test creates the leaf and checks both refusals. On the helper verbs, agreed: install-host-classifier.sh on main maps lane-add/lane-remove, so the observation was against an older copy. Nothing further from me; the approval stands.

Merged via the queue into main with commit 8860728 Nov 10, 2026
36 checks passed
@norrietaylor
norrietaylor deleted the fix/guest-root-daemon-host-ip-lane branch November 10, 2026 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants