Repository navigation
fix(minimald): let a host-address grant box on a VM host reach the proxy - #3214
Conversation
Two gaps kept a host_ip box with a GitHub grant from working on a VM host: - The guest daemon's root eth0 kept a kernel-random MAC. The switch's Box Egress Proxy leg admits ARP claims and connections only from an address's switch-derived MAC, so every host-address box, which shares that interface, lost its ARP for the proxy's address and timed out. The tap now takes the daemon address's derived MAC before it comes up, as a box's tap does. - The UDP 443 block always ran its helper through sudo. The guest daemon runs as root with neither, so a box whose quic443 stance blocks failed its launch. The blocked-leaves set now changes through the same root-aware path as the proxy-lane set: nft directly as root, the helper through sudo otherwise.
|
Warning Review limit reached
This review includes 2 billable files and costs up to $0.50.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 1 minute for your next included review. View limit details
|
|
Approved: no findings at Nothing to do. Clobber: none seen. |
|
/arch-review |
bryan-minimal
left a comment
There was a problem hiding this comment.
Thanks, Norrie. Both halves check out against the code. The switch's proxy leg admits an ARP claim and a SYN only from an address's switch-derived MAC (ArpClaim::holds, crates/switch/src/bep_host.rs:1303), a box's tap carries that MAC, and the daemon's eth0 did not, so every host-address box's traffic was dropped at the leg; set_interface_mac gives eth0 MacAddr::for_switch_ip(daemon_ip) before configure_interface_v4 brings it up, and the ifreq layout (16-byte name, 16-byte sockaddr, 8 bytes of padding) is the kernel's 40 bytes. For the UDP 443 block, change_leaf_set and run_leaf_set_nft are the lane path of BEP-083 widened to the blocked-leaves set with the right nft verb per helper verb, and the guest table declares quic443_blocked_leaves (it is the installer's rendered ruleset, guest_table_is_the_installers_rendered_ruleset), so the root nft add element lands in an existing set. The renamed test now pins all four verbs and both refusals.
Findings
-
Medium: a failed MAC change now takes the daemon's own egress down with it.
crates/minimald/src/guest.rs:1391.set_interface_mac(...)?returns frombring_up_root_egressbefore the address, the route and the relay exist, so a guest whoseSIOCSIFHWADDRfails (a tap driver or kernel that refuses it, or a future rootfs change) boots with no network at all, where before this change it had egress and only its host-address grant boxes could not reach the proxy. The MAC is needed by those boxes alone; the daemon's own fetches,pkgsclone and DNS never depended on it. Suggested: log the failure (tracing::warn!with the error) and go on with the random MAC, so the degradation stays confined to the boxes that need the derived MAC, and they still fail loudly at the switch as they did before (the dropped-claim warning and the 3 s connect timeout). -
Low: the
ifreqlayout has no compile-time check.guest.rs:1468-1473.configure_interface_v4gets away without one, but the hardware-address variant is the one most likely to be miscounted (asockaddris 16 bytes, so the pad is 8). Aconst _: () = assert!(std::mem::size_of::<IfReqHwAddr>() == 40);next to the struct costs nothing and catches a padding slip on a new target, the wayFstrimRangehas its size pinned atguest.rs:1812. -
Low: the root path is looser than the helper it stands in for.
crates/minimald/src/box_actor/host.rs:143(run_leaf_set_nft). The installed helper refuses anaddwhose leaf is a symlink or not a directory, and a leaf that is not a single safe name under the allow root. The root path only requires the leaf to sit below the cgroup2 mount. Root needs no delegation boundary, so this is not a hole, but oneleaf.is_dir()check onaddandlane-addwould keep the two paths honest to each other and turn a typo in a caller into the sameInvalidInputthe helper gives.
Checked and fine
addr.sa_family = libc::ARPHRD_ETHERis the kernel's expectation forSIOCSIFHWADDR(the family field carries the ARP hardware type), and the MAC bytes go intosa_data[0..6].- The change happens while the interface is down:
set_interface_macruns beforeconfigure_interface_v4raisesIFF_UP, which is whereSIOCSIFHWADDRis allowed on a tap. - The relay's NET-084 check is on the IPv4 source, not the MAC, so the daemon address lease is unaffected; host-address boxes share the namespace and so the address and now the MAC.
for_switch_ipis deterministic from the low three octets, so the daemon's MAC cannot collide with a box's (distinct addresses on the same /16).- The verb map in
run_leaf_set_nftis exhaustive over the helper's four verbs and refuses anything else before runningnft; the test covers the four and the two refusals. - The non-root path is unchanged byte for byte:
run_quic443_leaf_helper(sudo, helper, verb, leaf)with the same verbs as before.
Observation (outside this PR)
The installed helper (scripts/install-host-classifier.sh:705-709) knows only add and remove, while its header comment at line 126 and the non-root arm of change_leaf_set send it lane-add and lane-remove. On a native host, a non-root daemon's add_lane_leaf is therefore refused by the helper and logged with "its connections there stay refused" (classifier.rs:1127). If the lane set matters on native hosts, the helper needs the two verbs; if it is VM-only by design, a comment at the call site would save the next reader the trip. Not this PR's bug.
Verdict
Approve with nits. Finding 1 is the one I would change before merge, since it turns a boxes-only problem into a whole-guest one on the failure path; 2 and 3 are quick. The lab can give this a pre-push (CI's exact commands, including the renamed test) and the native host_ip egress scenario on two distros today if that helps the RC; the VM-host grant cases themselves need a macOS minvmd lane, which the lab does not have.
Reviewed with lab tooling on the branch head; the lab round, where one ran, is cited in the text. Questions welcome.
There was a problem hiding this comment.
arch-review: PASS · the full ruling
The change conforms. A root guest daemon now changes quic443_blocked_leaves with nft, the same way it changes the lane set. A non-root daemon still uses sudo -n and the helper, so the §7.4 privilege boundary does not move. A refusal still stops the box, so the §5.3 block fails closed. The guest eth0 now carries the switch-derived MAC of its own address. The §3 anti-spoof check on the switch does not change.
Ruled at 780b1b7 against gominimal/arch@9e2ff817adba by claude-opus-5-5.
bryan-minimal
left a comment
There was a problem hiding this comment.
Approving with nits (in the review above). The one I'd change before merge: a failed SIOCSIFHWADDR now fails the whole root egress bring-up, where only the grant boxes need the derived MAC; warn and continue would keep the daemon's own egress.
…e adding a missing leaf Address review: a refused SIOCSIFHWADDR now warns and keeps the kernel's MAC instead of failing the guest's root egress, the hardware-address ifreq has its 40-byte size pinned at compile time, and the root nft path refuses an add or lane-add of a leaf that is not a directory, as the helper does.
|
Thanks, Bryan. All three are in
On the observation: the helper on I rebuilt the guest and host binaries from this head and reran the three live T122 cases on macOS against a minvmd VM. All three pass: proxy lane, Generated by Claude Code |
|
Lab round on 780b1b7, the head the review read:
739697a I checked by read: the MAC refusal now warns and keeps the kernel's MAC with the address, route and relay coming up after it; the 40-byte assert sits on the struct; |
Summary
After #3119, a
host_ipbox with a GitHub grant starts on a VM host. However, it still cannot reach the Box Egress Proxy, for two separate reasons. This PR fixes both. Live runs of the T122 cases on macOS found them; see #2500.1. The guest's root
eth0has a random MAC addressbring_up_root_egress(crates/minimald/src/guest.rs) never sets a MAC address on the daemon'seth0tap, so the kernel picks a random one.ArpClaim::holds) and a SYN (prescreen) only from the address's switch-derived MAC,MacAddr::for_switch_ip. A box's tap already carries that MAC (tap_netns_commands).eth0. The switch dropped every ARP request it sent for100.64.255.252and loggeddropped an ARP claim whose sender pair is neither pinned nor the address's switch-derived MAC.gitfailed withFailed to connect to github.com:443 over proxy 100.64.255.252 after 3069 ms, andbepnever saw a connection.Fix: a new
set_interface_macfunction uses theSIOCSIFHWADDRioctl to giveeth0the daemon address's derived MAC before the interface comes up. The guest rootfs has noipbinary, so this followsconfigure_interface_v4.2. The UDP 443 block needs
sudoin the guestHostMoves::add_blocked_leafandremove_blocked_leafalways runsudo -n /usr/local/lib/minimal/quic443-leaf.sudonor the helper. So a box whosequic443setting resolves to block failed at launch. Example:autowith a GitHub grant and DNS steering. The error wasthe session was stopped because QUIC (UDP to port 443) could not be blocked.change_lane_set).Fix: both sets now go through one root-aware path.
change_leaf_setandrun_leaf_set_nftmap the helper'saddandremoveverbs toquic443_blocked_leaves, andlane-addandlane-removetobep_lane_leaves. As root they callnftdirectly; otherwise they go throughsudo, as before.Verification
scripts/session-e2e.shpass on macOS against a minvmd VM, with the initramfs built from this branch:bep_host_ip_grant_box_reaches_the_proxy_lane,bep_host_ip_git_and_gh_under_proxy_envandbep_host_ip_git_and_gh_under_dns_steering. Before this change, the first and third failed with the UDP 443 error and the second with the connect timeout.root_leaf_set_change_runs_nft_on_the_named_set(formerlyroot_lane_change_runs_nft_on_the_lane_set) now also checks thataddandremovereachquic443_blocked_leaves.just fmt-checkpasses.just test-cross(clippy and tests forminimald). The Linux CI lanes cover it.No unit test covers the MAC fix: setting it needs a real tap device. The live cases above check it.