Repository navigation
T75 — Load the classifier's one ruleset inside the guest kernel so a VM host decides a host-address box's verdict per box - #1886
Conversation
…0s ran out the coder's uncommitted work as it stood when its 7200s ran out The coder was stopped at the end of its time with these changes not committed. The runtime committed its working tree as it stood so the work is kept: this commit is a snapshot, not a finished step, and may not build or pass its tests.
…lassifier-s-one-ruleset-inside-the-gue
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
… test The guest's recheck reads its table back out of its kernel with nft list, so a guest that decides per box is decided over a listing a loaded table answers with — the decision test's guest arm now hands it that fact instead of reading a test host's missing /usr/sbin/nft, which read the marker's guest as not effective and failed the suite. Also collapse the three nested ifs clippy refused in the guest's render and load, the same code with one fewer branch.
The guest half of NET-079 — the boot's render, check, load and marker, the recheck behind the marker, the effect probe over held listeners, and the per-launch decision the network plan follows — is now pinned by the tests the task names: - the guest's render is the installer's own, byte for byte, over both spellings of the two identities (the collapsed pair an un-enrolled guest has, and the harness pair), and the digest of a load covers exactly the bytes piped to nft, in the guest's lane and the install's, over one render, never a re-render; - the installer runs from the one compiled-in file on stdin, with its environment cleared and every parameter as argv after --print-ruleset, never interpolated into script text; - the marker is the load's commit point: written only after the check and the load, cleared before either, absent on any failure, and a failed load leaves the guest refusing deny-all boxes again on its own cause, which names no install command inside a microVM; - the decision re-reads the table behind its marker with nft list before anything rests on it, and only a refusal the chain read as decides per box — a reading that could not run, or a probe whose listener is not live, reads as unknown and refuses the deny-all box rather than guess either way; - the dstnat carve-out is the deny subtree's alone: one rule, the answerer's address and port, never loopback-wide :53 and never the node plane's DNS; - a deny-all host-address box on a VM host resolves through the guest's own answerer when its launch decided per box, and falls back to the node's DNS layer when it did not — the memo the launch's reader recorded, read per launch. Also fixes the clippy lane the CI check names: three let-chains where the new guest code nested ifs, and the decision test the guest's recheck made read a table no test host can list.
|
Round 1 of 6 on a failed check suite, pushed as The coder ran out of its 7200s before it finished. If it left work uncommitted, its last commit is the runtime's snapshot of that work as it stood — the commit message says so — and not a finished step: it may not build or pass its tests.
Verify Outside the task's
Converted to a draft: something above needs a person before review. On a VM-backed host, the guest kernel accepts and loads the classifier table the native installer renders, byte for byte the installer's own render for the same parameters, the digest taken over the exact bytes written to nft's stdin in both the guest and the native installer's load, never a re-render; a deny-all host-address box's outbound probe is refused while its resolver at 127.0.0.1:7656 still answers, and its session shows host_ip_enforcement per_box; per_box rests on the daemon's own control connect to a loopback listener it holds succeeding and, immediately after, a deny-subtree probe child's connect to that listener failing fast with EHOSTUNREACH, EACCES or EPERM in every family probed (IPv4 in the guest, ::1 too wherever IPv6 loopback is enabled); a guest whose check or load failed, whose table is gone behind its marker, whose control leg fails, or whose probe connects, times out or sees any other errno reports none, refuses the deny-all box again and says why in the bundle. |
|
/revise Round 2. All lanes are green at 9f34532, but two plan layers are missing, and the guest load path has defects.
Keep |
|
/revise Round 3. Round 2 ran out of its 7200 s, and nothing it did was published: its work modified Never modify
Round 2's items 5 to 7 (bash and nft timeouts, the root-lane digest assertion, the native log level) come next round. Keep |
…lassifier-s-one-ruleset-inside-the-gue
The vmhost layer the task's third round owes: eight guest_kernel / guest_classifier proofs, one VM boot each through the supervisor path, each asserting what the layer states — that the guest's nft -c accepted the rendered table and the load line carries the digest of exactly the bytes piped to nft -f -; that the digest equals the sha256 of the native installer's --print-ruleset output for the same parameters, so the guest and the native installer load one ruleset; that the deny-all box launches over the daemon's own nft-list recheck; that the box's own outbound probe reads EHOSTUNREACH; that its own box name resolves through 127.0.0.1:7656, the one carve-out, with the answerer's serving and answered-lookup lines tying the reply to that port; that a connection another box opens to it through the hostname proxy is answered, the reply-direction admission; that the launch's own record says host_ip_enforcement=per_box; and that every effect probe the guest ran read a refusal per family with an errno inside EHOSTUNREACH, EACCES and EPERM, named in the probe's own record. All eight are gated behind minvmd_libkrun plus MINVMD_E2E=1 with MINVMD_VM_LANE turning a missing precondition into a lane failure, the arm egress_allowlist defines; on this box, where libkrun is absent, the file type-checks only under the forced cfg.
The guest's boot load dropped two of its four failure variants on the floor: a render that could not run and a marker that could not come or be written returned from load_guest_table with no line of their own, and the boot's caller drops the Result outright, so a guest whose load stopped before its check — a bash that is not there, a marker a daemon death left undeletable, a marker that could not be written over a loaded table — said nothing at all while reporting no per-box verdict. Every variant now logs at error level at its own failure site, the same discipline the check and load lines already keep, each naming its own consequence: the stale marker that stopped the load before anything rendered, the render that ran no check and wrote no marker, and the marker that could not vouch for a table that loaded. The proof is the render half: a load through a bash path that does not exist returns Render with the spawn's own words naming it, runs no check behind it, leaves no marker — not even the one a previous load wrote — and puts the error line on the daemon's log.
The guest's network plan read the verdict decision through a process-wide memo: decide_now wrote it, plan() read it back, and the awaits between the two let a concurrent launch's reading stand in for the launch's own - a deny-all host-address box could resolve through the answerer on a decision another launch read, and a task's host-address plan could follow a session's. network_for now takes the decision as a parameter, the launch passes the value its own reader read, and task_network passes None on the ground that a task places no classifier leaf. The memo (READ_DECISION, note_decision, freshest_decision) is deleted; the provider test takes the decision as a parameter across its scenarios, so the two serial attributes that existed only to keep tests off one memo go too.
…lassifier-s-one-ruleset-inside-the-gue
|
Round 3 of 6 on a /revise note, pushed as
Verify Outside the task's
Converted to a draft: something above needs a person before review. At this branch the guest classifier load is complete in the daemon and asserted at the VM host: the installer's own single render feeds the guest's nft -c -f then nft -f through a cleared environment by absolute path, the marker is written only after the load succeeds, and every render, check, load or marker failure now logs its own cause at error level and leaves no marker, so no per-box verdict is claimed on a boot that did not produce one; the new eight-test guest_classifier_integration layer boots a VM through the supervisor path and asserts the guest kernel accepting the rendered bytes, the loaded digest equal to the native installer's digest over exactly the bytes piped to nft's stdin, a deny-all host-address box's outbound probe refused with EHOSTUNREACH on IPv4 while its box-name lookup is answered through 127.0.0.1:7656, its session record reading host_ip_enforcement per_box, and the effect probe's per-family errno record; and each launch's network plan now follows the decision that launch itself read, never a concurrent launch's. The VM demonstration itself runs only on a libkrun-capable lane, so what a person can run here today is the native-lane half — render identity, digest over piped bytes, cleared env and argv, marker-after-load, dstnat scoping, per-family refusal record, per-launch resolver — all green, with the eight gated tests waiting on the vm lane and the rootfs pin. Notes from the agent: This round's three items, one commit each: bd1526c (the vmhost layer), 1f73f61 (every GuestLoadFailure variant logs at error level), 708a780 (the decision passes by parameter and the memo is deleted). PR body line the round asked for: the guest's /usr/sbin/nft and the kernel expressions the rendered table uses depend on the microvm-rootfs pin that moves on main in #1875 (Alpine's nftables apk plus its missing libraries, and NFT_SOCKET in the guest kernel); the guest_classifier_integration tests need that pin to pass on the vm lane. Deferred t…[truncated] |
|
/revise Round 4. Round 3 landed all three of its items. Do the items below, commit after each one, and stop. Never modify At
Keep |
…e's listeners The guest's kernel leaves lo down until something brings it up, and nothing in the boot did before the effect probe's listeners were held: the daemon's own egress step, which does, runs after READY. A bind on 127.0.0.1 over a loopback that is not up is EADDRNOTAVAIL — the error the macOS guest logged at boot, and with it a daemon holding no V4 listener that reads its loaded table's effect as unknown and refuses every deny-all host-address box for a cause its own boot could have ended. Bring lo up first, as the first step of the guest's classifier boot, now composed in one place so the order is the thing a test drives rather than what it assumes: loopback up, then the listeners held for the daemon's life, then the table load. Each half is still the boot's to attempt whatever the one before it did — a loopback that will not come up, and listeners that will not bind, are logged and the load runs behind them, because the marker's table is still the decision's own fact. The bind keeps its tolerance for a family this host does not carry.
…ives the bound The guest's render, check and load ran in pid 1 before READY over waits with no deadline: a bash or an nft that never returned would stop the daemon from ever serving, with nobody watching to interrupt it. Every run is now held to one bound, the boot's own GUEST_CHILD_DEADLINE, generous for runs that take milliseconds each. The feed is part of the run, so the script and the ruleset are piped from a thread beside the wait — a child that never reads cannot hold the write past the bound — and past it the child is killed, reaped so pid 1 carries no zombie, and the half it was running fails as its own GuestLoadFailure, logged like any other with the bound it overran. No marker is written for a table nobody loaded, and the previous table is untouched: the batch never commits. The check and load lines no longer say refused, because a run that never answered did not refuse anything; the VM lane's grep for the check's failure line moves with the wording. The hook runner's own bound keeps the same caveat this one now documents: the bound covers the child the boot spawned, not anything that child backgrounded.
…account The digest test's install lane returned early whenever the suite ran as root, so its digest equality — the one text the guest pipes, the installer's print, and the install's own load all agree on — never ran on the cross or Docker lanes at all. Root had no account to delegate to: the step's sudo seam names the account that ran sudo, and on a root-run lane that account is root itself, the one id the step refuses to delegate its tree to. So the lane now names a non-root id its kernel has a mapping for instead, read from the namespace's own uid and gid maps — a numeric chown to an id no namespace maps fails with EINVAL, so the map, not a guess, decides — with nobody's id standing in when a map carries no non-root one. The derivation is pinned by a test that runs on every lane, root or not, and the install lane itself no longer skips, so the digest it asserts is asserted everywhere the suite runs.
The task's one shared reader had raised both lanes' not-refused probe record to warn, which changed what T48 (#1805) had chosen for the native lane: the reading over a table a person installed, carried at info so the launch's own record beside it is what alarms. The two lanes keep their own levels now, threaded the way decide() already threads the lane itself: the guest's reading is its boot's own product, and a launch that reads anything but a refusal is the one line a bundle carries of why that guest refuses every deny-all host-address box, so it stays a warn there; the native lane is T48's and is not changed here, so it reads at info again.
…lassifier-s-one-ruleset-inside-the-gue
|
Round 4 of 6 on a /revise note, pushed as
Verify Outside the task's
Converted to a draft: something above needs a person before review. On a VM-backed host the guest's boot now brings its loopback up before it holds the probe's listeners, so the V4 listener the effect probe connects to is held whenever the boot reaches its load, and every render, check and load that boot runs is bounded: a bash or nft that never answers is killed and its half fails as its own GuestLoadFailure, logged like any other with no marker written for a table nobody loaded. The guest kernel accepts and loads the classifier table the native installer renders, byte for byte the installer's own render for the same parameters, the digest taken over the exact bytes written to nft's stdin in both the guest and the native installer's load, never a re-render — an equality the digest test asserts on root lanes too, delegating its rehearsal to a non-root account its kernel has a mapping for. A deny-all host-address box's outbound probe is refused while its resolver at 127.0.0.1:7656 still answers, and its session shows host_ip_enforcement per_box; per_box rests on the daemon's own control connect to a loopback listener it holds succeeding and, immediately after, a deny-subtree probe child's connect failing fast with EHOSTUNREACH, EACCES or EPERM in every family probed. A guest whose check or load failed, whose table is gone behind its marker, whose control leg fails, or whose probe connects, times out or sees any other errno reports none, refuses the deny-all box again and says why in the bundle, at warn on the guest lane while the native lane keeps T48's own level. Notes from the agent: All four directive items landed, one commit each: (1) guest.rs gained bring_up_loopback and main.rs's microvm block now composes boot_guest_classifier — loopback up, then listeners held, then the load — so the V4 listener is there at load time (the macOS EADDRNOTAVAIL was lo still down: nothing brought it up before the hold, the only lo step ran after READY; the bind tolerance was not widened); (2) render_guest_ruleset and run_guest_nft ar…[truncated] |
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
1 similar comment
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
The VM-backed guest's node DNS layer applies no per-box name rule to host-address boxes, so the guest's table carves no resolver out. The installer gains --no-resolver-carve-out, which omits the answerer admission in deny_out and the dstnat chain; the guest renders with it and logs one line naming the follow-up (gominimal/inbox#897). A deny-all host-address box in a deciding guest gets an empty resolver, and its DNS to the gateway is refused like any other destination. The native render is unchanged.
… a vm host Replace guest_deny_all_resolver_reachable with guest_deny_all_dns_to_gateway_refused: a deny-all box's udp and tcp 53 to the gateway reads an errno in the reject set, and the guest's render logs that it carves no resolver out. The native digest comparison renders with the guest's own parameters, --no-resolver-carve-out included.
…e live answerer The installer records the resolver carve-out it loaded beside the marker (carve-out-<address>-<port>) and clears it with the marker, as it does the ct-mark mask. The native decision reads that record, and server.rs keeps a process-wide cell with the address and port the zone answerer actually bound, cleared when its receive loop exits. A native deny-all launch over a deciding table is refused as TableNotEffective "stale carve-out", naming both values, when the record differs from the live bind, no answerer is bound, or no carve-out was recorded; the hint re-renders the install with --answerer-address/--answerer-port set to the live bind. The box's resolver is the live bind's address. A box that is not deny-all is unaffected.
A udp send to a refused destination often succeeds locally, so the udp leg asserts no reply arrives within the bound; the reject's errno or ECONNREFUSED on the connected socket also passes. The deny chain rejects, so the tcp leg keeps the reject-set errno.
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
The per-launch classifier line is written when the box's leaf is placed at its first launch, not at create. Run one command in the box, then read the line within the log deadline.
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
…g it The daemon's tracing output is coloured, so a structured field reads with escape codes between the key and its value and a plain key=value needle never matches.
|
/review |
|
Reviewed: 1 finding, none blocking at Address the comments on the diff. Clobber: none: |
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
The recheck runs before each host-address launch with no deadline, so an nft wedged on its netlink read would hold the launch forever. Run it through the bounded child wait every other guest child uses; past five seconds the table reads as gone.
|
/review |
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
The per-launch reading bound a fresh loopback listener on a guest as on a native host, so the listener the guest's boot holds for its effect probe was used only at start. A guest launch now reads over the held listener; a native host still binds its own per reading.
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
|
/review |
|
The runtime will not revise this pull request again on its own: round 6 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
norrietaylor
left a comment
There was a problem hiding this comment.
T75 matches the amended plan in #1812. On a VM host it renders no resolver carve-out, and DNS to the gateway is refused (inbox#897 follows up). On a native host the carve-out is pinned to the answerer's live bind and refused when stale. The per-launch recheck is bounded and probes over the boot-held listener. ARCH passed it, CI is green on every lane including KVM, and the review findings are fixed or deferred (inbox#901).
…minimal#1962) * build: bump the pkgs locked_commit for the guest kernel's conntrack marks Picks up gominimal/pkgs#803, which enables CONFIG_NF_CONNTRACK_MARK in the virtio-linux guest kernel. The guest classifier table's ct mark rule needs it (T75, gominimal#1886). One pkgs commit; the only file it changes is packages/virtio-linux/build.sh. * build: bump the pkgs locked_commit for the timezone database in base gominimal/pkgs#804 adds tzdata to base, so boxes resolve TZ=Area/City and Python's zoneinfo finds its zones.
T75 — Load the classifier's one ruleset inside the guest kernel so a VM host decides a host-address box's verdict per box
slice: S11
kind: task
covers: NET-079
size: L
est: 380 code, 560 tests
executor: agent
difficulty: unknown ground
model: strongest
diagnostics: the bundle's guest daemon log tail carries one line at boot with nft -c's verdict on the rendered table and nft's own error when it refuses, one line with the load's outcome and the sha256 of the exact text handed to nft, and one line per launch whose table recheck or effect probe failed behind a marker, naming which, the control leg's outcome, and each probed family's observed errno, success or timeout, so a VM host still refusing deny-all host-address boxes reads why in a bundle
observability: one info log line at guest boot when the table loads and the marker is written, naming the ruleset digest; one error line naming the cause when the check or the load fails; one warn line when a launch's recheck finds the table gone, its control leg fails, or a probed family connects, times out or fails with an errno outside EHOSTUNREACH, EACCES and EPERM; the existing per-launch classifier line then reads host_ip_enforcement per_box for a guest box
layers:
- daemon:
- .minimal/minimal.toml: locked_commit moves, by hand and never through mip update, to the gominimal/pkgs commit whose guest kernel builds in every expression the rendered table uses (NFT_SOCKET is the one the kernel lacks today) and whose microvm-rootfs closure pins Alpine's nftables apk and its missing libraries; nothing nft-related is packed into the initramfs
- crates/minimald/src/net/classifier.rs: a guest-only table load, run after the daemon's leaf and the deny and allow subtrees exist. The installer is compiled into minimald from scripts/install-host-classifier.sh itself (include_str of that one file, never a copy) and fed on stdin to /bin/bash -s, invoked by absolute path from the read-only rootfs with its environment cleared (no BASH_ENV, no ENV, no inherited PATH) and every parameter passed as argv after --print-ruleset, never interpolated into the script text, so no installer file is staged anywhere a box could reach; the render takes the node-plane identity and the host-address cohort identity as two separate inputs, and the guest's tree root and the answerer at 127.0.0.1 on the answerer port the guest binds. The rendered bytes go unchanged, by absolute path and with the same cleared environment, to the rootfs's root-owned /usr/sbin/nft first as nft -c -f -, then as nft -f - in one transaction; the logged sha256 covers exactly the bytes written to nft's stdin, never a re-render. The marker is created only after that load succeeds and stays only the fact that the load ran; any failure writes no marker and logs the cause and nft's error. The table's dstnat rule matches only deny-subtree sockets sending to 127.0.0.1 on udp 53, never loopback-wide :53 and never the node plane's DNS. In the guest, decide() and recorded() (before each host-address launch) decide per box only on effect: with no marker nothing is spawned and the guest does not decide; with a marker, nft list must show table inet minimal_class and its deny_out chain, and the effect probe must read rejected for per_box. The probe: immediately before it, a control leg connects from the daemon's own cgroup, outside the deny subtree, to a loopback listener the daemon holds on a port other than the answerer's, and that connect must succeed, or the probe is inconclusive, reports none and logs the cause; then a short-lived probe child placed in a leaf of the deny subtree connect()s to the same listener, once per family the table is inet over: 127.0.0.1 always, and ::1 wherever IPv6 loopback is enabled (the guest boots with ipv6.disable=1, so it probes IPv4 only). A family reads rejected only on EHOSTUNREACH, EACCES or EPERM within a short deadline (the design §5.7 active reject: icmpx admin-prohibited surfaces as EHOSTUNREACH on IPv4, ICMP code 13, and EACCES on IPv6, ICMPv6 code 1; EPERM is an output-hook verdict); any other errno, ECONNREFUSED and ENETUNREACH included, a success or a timeout reads none, and any probed family that does not read rejected makes the whole probe none, because a v6 bypass is a full bypass. A probe that reads none refuses a deny-all box again. The native decision is T48's (#1805) and is not changed here
- crates/minimald/src/main.rs: runs the guest load between entering the daemon's leaf and the start-time decide and record, and holds the loopback listener the effect probe connects to, so host_ip_enforcement flips to per_box, and the guest's refusal of a deny-all host-address box lifts, only through a decision that a successful load, a listed table and a fast-rejected probe produce
- crates/minimald/src/net/provider.rs: in a guest that decides per box, a deny-all host-address box resolves through the guest's answerer at 127.0.0.1, the one carve-out the loaded table admits by address and port: its dstnat rule retargets that box's udp 53 to 127.0.0.1 onto the answerer's port, scoped to the deny subtree, never loopback-wide and never the node plane's DNS; a guest that does not decide keeps the node's DNS layer
- scripts/install-host-classifier.sh: the install mode's load pipes render_ruleset's output straight into nft -f - instead of through a temp file and prints the sha256 of exactly the bytes it piped, so both lanes load, and digest, one render function's bytes; no rule text changes
test: cargo nextest run -p minimald guest_table_is_the_installers_rendered_ruleset
test: cargo nextest run -p minimald ruleset_digest_covers_bytes_piped_to_nft_in_both_lanes
test: cargo nextest run -p minimald guest_installer_runs_with_cleared_env_and_argv_params
test: cargo nextest run -p minimald guest_effect_probe_decides_per_box
test: cargo nextest run -p minimald guest_effect_probe_inconclusive_without_live_listener
test: cargo nextest run -p minimald guest_render_takes_node_plane_and_cohort_separately
test: cargo nextest run -p minimald guest_marker_written_only_after_the_table_loads
test: cargo nextest run -p minimald guest_load_failure_keeps_deny_all_refused
test: cargo nextest run -p minimald guest_decide_rechecks_table_not_only_marker
test: cargo nextest run -p minimald guest_dstnat_scoped_to_deny_subtree
test: cargo nextest run -p minimald guest_deny_all_box_resolves_through_answerer_once_decided
- vmhost:
- crates/minvmd/tests/guest_classifier_integration.rs (new): boots a VM through the supervisor path; asserts the guest's nft -c -f accepted the rendered text against the guest kernel and fails with nft's own error naming the missing expression when it did not; asserts the digest of the bytes the guest wrote to nft's stdin equals the digest the native installer's load path prints for the same parameters, and the sha256 of its --print-ruleset output; asserts inet minimal_class is listed in the guest; starts a deny-all host-address box whose outbound probe to an address off the guest is refused, whose lookup of its own box name answers through 127.0.0.1:7656, and whose session record shows host_ip_enforcement per_box; records the effect probe's observed errno per probed family and asserts it is in EHOSTUNREACH, EACCES and EPERM, expecting EHOSTUNREACH on IPv4 and, where IPv6 loopback is enabled, EACCES on IPv6
test (deferred: vm): cargo nextest run -p minvmd guest_kernel_accepts_rendered_ruleset
test (deferred: vm): cargo nextest run -p minvmd guest_loaded_ruleset_matches_native_print_ruleset
test (deferred: vm): cargo nextest run -p minvmd guest_classifier_table_present
test (deferred: vm): cargo nextest run -p minvmd guest_deny_all_probe_refused
test (deferred: vm): cargo nextest run -p minvmd guest_deny_all_resolver_reachable
test (deferred: vm): cargo nextest run -p minvmd guest_deny_all_box_answers_inbound_through_the_proxy
test (deferred: vm): cargo nextest run -p minvmd guest_host_ip_enforcement_per_box_after_load
test (deferred: vm): cargo nextest run -p minvmd guest_effect_probe_errno_per_family
produces: none
consumes:
- crates/minimald/src/net/classifier.rs
- crates/minimald/src/session_host.rs
- crates/minimald/src/net/answerer.rs
proof: On a VM-backed host, the guest kernel accepts and loads the classifier table the native installer renders, byte for byte the installer's own render for the same parameters, the digest taken over the exact bytes written to nft's stdin in both the guest and the native installer's load, never a re-render; a deny-all host-address box's outbound probe is refused while its resolver at 127.0.0.1:7656 still answers, and its session shows host_ip_enforcement per_box; per_box rests on the daemon's own control connect to a loopback listener it holds succeeding and, immediately after, a deny-subtree probe child's connect to that listener failing fast with EHOSTUNREACH, EACCES or EPERM in every family probed (IPv4 in the guest, ::1 too wherever IPv6 loopback is enabled); a guest whose check or load failed, whose table is gone behind its marker, whose control leg fails, or whose probe connects, times out or sees any other errno reports none, refuses the deny-all box again and says why in the bundle.
depends on:
- blocked by #1631
- waits on a gominimal/pkgs issue (to be filed): guest kernel nft expressions + nftables apk in microvm-rootfs
- assumes the node-plane and cohort identities collapse to the guest's one address only on an un-enrolled host (local-minvmd); an enrolled guest passes the design §4.2 addresses as the two inputs
- assumes the guest already meets the inescapability requirement: each box's cgroup-namespace root is its own leaf, no box writes cgroupfs, no box holds CAP_NET_ADMIN, and nft is root-owned on the read-only rootfs while the installer text never touches a file
Design ruling
Filed from the architecture review of #1805 (T48, #1631). On a VM-backed host a deny-all host-address box is not enforced: nothing loads the
inet minimal_classtable in the guest kernel, while on native Linuxscripts/install-host-classifier.shloads it. T48's interim stands as it is: on a VM-backed host a deny-all host-address box is refused, and every other host-address box runs withhost_ip_enforcementnone, surfaced at every session start. #1806 (T73) only surfaces that verdict, so the fix is this task.The permanent fix loads the same ruleset text inside the guest kernel, from one definition: the installer's own
--print-rulesetoutput, never a second copy.nftreaches the guest as Alpine'snftablesapk in themicrovm-rootfsclosure, pinned by that closure's per-apk digests, rather than as a static binary in the initramfs. The installer reaches the guest compiled into minimald from the one script file and piped tobash -s, so no installer file is staged. A netlink nftables writer inside minimald is the fallback, taken only if the image size or a no-shell-out policy rules thenftroute out. Enforcement in the host's egress gate is ruled out: past the VM's SNAT every host-address box shares the one guest source address, so the host cannot tell them apart.Conditions on the design:
per_boxonly on effect, at start and before each host-address launch.noneand logs the cause.inet. The guest boots withipv6.disable=1, so it probes IPv4 only. Wherever IPv6 loopback is enabled,::1is probed too.icmpx admin-prohibitedsurfaces as EHOSTUNREACH on IPv4 (ICMP code 13) and EACCES on IPv6 (ICMPv6 code 1). EPERM is an output-hook verdict.none. A family that connects makes the whole probenone, because a v6 bypass is a full bypass.nonerefuses a deny-all box again.deny_outchain must also be listed bynft list. The presence marker is only the fact that the load ran./bin/bash,/usr/sbin/nft) with its environment cleared: noBASH_ENV, noENV, no inheritedPATH. Every parameter is passed as argv, never interpolated into the script text.nftis root-owned on the read-only rootfs, and no box holdsCAP_NET_ADMIN. The guest already meets the inescapability requirement: each box's cgroup-namespace root is its own leaf, and no box writes cgroupfs.127.0.0.1:7656, never loopback-wide. The dstnat rule matches only deny-subtree sockets sending to127.0.0.1on udp 53. It never catches loopback-wide :53 and never touches the node plane's DNS.Acceptance:
nft -c -f, and a missing kernel option fails loudly;--print-rulesetoutput for the same parameters;host_ip_enforcementflips toper_boxonly after the load succeeds;This task waits on a gominimal/pkgs change that is not filed yet. The guest kernel has to build in
NFT_SOCKET, andmicrovm-rootfshas to carry thenftablesapk closure.locked_committhen moves to a commit that carries both.Out of time
The coder ran out of its 7200s before it finished. If it left work uncommitted, its last commit is the runtime's snapshot of that work as it stood — the commit message says so — and not a finished step: it may not build or pass its tests. So this PR is a draft.
Proof
On a VM-backed host, the guest kernel accepts and loads the classifier table the native installer renders, byte for byte the installer's own render for the same parameters, the digest taken over the exact bytes written to nft's stdin in both the guest and the native installer's load, never a re-render; a deny-all host-address box's outbound probe is refused while its resolver at 127.0.0.1:7656 still answers, and its session shows host_ip_enforcement per_box; per_box rests on the daemon's own control connect to a loopback listener it holds succeeding and, immediately after, a deny-subtree probe child's connect to that listener failing fast with EHOSTUNREACH, EACCES or EPERM in every family probed (IPv4 in the guest, ::1 too wherever IPv6 loopback is enabled); a guest whose check or load failed, whose table is gone behind its marker, whose control leg fails, or whose probe connects, times out or sees any other errno reports none, refuses the deny-all box again and says why in the bundle.
Commit headers reworded by the runtime to fit commitlint. The header the coder wrote is the first paragraph of each of these commits' bodies:
feat(minimald): the coder's uncommitted work as it stood when its 7200s ran outTests
cargo nextest run -p minimald guest_table_is_the_installers_rendered_rulesetcargo nextest run -p minimald ruleset_digest_covers_bytes_piped_to_nft_in_both_lanescargo nextest run -p minimald guest_installer_runs_with_cleared_env_and_argv_paramscargo nextest run -p minimald guest_effect_probe_decides_per_boxcargo nextest run -p minimald guest_effect_probe_inconclusive_without_live_listenercargo nextest run -p minimald guest_render_takes_node_plane_and_cohort_separatelycargo nextest run -p minimald guest_marker_written_only_after_the_table_loadscargo nextest run -p minimald guest_load_failure_keeps_deny_all_refusedcargo nextest run -p minimald guest_decide_rechecks_table_not_only_markercargo nextest run -p minimald guest_dstnat_scoped_to_deny_subtreecargo nextest run -p minimald guest_deny_all_box_resolves_through_answerer_once_decidedVerify
min task run agent-verifyexited 101, so this PR is a draft.Base
The base branch moved while the coder worked and was merged in before the tests ran. These files changed on BOTH sides since the branch point — read them first, because a rewrite from a stale reading can undo a merge without a conflict:
Cargo.lockDeferred
The plan marked these test lines deferred: the executor lacks the lane, so nothing here ran them and they still owe evidence.
cargo nextest run -p minvmd guest_kernel_accepts_rendered_rulesetcargo nextest run -p minvmd guest_loaded_ruleset_matches_native_print_rulesetcargo nextest run -p minvmd guest_classifier_table_presentcargo nextest run -p minvmd guest_deny_all_probe_refusedcargo nextest run -p minvmd guest_deny_all_resolver_reachablecargo nextest run -p minvmd guest_deny_all_box_answers_inbound_through_the_proxycargo nextest run -p minvmd guest_host_ip_enforcement_per_box_after_loadcargo nextest run -p minvmd guest_effect_probe_errno_per_familyOut of scope
These files are outside the task's
layers:, so this PR is a draft:crates/minimald/Cargo.tomlCloses #1812
Amended scope (design ruling on this PR)
VM-backed host: no resolver carve-out. The guest's node DNS layer (gvproxy at the gateway) forwards every non-zone name upstream, applies no per-box name rule to host-address boxes, and does not answer sibling host-address names. So the guest render (
--no-resolver-carve-out) carries no answerer accept and no dstnat chain, and logs at render: "deny-all host-address box on VM host: no resolver carve-out (follow-up gominimal/inbox#897)". A deny-all box in a deciding guest gets an empty resolv.conf. The KVM testguest_deny_all_dns_to_gateway_refusedasserts:The deny verdict is
reject with icmpx admin-prohibited.Known gap, filed as gominimal/inbox#897. That issue covers per-box name rules for host-address boxes in the guest's DNS layer, and restoring the resolver carve-out and its tests. It also records that an allow-list host-address box on a VM host resolves any name today.
Native host: the carve-out follows the live answerer.
server.rs, besideset_zone_answerer_port). The cell is cleared when the answerer's receive loop exits.--answerer-address/--answerer-port. Otherwise the box resolves through the live bind.Script layer: the native rule text is unchanged. The guest render omits the carve-out, as the amended ruling requires.
Behaviour change for existing native installs
A native host whose classifier was installed before this change has no carve-out record. Its deny-all host-address boxes are refused, as a stale carve-out, until the classifier is re-installed. The refusal prints the exact re-install command, with
--answerer-address/--answerer-portset to the live bind. Allow-list and default host-address boxes are unaffected. This belongs in the release notes for the release that ships it.Residual: the check runs at plan time only
A deny-all box that is already running keeps its carve-out if minimald's answerer stops while the daemon lives. If a foreign process then takes that loopback port, the box could reach it. The window is narrow: the answerer must die mid-session and the port must be taken before a restart. It remains an open residual, carried on T90 #1942: re-check or revoke the carve-out for live deny-all boxes when the live bind changes or clears.