Repository navigation
fix(minimal,sandbox2): ship the classifier step from min finalize-install - #2144
Conversation
One privileged step for every host need (NET-122, #2127): the user-namespace profile, box names (resolver hook, minzoned, the reserved range on macOS), the classifier tree and KVM group membership, each probed on this host and carried in one script only while it is missing. `--show` names the items, `--show --script` prints the exact script a run executes, `--show --json` reports them under `min/v1/finalize-install`, and `--undo` removes what the step installed. Without a terminal the run goes through `sudo -n` or exits 1 with the file route. BREAKING CHANGE: `min net setup` is removed with no alias; run `min finalize-install` (`--show --script` replaces `--print`). Closes #2127
…inalize-install's exit rules A names item with no daemon port is a waiting item, not a host fact no script can change: it is listed as waiting on a daemon under the summary, the other items still run, and the host is not finished. The classifier item is omitted until the step that renders it exists (#2128), so a native Linux host can finish. A run that added the operator to the kvm group ends with the login note; --undo takes --show only with --script; the no-TTY pointer uses mktemp; the run's summary goes to stdout; a daemon path an AppArmor tunable cannot name blocks the userns item.
…tall The classifier's privileged step refused an install without the two source identities, every advisory told a person to curl the script from GitHub and fill in placeholders, and a daemon restart needed a fresh `sudo … --pid` to be placed in its leaf. The step now accepts neither identity (NET-078): an un-enrolled native host classifies the two identities as two cgroup matches and translates nothing, its postrouting chain present and empty, so a later association adds the reserved addresses without a reinstall. Exactly one identity is still refused. `min finalize-install` renders the classifier item from the bytes the binary carries — a root-owned copy of the step under /usr/local/lib/minimal — and installs a systemd path unit on the daemon's socket whose oneshot runs the step's new `--place-listener`, so every daemon start is placed by the manager. `--undo` removes the pair and the copy. Every remedy names `min finalize-install`; a convention test keeps placeholders and remote fetches out of the Rust tree.
|
Warning Review limit reached
This review includes 13 billable files and costs up to $3.25.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 11 minutes for your next included review. View limit details
Comment |
|
Reading This comment updates when the review finishes. |
mitodrummer
left a comment
There was a problem hiding this comment.
One finding sits outside the diff. stale_carve_out_refusal in crates/minimald/src/net/classifier.rs (~L2238) still builds its remedy as install_hint() plus --answerer-address A --answerer-port P. With this change that renders run: min finalize-install --answerer-address 127.0.0.1 --answerer-port 7700, and clap rejects it because FinalizeInstallArgs only takes --show/--script/--json/--undo. Even without the flags, the classifier step min finalize-install renders passes no answerer flags, and the item reads Done while the marker exists (see the comment on L798), so re-running the verb can't re-render the carve-out. The remedy needs a path that actually reaches the step with the live bind, or finalize-install needs to carry the answerer bind itself.
| operator: &str, | ||
| socket: &str, | ||
| ) -> Item { | ||
| if marker_present { |
There was a problem hiding this comment.
The item is Done as soon as the marker exists, and only Missing items run. Three cases miss out: (1) Cause::TableNotEffective (marker present, probe doesn't see the table) now prescribes run: min finalize-install, which reports Done and does nothing. (2) A host installed the old way (curl + --pid) has the marker, so it never gets the root-owned copy or the minimald-place units, and every daemon restart still runs unplaced. (3) After an upgrade, /usr/local/lib/minimal/install-host-classifier.sh is never refreshed: the path unit keeps running the old copy while --undo uses the new embedded bytes. Suggest gating Done on the marker plus the units plus a copy whose bytes match CLASSIFIER_SCRIPT, and giving TableNotEffective a remedy that re-runs the step.
There was a problem hiding this comment.
Fixed in 5a410b6 and 568ddc4 (head 5373cef). classifier_item_over now decides over a ClassifierFacts struct and reads Done only when the table is present the way minimald reads it (table_present: the marker plus exactly one well-formed ct-mark-mask-0x… record, per your #2142 thread; a marker alone is a step that did not finish), this build's copy is there (byte compare) and, where systemctl exists, the three units plus their multi-user.target.wants links are all present. So (2) an old-style curl + --pid install (marker, no copy) and (3) a stale copy after an upgrade are both Missing again, and the same block refreshes everything; pinned by the stale and unitless rows of classifier_item_table and by classifier_table_presence_needs_the_marker_and_one_mask_record.
(1) Cause::TableNotEffective (marker and record present, probe does not see the table) still points at min finalize-install, and with the table, copy and units present that reads Done. Deciding otherwise needs the table itself read, which is root's (nft list), and min finalize-install runs unprivileged until its one sudo. The boot oneshot added for the thread below shrinks the main way to get there (a reboot losing the table while the marker survives would now re-run the step at boot). Making the item probe the table from inside the elevated block is a follow-up I would rather not fold in here; happy to file it if you want it tracked.
| if operator.is_empty() || operator.contains(['\'', '\n', ' ']) { | ||
| "this process's user name did not read, so there is no account to \ | ||
| delegate the classifier tree to" | ||
| } else if socket.is_empty() || socket.contains(['\'', '\n']) { |
There was a problem hiding this comment.
The socket path is only checked for ' and newline. A space breaks --place-listener: /proc/net/unix splits on whitespace, so awk '$8 == p' never matches and the unit logs "nobody holds" and exits 0, silently placing nothing. awk -v also processes backslash escapes in p, and systemd expands % specifiers in ExecStart= (and in PathChanged=). A --minimal-dir under a home like /home/a b/ hits the first case. Suggest an allowlist ([A-Za-z0-9._/-]) and a Cannot cause for anything else, the same treatment the operator name gets.
There was a problem hiding this comment.
Fixed in 5a410b6 (head 5373cef): allowlists as suggested, [A-Za-z0-9._-] for the operator and [A-Za-z0-9._/-] for the socket path, each with a Cannot cause naming the set and what reads the name (shell word, unit file, awk -v, the /proc/net/unix field). The denylist constant is gone. Table rows cover space, tab, ', ", %, $ and \ for both.
| # bytes this binary carries, with no source identity (an un-enrolled host\n\ | ||
| # translates nothing; an association adds the reserved addresses later).\n\ | ||
| mkdir -p {ANSWERER_PROGRAM_DIR}\n\ | ||
| cat > {CLASSIFIER_PROGRAM_PATH} <<\\{CLASSIFIER_SCRIPT_HEREDOC}\n\ |
There was a problem hiding this comment.
Nit: cat > truncates and rewrites the copy in place. If minimald-place.service is already installed and fires during this block (a daemon restart at the same moment), bash, which reads its script incrementally, can execute a half-written file. The window is narrow, but writing to a temp file in the same directory, then chmod and mv -f over the target, closes it.
| Description=Watch minimald's socket to place its listener\n\ | ||
| \n\ | ||
| [Path]\n\ | ||
| PathChanged={socket}\n\ |
There was a problem hiding this comment.
The PR notes the path unit was never exercised on a real systemd host. Does PathChanged= reliably fire on the daemon's unlink + bind() of the socket? It should, through the inotify watch on the parent directory, but it is the one mechanism this PR depends on and it has no test. Two more questions: does it fire when the home directory is mounted after pid 1 sets the watch (systemd-homed, encrypted home)? And would it be more robust to have the daemon call the placement itself (an ExecStartPost= in a user unit, or a polkit-gated helper)?
There was a problem hiding this comment.
Rehearsed on the lima guest's real systemd with a throwaway path unit + oneshot pair (not the shipped block) on a socket path whose parent directories did not exist at watch time: the path unit started active with the parents absent, fired on the first bind() after the parents were created, and fired again on each unlink + re-bind() (3 of 3 events, each logged with the socket present), and stayed active, not failed. So the inotify watch on the nearest existing ancestor does walk down as directories appear, and unlink + bind is seen as a change each time.
Not covered: a home mounted after pid 1 set the watch (systemd-homed, encrypted home). inotify does not follow mounts, so a watch set on the pre-mount directory would miss it; I have not tested that and the PR body says so. On the alternative: having the daemon place itself needs either a user unit with an ExecStartPost= that can write a root-owned cgroup.procs (it cannot; the daemon leaf is root's by design, so a box cannot write it either) or a polkit-gated setuid helper, which is a bigger surface than a root oneshot reacting to the socket. I would keep the path unit and treat the mount-after-watch case as a follow-up (a PathExists= on the home mount that re-arms the watch, or the daemon's socket under /run where no late mount sits). Head 5373cef.
| Unit={PLACE_SYSTEMD_UNIT}.service\n\ | ||
| \n\ | ||
| [Install]\n\ | ||
| WantedBy=multi-user.target\n\ |
There was a problem hiding this comment.
WantedBy=multi-user.target brings the path unit back at boot, but nothing recreates /sys/fs/cgroup/minimald.slice or the nft table after a reboot: the tree lives in cgroupfs and the step installs no boot unit for it. So on every boot the first socket event runs --place-listener, which dies with "…is not installed" and leaves a failed unit, and boxes run unenforced until someone re-runs min finalize-install. The tree not surviving reboot predates this PR, but this unit now advertises boot persistence. Consider a oneshot minimald-classifier.service (WantedBy=multi-user.target) that re-runs the step from the root-owned copy, ordered before the path unit.
There was a problem hiding this comment.
Fixed in 5a410b6 (head 5373cef): the block now installs a oneshot minimald-classifier.service (WantedBy=multi-user.target, Before=minimald-place.path, ExecStart=<copy> --user <operator>) that re-runs the step from the root-owned copy at boot; the path unit carries After=minimald-classifier.service. --undo disables and removes it with the pair, and classifier_item_on_this_host counts it among the units Done requires. Pinned in classifier_step_places_the_daemon_on_every_start and the two undo tests. The reboot itself was not rehearsed (noted under Unverified).
| && sudo bash ./install-host-classifier.sh --user {account} \ | ||
| --cohort-address <cohort address> --node-plane-address <node-plane address>" | ||
| ) | ||
| "run: min finalize-install".to_string() |
There was a problem hiding this comment.
Dropping own_account() moves the account choice from the daemon (which knows its uid) to whoever runs the hint. min finalize-install takes the operator from $USER/$LOGNAME and the socket from the invoker's minimal dir. The old hint carried sudo, so a person who runs sudo min finalize-install out of habit installs --user 'root' and watches root's socket. No error results, but the daemon stays unplaced. Suggest refusing to run as euid 0 (or reading SUDO_USER), or keeping the account in the hint (run: min finalize-install # as <account>).
There was a problem hiding this comment.
Fixed in 5a410b6 (head 5373cef): the item refuses euid 0 with a Cannot cause ("this command runs as root, but the classifier is installed for the account whose daemon it places ... run it as that account (it asks for sudo itself)"), checked before anything reads Done or Missing, so sudo min finalize-install cannot install for root and watch root's socket. I preferred the refusal over reading SUDO_USER: the socket path comes from the invoker's minimal dir too, and SUDO_USER gives the account but not the dir. Rows for root over an installed and a bare host in classifier_item_table.
| return 0 | ||
| fi | ||
| placed=0 | ||
| for dir in "$PROC"/[0-9]*; do |
There was a problem hiding this comment.
Nit: this forks a readlink for every fd of every process on the host before the uid filter, and runs on every socket event. On a busy host that is thousands of forks per daemon start. Reading the Uid: from $dir/status first and skipping other accounts' processes avoids almost all of them, and the uid check stays as the guard.
There was a problem hiding this comment.
Fixed in 273cdfe (head 5373cef), by a different route than the uid-first filter: listener_holders runs one find <proc>/[0-9]*/fd -lname "socket:\[<inode>\]" per listener inode, so the fd tables are walked by a single find with no fork per descriptor, and the uid is read from status only for the handful of pids that hold the inode. The uid check stays as the guard. Rehearsed over the fake /proc in install_test.sh.
… copy current The --place-listener step moved every same-uid holder of the listener's path; a box running as that account could bind a socket spelled like it and be lifted out of its deny leaf. It now reads each holder's cgroup and never moves a process already under the slice, nor one whose cgroup it cannot read; the fd scan is one find per inode, a vanished holder is a note rather than a failure, and a listed socket with no holder is looked for once more before it reads as unheld. min finalize-install reads the classifier item as missing while the root-owned copy of the step differs from this binary's bytes, as another account's install when the tree is delegated elsewhere, and as blocked when the operator name or socket path carries a character the shell word or the unit file cannot take (quote, whitespace, %, $, backslash). The unit pair is skipped with a note where there is no systemctl. The install hint is the bare verb, which the advisory sentence formats. The daemon pins that a launch before the placement never reads per_box, and the native e2e lane places its daemon through --place-listener.
std::process::exit runs no destructor and a piped stdout is block-buffered, so `min finalize-install --show --json > report.json` on an unfinished host wrote an empty file. Every exit in the command now flushes both streams first, and a CLI test runs the binary with stdout piped and asserts the whole report arrives with exit 1.
… keeps a foreign profile
…e, stock @{HOME} scope
…failure to the end
…t its names The classifier item reads done only when the marker, this build's copy and (where systemd is there) its enabled units are all present, so a run that died at systemctl, an old-style install or a stale copy is missing again and the same block refreshes it. The copy is written beside its target and moved over it. A boot oneshot re-installs the tree and the table after a reboot, ordered before the placement path unit, and --undo removes it. Running as root is refused: the item is the daemon account's. The operator name and socket path are allowlisted instead of denylisted, and the step's own die messages name the installed verb, which the remedy gate now checks in scripts too.
It names the retired spellings to assert their absence from the step's output, which is the opposite of carrying one.
…g lines when both apply
…holds it The closing line fired whenever nothing was moved, including when every holder was deliberately left where it stands, and claimed the socket was unheld. It now distinguishes an unheld socket from holders left alone.
… undo only its own tree Done needs the marker and exactly one well-formed ct-mark-mask record beside it (a marker alone is a step that did not finish), matching the daemon's table_marker_present. The step leaves an ownership record, and --undo uninstalls the tree only by that record; a tree installed another way stays with a note that says how to remove it.
The install refuses exactly one source identity, so a hint that kept it named a command that dies. The hint now drops the lone identity and keeps the account and root asked about.
…rded member on --undo One operator per line, appended only when absent, so a second user's run no longer overwrites the first's name, and --undo (and the installer's manual remedy) walk every line instead of a single cat. Also renames the installer's test-only root override from MINIMAL_OVERRIDE_NET_SETUP_ROOT to MINIMAL_OVERRIDE_FINALIZE_INSTALL_ROOT to match the command it serves.
|
Thanks, the inline points are all addressed or answered. One point from the review body is still open, and it was easy to miss without a thread: |
… its detection maybe_remove_finalize_install detected the classifier tree under $_fi_root but printed a remedy against the bare $_cls_root; every other artifact roots both. The remedy now quotes "$_fi_root$_cls_root", and the installer test asserts it names the seeded root.
mitodrummer
left a comment
There was a problem hiding this comment.
Our review comments are addressed; thanks.
Root cause
The classifier's privileged step (
scripts/install-host-classifier.sh) refused to install unless both egress source identities were given, so an un-enrolled native host could never install it. Every remedy the daemon, the sandbox crate and the CLI printed told a person tocurlthe script fromraw.githubusercontent.comand fill in<cohort address>/<node-plane address>placeholders by hand. And because the daemon runs from the operator's session scope, each daemon restart needed a freshsudo … --pidto be placed in its leaf, or its boxes ran unenforced.Fix
require_identitiesdies only when exactly one is given. Without a pair the render keeps the two identities as two distinct classifier matches (the boxes cgroup subtree = host-address cohort; the slice outside it = node plane) and translates nothing: thepostroutingchain is present and empty, so a later association adds the reserved SNAT addresses to the same matches without a reinstall. The VM-backed path is untouched; given a pair, the SNAT rules render as before.min finalize-installrenders the classifier item on a native Linux host from the bytes the binary already carries (include_str!): the block writes a root-owned copy to/usr/local/lib/minimal/install-host-classifier.sh(beside the target, thenmv -fover it, so a unit firing mid-write never runs half a script) and runs it with--user <operator>and no identities. No remote fetch anywhere. The item reads Done only when the table is present the way minimald reads it (the marker plus exactly one well-formedct-mark-mask-0x…record; a marker alone is a step that did not finish), this build's copy is there, and (wheresystemctlexists) the enabled units are all present; it reads as missing again when the copy's bytes differ from this build's (an upgrade, or an old-stylecurl+--pidinstall that has the marker and no copy), or when a run died atsystemctlafter the tree was written. It reads as blocked when the tree is delegated to another account, when the command runs as root (sudo min finalize-installwould install for root and watch root's socket), and when the operator name or socket path falls outside the allowlists ([A-Za-z0-9._-]and[A-Za-z0-9._/-]) the shell word, the unit file,awk -vand the/proc/net/unixfield lookup can all take.minimald-classifier.service(WantedBy=multi-user.target,Before=minimald-place.path) that re-runs the step from the root-owned copy, since the tree lives in cgroupfs and the table in the kernel and neither outlives a boot. It also installsminimald-place.path(PathChanged=on the native daemon socket,WantedBy=multi-user.target) and a oneshotminimald-place.servicethat runs the step's new--place-listener SOCKmode. The mode finds the holders of the socket's inode (onefind -lnameper inode, owner-uid checked) and writes each into the daemon leaf — but never a process already anywhere under the slice, boxes subtree included, nor one whose cgroup cannot be read (design §4.1: a box bound to a socket spelled like the listener stays in its deny leaf). A vanished holder is a note, not a failure; a listed socket with no holder is scanned once more after a second. The mode's closing line distinguishes an unheld socket from holders it left alone. The block leaves an ownership record (/var/lib/minimal/finalize-install-classifier, the shape of the base's AppArmor record);--undodisables and removes the three units and the copy, and uninstalls the tree only by that record — a tree installed another way stays with a note saying how to remove it, and a held tree keeps its record for the next--undo. Where there is nosystemctlthe block skips the units with a note.min finalize-install.sandbox2::classifier::install_hint()is the baremin finalize-install(PR feat(minimal,minimald): one host line at start, classifier advisory only on declared egress #2145 formats it into its sentence); the old placeholder/curl wording is gone from the daemon, the rpc crate, the CLI fixtures and the e2e assertions. Advisory gating/wording stays with Session start: one host line; classifier advisory only when the box declares egress #2131.Behaviour notes
per_box. NET-080's record reads the daemon's leaf live (daemon_fetch_leaf), so a fetch in that window names no node-plane leaf either. Pinned bylaunch_before_placement_reads_not_per_box.--minimal-dirmin finalize-installran with (<state>/providers/local-minimald0/ssh.sock); a daemon autospawned under another--minimal-diris never placed by it.association_applies_reserved_snat_without_classifier_reinstallproves the render with a pair is the un-enrolled render plus two postrouting rules, not the act of loading one over the other.systemctl reset-failed minimald-place.path(said in the block's comment).PathChanged=on a unix socket path, rehearsed on the lima guest's real systemd (a throwaway path+oneshot pair, not the shipped block): the path unit activates with the socket's parent directories absent, fires on the firstbind()after they are created, and fires again on each unlink + re-bind()(3 of 3), stayingactive. A home mounted after pid 1 set the watch (systemd-homed, encrypted home) is not covered by inotify and was not tested; see the thread reply for the self-placement alternative.Acceptance
Spec tests (NET-078 verify names):
minimald net::classifier::tests::unenrolled_native_identities_are_cgroup_matches_without_snatminimald net::classifier::tests::association_applies_reserved_snat_without_classifier_reinstallAdditional tests:
--checkhint has no placeholders and drops a lone identityscripts/install_test.shcasehost_classifier_tree_installed(half_identity,unenrolled_print,unenrolled,--checkhint)--place-listener: same-uid holder outside the slice placed; other uid, other socket, same-uid holder insideboxes/deny, unreadable cgroup left alone; idempotent; unbound socket and vanished holder (re-scan) exit 0install_test.shrehearsal over a fake/proc(MINIMAL_OVERRIDE_PROC)per_box; fetch in the window names no leafminimald net::classifier::tests::launch_before_placement_reads_not_per_box--userrun, boot oneshot + path unit + placement oneshot with their ordering,--place-listener, systemd guard, no--pid/identities/fetchminimal cmd::finalize_install::tests::classifier_step_places_the_daemon_on_every_startminimal cmd::finalize_install::tests::classifier_table_presence_needs_the_marker_and_one_mask_recordsystemctl) / Cannot (mount, operator or socket outside the allowlists incl. space, tab,%,$,\,", another account's tree, running as root) / Missing (no marker, stale copy, units not installed; one block for all three)minimal cmd::finalize_install::tests::classifier_item_table--place-listenerclosing line: holders left alone are said so, an unheld socket is said soinstall_test.shrehearsal (place_listener_skipped,place_listener_nobody)--undoremoves the three units and the copy, succeeds on a clean host; uninstalls the tree only by its record (unowned tree stays with a note; stale record cleared); a held tree keeps its recordminimal resolver::tests::finalize_install_undo_removes_what_the_step_installs,finalize_install_undo_succeeds_on_a_clean_host,finalize_install_undo_removes_only_the_classifier_tree_it_recorded,finalize_install_undo_removes_the_other_items_before_a_held_classifier_treesudosandbox2 the_install_hint_names_the_installed_verb;minimaldcommand/advisory tests;minimald-rpcadvisory fixture<cohort address>,<node-plane address>,<the account minimald runs as>or araw.githubusercontent.com/gominimal/minimalfetchcommon::remedy_strings::no_rust_remedy_carries_a_placeholder_or_a_remote_fetch(convention-discovered overcrates/**/*.rsandscripts/*.sh, sparing the rehearsal that asserts their absence; on macOS it runs only viajust test-cross)per_boxscripts/session-e2e.shnative lane (--place-listenerreplaces--pidin the HIDA case)Test plan
Run in the lima guest at head
274c40b16(origin/mainmerged in after #2142 squash-merged; the three files that conflicted —finalize_install.rs,resolver.rs,cli-min.md— were taken from main and this PR's own diff re-applied on top, so the diff againstorigin/mainlists only this PR's files):cargo nextest run --locked --no-fail-fast -p minimal -p common -p minimald -p sandbox2(crates rebuilt clean) →Summary [ 35.972s] 1589 tests run: 1589 passed (1 leaky), 18 skippedcargo clippy --locked --workspace --all-targets -- -D warnings→Finished, no warningscargo fmt --all -- --check→ cleanjust test-installer(full suite) →# 580 passed, 0 failed(sh and dash)Natively (macOS):
just test-installer host_classifier_tree_installed→# 182 passed, 0 failed;just lint-shell→52 script(s) passed shellcheck;cargo fmt --all -- --checkclean;just lint-prose docs/reference/cli-min.mdclean on the edited lines.Unverified:
PathChanged=mechanism was rehearsed on the lima guest's real systemd (above), but the native e2e lane runs the repo script with identities andsudo -n, not the installedmin finalize-installblock, so the lane proves--place-listener(daemon found by its socket, pid in the daemon leaf, next launchper_box), not the installed units.scripts/session-e2e.shwas not run here; a reboot was not rehearsed.just clippy-strict.Related
Was stacked on #2142 (
feat/2127-finalize-install), which has merged; this PR now targetsmainwithorigin/mainmerged in. Advisory wording and gating: #2131.PR #2145 (
feat/2131-one-host-line) also removesown_account()and makesinstall_hint()the bare verb; this PR adopts the same string. #2145 lands after this one and reconciles onto it: itshost_install_for_this_host()callsothers.extend(linux::classifier_item_on_this_host())against the oldOption<Item>signature; with this PR's-> Itemit must becomeothers.push(linux::classifier_item_on_this_host(global)).Closes #2128