Repository navigation
T93 — Give each VM a hostname-proxy port no other VM holds, and fail loudly when it cannot - #1957
Conversation
The node-port assignment now takes a host-wide reservation on the port it hands, before the boot uses it: an exclusive flock on a per-port file in a per-user directory ($XDG_RUNTIME_DIR/minimal/ports on Linux, falling back to /tmp/minimal-<uid>/ports, and the Application Support runtime dir on macOS; created 0700 and refused when another user owns it or it is wider). The supervisor holds the lock's file description for the VM's life, so the kernel releases it on stop, exit or crash — no pid files, no stale-entry cleanup, and the lock files themselves are never unlinked. Every candidate the draw considers is now checked and logged — one info line per skipped candidate naming the port and the reason (another VM's reservation, a port the node itself reserves such as the zone answerer's hook port, a bind refusal, or a listener already answering on loopback), and one per reserved port. The operator's pin is held to the same checks and never redraws: a pin some check refuses fails the start naming the port and the holder. Over the boot-marker channel — the one existing control path from inside a microVM to the VM host daemon — the guest's in-VM daemon now reports its hostname-proxy publish outcome: PROXY_SERVING on success, or a terminal PROXY_PORT_HELD when the publish is refused for address-in-use. The supervisor's gate owns the marker listener for its life, watches each boot's publish, re-probes the port itself, and decides: a drawn port redraws under the reservation and asks the guest to publish again, at most three tries, then fails the start naming the port and the holder; a configured port fails on the first refusal; nothing the guest says changes which port is reserved. The failure's cause rides the control socket's read-only status read (ZoneAnswererStatus::ProxyNotServing), for the CLI's min ls row and session-start message to name.
Inside a microVM, the hostname proxy publishes on the port the VM host handed down the boot line, and that port has no rung of its own to walk to: when the host already holds it, the daemon now says so over the boot-marker channel (vsock CID 2, the same channel the READY and MOUNT_FAILED beacons travel) as a two-line PROXY_PORT_HELD report naming the port, records the unavailable note `min ls` warns from, and ends the publish drive there. The VM host holds the port's reservation, so it owns the retry — a redraw onto a fresh port, or a start failure naming this one — and it cannot own what it never hears: a backoff loop here would leave the VM up with no hostname proxy. The healthy publication reports PROXY_SERVING the same way, so the host watch ends without waiting out its bound. Both reports are best-effort with short dial retries, mirroring guest.rs's emit_marker, because the host bounds its own watch and falls back to its own probes when nothing arrives; other publish failures — transient, or a walk that exhausted its rungs — keep the existing backoff (NET-021), and the native daemon's pinned-port bind retry keeps its NET-024 behavior untouched. The old pinned-refusal publish test asserted the retry-forever arm this removes for the VM path; it is rewritten into the terminal contract as proxy_publish_in_use_is_reported_not_retried, with a format test for the two report payloads beside it.
The NAME SURFACE row in `min ls` and the session-start message now say why the hostname proxy is not serving — the port the failure is about and the cause the VM host daemon reported: another process on the host holds the port, or the redraws ran out — instead of a bare "not serving". The cause rides the existing status reply the verbs already read: ProxyNotServing settles the surface verdict whole, with no detection, liveness query, or range probe, because the VM host daemon's verdict on the proxy's publication is the fact and no host read can move it. The session start reads the same status through host_answerer_read, which now carries the named cause out beside the answerer facts; the start's third arm prints it before the answerer arms, claiming nothing the status did not say. The ZONE ANSWERER row keeps its silence for a status that speaks only about the proxy, as it does for the pre-acquisition state. cmd/session.rs changes because the read's call site is there: the destructure grows the proxy_down fact and the start gains the arm that prints it. resolver.rs owns everything the fact is decided and spelled in, so the two verbs keep printing from the one function that cannot disagree with itself.
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
The assignment keeps its reservation only to hold the flock for the VM's life, so the field is `_reservation`; the reservation's own unread port field goes. The /tmp fallback base is created 0700 and owner-checked like the ports dir, since /tmp is shared.
The vsock dial to the VM host can take seconds per try, so awaiting it held the drive past the serving record and past the terminal return. Record first, then send the report on a spawned task.
Both activates run together and each VM must serve on its own port; a second bind of either published port must fail with address-in-use. A plain listener on 7654 then makes a drawn port redraw (the log names the skipped candidate) and makes a pinned port fail the start naming it.
A start that fails on the hostname proxy's port now fails with one line naming the port, the holder (pid and exe via procfs or lsof, when the host shows it) and why: configured ports never redraw, or the publish tries ran out. `run --detach` lifts that line from run.log, so the CLI's autospawn error carries it inline. The reservation directory is now a parameter of the draw, so every unit test reserves in its own tempdir and none takes a flock in the user's real directory. The supervisor resolves the directory once, as before.
The detach's stderr, which the CLI's autospawn error carries, must name the held port, its holder, and that configured ports never redraw.
The run reference names the hostname proxy's default port, the fallback when it is held, and the env knob that pins it from minvmd run or the min command that autospawns it.
|
Reviewed: 1 finding, none blocking at Address the comments on the diff. Clobber: none: |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
The held-port beats assumed stopping the default VM frees 7654. The named VM can hold it instead, and its proxy's TIME_WAIT connections refuse a plain bind for up to a minute on Linux and macOS. Wait until the port binds, and set SO_REUSEADDR on the probe and the holder; a live listener still refuses them.
… is late A publish watch that expired without the guest's report no longer maps to up blindly. The supervisor names the port's holder and matches its pid against the processes it spawned for the VM (the host switch and the VMM child): its own forwarder means the report was late and the VM is up; another process is treated like a refused publish (drawn ports redraw, pins fail naming the holder); a holder the host will not name gets a short grace for the report and then comes up unconfirmed, as does a port nothing answers on. An unconfirmed publish rides the status read as the new publish_unconfirmed cause, so min ls says so instead of serving, until the guest's late report clears it.
A refusal the guest reports after an unconfirmed start now records PortHeldAfterStart with the holder the host names, and `min ls` and the session start say the VM is up without a hostname proxy, naming the port and the holder, instead of the start failure's words. ProxyDownCause and the statuses that carry it lose Copy to hold the holder label. port_holder takes its procfs root or lsof binary through port_holder_with, so a missing or failing lookup is proven to name no holder (unknown, never foreign), and a child-process listener is proven to be named by pid on both Linux (procfs) and macOS (lsof).
With only the named VM up, `min --vm <vm> ls` prints the discovery row without a VM column, so the redraw beat read no port. Accept that form for the scoped listing.
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was a failed check suite). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
An unscoped listing's lone row belongs to whichever VM is up, so the stopped-VM beat read the default VM's port as the named VM's. Only the redraw beat, which lists with --vm, opts in.
The start failure reads "held by {holder}", and the fallback carried its own
verb, so an unnamed holder rendered "held by another process on the host holds
it".
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
The READY wait dropped every publish report as a killed boot's straggler, so a refusal this boot sent before READY never reached the publish decision. The supervisor now draws a publish generation per boot and hands it beside the proxy port on the boot line; the guest echoes it as a third line of every publish report. A report with this boot's generation is kept before or after READY and handed to the watch; any other generation is a straggler, dropped with a line naming both. A report with no generation comes from an older guest: dropped before READY, kept after it.
A publish report that lost its generation on the way is still kept after READY, as an older guest's, so a break in the command line to guest to report chain left the e2e green. The supervisor now logs the generation it draws for every boot, and each kept publish report logs the generation it carried (or absent) beside the boot's. The two-named-VMs proof reads both lines for each VM's boot and fails when either is missing, the report's generation is absent, or the two differ.
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 2 would exceed the cap of 1 (the last trigger was the runtime's review findings). A person decides now. Remove |
norrietaylor
left a comment
There was a problem hiding this comment.
Reviewed at 72092e7. Both of the T93 publish-generation gaps are closed: per-boot token, straggler drop, and a log line per decision, asserted in the e2e. The redraw now releases its own flock before redrawing (72092e7). The re-probe-gating finding is declined and ARCH concurs. ARCH PASS. All lanes green on head.
T93 — Give each VM a hostname-proxy port no other VM holds, and fail loudly when it cannot
slice: S15
kind: task
covers: none
size: M
est: 260 code, 300 tests
executor: agent
difficulty: needs care
model: strongest
diagnostics: the VM host daemon log names the proxy port it reserved, every candidate it skipped and why (another VM's reservation, a port the node reserves for itself, or bound on the host), and the publish outcome minvmd checked for that port; a failed start is one error line naming the port and the holder when known
observability: one info line per reserved node port; one info line per skipped candidate; one error line when the publish of the reserved port fails; the min ls NAME SURFACE row and the session-start message carry the named cause
layers:
- vmhost:
- crates/minvmd/src/cmd/run.rs:
assign_node_portreserves the drawn port host-wide before the boot uses it, with an OS advisory lock (flock/fcntl) on a per-port file that minvmd holds open for the VM's life, so the kernel releases it on stop, exit or crash; no pid files, no stale-entry cleanup, and minimal never unlinks a lock file; the directory is per user:$XDG_RUNTIME_DIR/minimal/ports/on Linux (falling back to/tmp/minimal-<uid>/ports/, created 0700 and refused if it exists with another owner or a wider mode) and~/Library/Application Support/minimal/run/ports/on macOS, never a cache or temp dir the OS purges; a candidate another VM has reserved, a port the node itself reserves (the answerer's hook port among them, whether or not it is bound at the probe) or a port bound on the published address 127.0.0.1 (with 0.0.0.0 probed as an extra holder check, the proxy never bound to the wildcard) is skipped and logged; another user's VMs are seen only by the bind probe, the residual the single-operator premise leaves- crates/minvmd/src/control.rs: over the existing control path, minvmd learns the guest's proxy publish outcome and re-probes the port itself, so nothing the guest says changes which port is reserved; for a port minvmd drew, an address-in-use publish redraws under the reservation and asks the guest to publish again, at most 3 tries (a constant), then fails the VM start naming the port and the holder; an operator-configured port never redraws and fails the start naming the port and the holder; a VM never stays up with no hostname proxy
test: cargo nextest run -p minvmd concurrent_boots_never_draw_the_same_proxy_port
test: cargo nextest run -p minvmd reservation_is_released_when_the_vm_stops
test: cargo nextest run -p minvmd publish_in_use_redraws_then_fails_named
test: cargo nextest run -p minvmd configured_proxy_port_in_use_fails_named_without_redraw
test: cargo nextest run -p minvmd draw_skips_the_answerer_hook_port_while_free
- daemon:
- crates/minimald/src/server.rs: an address-in-use failure of the hostname proxy's host publish is reported to the VM host daemon over the existing control path as a terminal publish failure naming the port, not retried with backoff forever; other publish errors keep the existing backoff
test: cargo nextest run -p minimald proxy_publish_in_use_is_reported_not_retried
- cli:
- crates/minimal/src/resolver.rs: the NAME SURFACE row in
min lsand the session-start message name the cause when the hostname proxy is not serving (the port and that another process holds it, or the redraws ran out), read from the VM host daemon's side, instead of a bare "not serving"test: cargo nextest run -p minimal name_surface_names_why_the_proxy_is_not_serving
- e2e:
- scripts/session-e2e.sh: the two-named-VM case also boots both VMs at once and asserts each VM's hostname proxy serves on its own port; a drawn port held by a non-minimal listener on 127.0.0.1 is redrawn (the VM serves on a port other than the held one and the log names the skipped port); a configured port held the same way fails the start naming the port; on macOS a second bind of the same address and port fails loudly
test: ./scripts/session-e2e.sh two_named_vms_on_one_machine
produces: none
consumes:
- crates/minvmd/src/cmd/run.rs
proof: Two VMs booting at the same moment get distinct hostname-proxy ports from a per-user reservation the kernel releases when a VM's daemon stops or dies; a drawn port found taken is redrawn at most three times and then fails the start naming the port, a configured port found taken fails the start naming the port without redraw, and no VM stays up with no hostname proxy; the draw never takes the answerer's hook port; min ls and the session-start message name the cause when the proxy is not serving.
Out of time
The coder ran out of its 7200s before it finished. If it left work uncommitted, its last commit is the runtime's snapshot of that work as it stood — the commit message says so — and not a finished step: it may not build or pass its tests. So this PR is a draft.
Proof
Two VMs booting at the same moment get distinct hostname-proxy ports from a per-user reservation the kernel releases when a VM's daemon stops or dies; a drawn port found taken is redrawn at most three times and then fails the start naming the port, a configured port found taken fails the start naming the port without redraw, and no VM stays up with no hostname proxy; the draw never takes the answerer's hook port; min ls and the session-start message name the cause when the proxy is not serving.
Tests
cargo nextest run -p minvmd concurrent_boots_never_draw_the_same_proxy_portcargo nextest run -p minvmd reservation_is_released_when_the_vm_stopscargo nextest run -p minvmd publish_in_use_redraws_then_fails_namedcargo nextest run -p minvmd configured_proxy_port_in_use_fails_named_without_redrawcargo nextest run -p minvmd draw_skips_the_answerer_hook_port_while_freecargo nextest run -p minimald proxy_publish_in_use_is_reported_not_retriedcargo nextest run -p minimal name_surface_names_why_the_proxy_is_not_serving./scripts/session-e2e.sh two_named_vms_on_one_machineVerify
min task run agent-verifyexited 101, so this PR is a draft.Out of scope
These files are outside the task's
layers:, so this PR is a draft:crates/minimal/src/cmd/session.rscrates/minimald-rpc/src/lib.rsCloses #1949