Repository navigation
fix(minimald,minimal): a cleanup destroy keeps a session that finished activating - #2233
Conversation
|
Reviewed: no findings; approval withheld, 4 threads unresolved at APPROVE withheld: 4 unresolved threads — norrietaylor crates/minimal/src/cmd/session.rs, norrietaylor crates/minimal/src/task.rs, norrietaylor crates/minimal/src/cmd/mod.rs, norrietaylor crates/minimald/src/box_manager.rs:1645. Resolving the last thread releases the approval on this head. Resolve the open threads. Resolving the last one releases the approval on this head; no new review is required. Clobber: none seen. |
There was a problem hiding this comment.
correctness
minimal-review — advisory review
Verdict COMMENT — 1 finding, 1 inline.
security
minimal-review — approved
Verdict APPROVE — No findings.
supply-chain
minimal-review — approved
Verdict APPROVE — No findings.
Panel: 3 of 3 personas ran (correctness, security, supply-chain).
norrietaylor
left a comment
There was a problem hiding this comment.
Review at head 64a2a38. The daemon-side rule is correct. DestroyIfUnfinalized is decided in mailbox order. A failed activation is still destroyed. An explicit min session destroy stays unconditional. An old daemon ignores the new field.
The client side has two problems that need a fix before merge: the registration row (comment on cmd/session.rs) and task boxes (comment on task.rs). The other two comments should be fixed or accepted on purpose.
Test gap: a_cleanup_destroy_keeps_a_finished_session checks an idle Active session and a Pending one. It does not queue a cleanup behind a finalize that is still running, so it does not reproduce the race this PR fixes. A slow finalize, or a park message like the one in #2231, can make that case deterministic. No test covers the client's kept handling, the reap path, or the lease.
| .and_then(|registration| registration.box_id), | ||
| ) | ||
| .await; | ||
| let kept = best_effort_destroy(&mut client, id, true).await; |
There was a problem hiding this comment.
A kept session has already lost its box registration row.
The lease moves into finalize_holding_lease above. When the finalize fails, that function drops the lease without a commit. Per the BoxLease doc, the VM host daemon withdraws the row when that connection closes. That happens before this line runs, so if !kept { withdraw_box_row(..) } protects nothing.
Failure case, on any host that uses the VM host daemon (the default on macOS):
- The activation registers and holds the lease.
- The finalize takes longer than the client deadline. The client drops the lease, and the host withdraws the row.
- The cleanup destroy keeps the session, which is now
Active, and the client says it is live.
Result: a running box with no published row. Its name does not resolve, and the host can give its addresses to another registration. The client also skips its later activation steps, such as the host_ip name hold.
The reap path in crates/minimald/src/server.rs has the same problem: the client is gone, so its lease closed first.
Required change: keep the lease across the cleanup destroy, and commit it when the session is kept. The reap path needs a daemon-side answer.
There was a problem hiding this comment.
Fixed in ff4f7aa. The lease is now held across the cleanup destroy too, and settled on its answer (settle_lease_after_cleanup): committed when the session is kept, dropped uncommitted with the row withdrawn when it was destroyed. A kept session also gets the host_ip name hold, as a successful activation does.
The reap path: behind a VM host daemon the reap stays unconditional. The gone client's lease closed with its connection, the host withdrew the row, and nothing in the VM can publish it again. A native daemon has no such row, and keeps the session. the_lease_follows_the_cleanup_destroys_answer covers the three answers against a fake daemon socket.
| crate::best_effort_destroy(&mut client, id).await; | ||
| // Only if unfinished: a finalize that completed after the client | ||
| // gave up leaves a live session, which is kept (inbox#690). | ||
| crate::best_effort_destroy(&mut client, id, true).await; |
There was a problem hiding this comment.
This change leaves task boxes behind.
Failure case: min task run test without --keep. The finalize passes the client deadline, then completes. The cleanup destroy keeps the box as Active, and the client returns the error. No exec starts, so the destroy on exec exit never runs. The box stays until a person destroys it.
This file's own rule is that a client killed mid-run strands nothing.
Required change: keep this destroy unconditional, or make it conditional only with --keep.
There was a problem hiding this comment.
Agreed. Reverted in ff4f7aa: min task run's cleanup after a failed finalize is unconditional again, with a comment saying why. Only min session activate's cleanup keeps a finished session.
| match tokio::time::timeout(DESTROY_TIMEOUT, call).await { | ||
| Ok(Ok(minimald_rpc::Errorable::Ok(_))) => {} | ||
| Ok(Ok(minimald_rpc::Errorable::Ok(resp))) => { | ||
| if resp.kept { |
There was a problem hiding this comment.
DESTROY_TIMEOUT (10 s, line 773) is now often shorter than the wait it covers. A conditional destroy waits behind the finalize in the actor's mailbox by design, and that finalize can take minutes on a stalled fetch.
Failure case: the finalize needs more than 10 s more. The client prints "could not remove the unfinished session", and kept is false, so it withdraws the row. Then the daemon processes the destroy and keeps the session. The message is wrong, and the row is withdrawn under a live box.
Required change: give the conditional destroy a timeout that covers the finalize, or treat a timeout as "unknown" and do not withdraw the row.
There was a problem hiding this comment.
Fixed in ff4f7aa, both ways. The conditional destroy now waits up to 30 s. Anything short of a confirmed destroy is reported as unknown: the lease is committed, the row stays, and the client says the finalize may still be running, the session may yet become live, and how to remove it. A row with no box behind it costs its addresses until a destroy. A live box with no row loses its name and can lose its addresses. So only a confirmed destroy withdraws the row.
| // it destroys. An idle session has no finalize in flight, so | ||
| // its record answers: a finished one is kept, an unfinished | ||
| // one takes the unconditional path below. | ||
| if only_if_unfinalized { |
There was a problem hiding this comment.
This path destroys a session outside the tearing_down set that #2224 added. The actor deletes its own record and breaks with SelfInitiated, but its id is never in tearing_down, and its handle stays in running until Evict.
Two results:
Listreadsstore.handles()and then callshandle.record(). If the record is deleted between the two calls,record()fails and the wholemin lsfails. The reap now always takes this path, so every interrupted activation opens this window. The window is small.- A user's unconditional destroy is not serialized with this one. With
[DestroyIfUnfinalized, Destroy]in the mailbox of anActivesession, the cleanup answerskept, and then the user's destroy deletes the session. The cleanup caller then says the session is live and skipswithdraw_box_row. This is the open minimal-foundry thread.
Required change: insert the id into tearing_down on this path too, or accept the race and say why.
There was a problem hiding this comment.
Fixed in ff4f7aa: the conditional path puts the id in tearing_down until the actor answers, with the same guard as the unconditional path. While it is in flight, a second destroy is refused, and the refusal now says to destroy again if the session is still listed afterwards. List skips the session while its actor may be deleting the record. A kept session leaves the set with the answer. a_cleanup_destroy_queued_behind_a_running_finalize_keeps_the_session asserts both.
64a2a38 to
ff4f7aa
Compare
|
On the test gap from the review: |
There was a problem hiding this comment.
correctness
minimal-review — approval withheld
Verdict COMMENT — APPROVE withheld: 5 unresolved threads — minimal-foundry[bot] crates/minimald/src/sessions.rs:1616, norrietaylor crates/minimal/src/cmd/session.rs, norrietaylor crates/minimal/src/task.rs, norrietaylor crates/minimal/src/cmd/mod.rs:793, norrietaylor crates/minimald/src/sessions.rs:1569. No findings.
Panel: 1 of 3 personas ran (correctness); did not complete: security, supply-chain.
ff4f7aa to
e1e3ad1
Compare
There was a problem hiding this comment.
correctness
minimal-review — approval withheld
Verdict COMMENT — APPROVE withheld: 5 unresolved threads — minimal-foundry[bot] crates/minimald/src/sessions.rs:1616, norrietaylor crates/minimal/src/cmd/session.rs, norrietaylor crates/minimal/src/task.rs, norrietaylor crates/minimal/src/cmd/mod.rs:793, norrietaylor crates/minimald/src/sessions.rs:1569. No findings.
Panel: 1 of 3 personas ran (correctness); did not complete: security, supply-chain.
e1e3ad1 to
205e053
Compare
cb0dbdd to
f6f8071
Compare
205e053 to
79a3cf2
Compare
|
Rebased onto main 11b0b44, through the renames of #2228 and #2229 ( |
f6f8071 to
7f00593
Compare
79a3cf2 to
be1fbe1
Compare
|
Rebased onto main 9ea9433. Main's new test-only actor message (#2304) sits beside this stack's |
7f00593 to
c30facf
Compare
be1fbe1 to
6f13094
Compare
There was a problem hiding this comment.
minimal-review — approval withheld
Verdict COMMENT — APPROVE withheld: 6 unresolved threads — norrietaylor crates/minimal/src/cmd/session.rs, norrietaylor crates/minimal/src/task.rs, norrietaylor crates/minimal/src/cmd/mod.rs, norrietaylor crates/minimald/src/box_manager.rs:1619, minimal-foundry[bot] crates/minimald/src/box_manager.rs:1685, …. No findings.
Panel: 3 of 3 personas ran (correctness, security, supply-chain).
c30facf to
2a4db77
Compare
6f13094 to
ff5dfbd
Compare
norrietaylor
left a comment
There was a problem hiding this comment.
The earlier threads are addressed: the lease is held across the cleanup, the task cleanup is unconditional again, the conditional destroy waits for its answer, and the conditional path goes through tearing_down. Two problems remain.
- Ctrl-C during the cleanup wait withdraws the row of a box that can still go live (
crates/minimal/src/cmd/mod.rs:887-889). While the conditional destroy is in flight the id is intearing_down, soGetRecordreturnsNone(box_manager.rs~1477), the Ctrl-C branch maps that toDestroyed, andsession.rs:1652drops the lease uncommitted and callswithdraw_box_row. The daemon'sDestroyIfUnfinalizedis not cancelled and runs on. On minvmd: finalize stalls, the user presses Ctrl-C after "Press Ctrl-C to stop waiting", the row goes, then finalize succeeds and the actor keeps the box. The box is Active with no row: its name does not resolve, its addresses can be handed out again, and nohost_iphold is made. That is the outcome this PR's table says it prevents. The note's "leaves the session as it is" is also not true, since the queued cleanup still destroys an unfinished box. Ctrl-C has to leave the lease committed (orUnknown) while the destroy is outstanding. - Every daemon error maps to
Unknown(mod.rs:873), the open bot thread. An error answer commits the lease, includingNotFoundwhen a concurrentmin session destroyremoved the box first. The row then stays with no box behind it, andmin session destroycan no longer find it.NotFoundshould map toDestroyed. - The Ctrl-C warning ("until the VM host daemon restarts") names an internal component and states a consequence the user cannot act on. Per the user-facing text rules in AGENTS.md it should say what happens to the box in the user's words, or name a command that clears it.
This is stacked on #2224, so no CI lane has compiled or tested it. A test merge into main conflicts in cmd/session.rs, task.rs, minimald-rpc/src/lib.rs, box_actor.rs, box_manager.rs and rpc.rs (#2479 and #2480); the stack needs rebasing and this retargeted to main before merge.
|
Thanks @norrietaylor, both were real. Rebased onto the new #2224 head as one ported commit (main's rename of the CLI and RPC files made a replay conflict whole-file), with both fixed:
The lease is still held across the cleanup, the task-path cleanup unconditional, and the conditional destroy still waits for its answer under the 24 h client deadline. CI mirror on the box green apart from the known |
2a4db77 to
470dbc4
Compare
ff5dfbd to
f9cbe92
Compare
|
The PR so nice I approved it twice! |
There was a problem hiding this comment.
minimal-review — approval withheld
Verdict COMMENT — APPROVE withheld: 4 unresolved threads — norrietaylor crates/minimal/src/cmd/session.rs, norrietaylor crates/minimal/src/task.rs, norrietaylor crates/minimal/src/cmd/mod.rs, norrietaylor crates/minimald/src/box_manager.rs:1628. No findings.
Panel: 3 of 3 personas ran (correctness, security, supply-chain).
…d activating An activation the client gave up on (a finalize past its deadline) sent an unconditional destroy that raced the finalize still running in the daemon: the session could finish, and be destroyed, or be kept with its box row withdrawn (inbox#690). The cleanup destroy is now conditional: the session decides, in mailbox order after any finalize ahead of it, and answers `kept` for one that finished. The client waits for that answer with the registration's lease held, and settles the lease and the row on it. Ported onto main's renamed tree on top of #2224's rebase. Two changes from the earlier rounds: a Ctrl-C during the wait leaves the outcome unknown (lease committed, row kept) rather than reading a record a destroy in flight hides, which withdrew the row of a box that could still go live; and a daemon answer that the session is already gone maps to destroyed, since nothing runs behind the row and no later destroy could find it.
A cleanup destroy after a failed finalize waits for the daemon's answer under a 24 h deadline, with the user's Ctrl-C as the real bound. With no terminal on stdin (CI, a script) nothing presses it, so the wait is bounded at 15 minutes instead and ends unknown: the row stays, the daemon still settles the session, and the warning says how to see what became of it.
548daa8 to
6285891
Compare
There was a problem hiding this comment.
minimal-review — approval withheld
Verdict COMMENT — APPROVE withheld: 4 unresolved threads — norrietaylor crates/minimal/src/cmd/session.rs, norrietaylor crates/minimal/src/task.rs, norrietaylor crates/minimal/src/cmd/mod.rs, norrietaylor crates/minimald/src/box_manager.rs:1645. No findings.
Panel: 3 of 3 personas ran (correctness, security, supply-chain).
Refs gominimal/inbox#690 (its destroy half; the deadline and progress half stays open). Stacked on #2224, whose tearing-down set it builds on.
What
A slow first activation outlasts the client's
FinalizeSessiondeadline. The client gives up and sends aDestroySessionto clean up, or is interrupted, and the daemon's connection-close reap sends one. That destroy queues behind the finalize still running; the finalize writesActive; then the destroy deletes the session it just finished. The user sees a timeout, then no session, though it would have been ready a moment later.The change
A cleanup destroy is conditional, and the session decides it:
DestroySessionRequestcarriesonly_if_unfinalized(default off).DestroyIfUnfinalizedwithout claiming it. The actor handles it in mailbox order, after any finalize ahead of it: anActivesession is kept, anything else is destroyed and the actor evicts itself. An idle session has no finalize in flight, so its record's status decides.min session activate's cleanup after a failed or timed-out finalize sends it. A user'smin session destroystays unconditional, and so doesmin task run's cleanup: a kept task box would have no exec whose end destroys it.The box row
min session activateholds the registration's lease across the finalize and, when the finalize fails, across the cleanup destroy too. The lease is settled on the destroy's answer:host_ipname hold is made as for a successful activationA row with no box behind it costs its addresses until the session is destroyed. A live box with no row has a name that does not resolve and addresses the host may hand to another box. So only a confirmed destroy withdraws the row.
Wire compatibility: the request flag defaults to off, so an older client's destroy is unconditional as before;
keptin the response is only ever set in answer to the flag, so an older client never sees it.Tests:
Activesession and destroys a pending one.Evidence
Lab (min-regress, Cloud Hypervisor, local-minimald): this branch at the review round on main 6fb3c55, beside main, four distros × 2 rounds (lab job 00902, batch
c5-690r2-matrix):The lab's daemon is native, so it exercises the reap's keep branch and not the VM host's unconditional one. The new unit tests also ran 30 times in a loop with no failure (lab job 00905).
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes