Skip to content

fix(minimald): end the box_actor flakes from the mint seam race and reset hangups - #2649

Merged
norrietaylor merged 1 commit into
mainfrom
fix/box-actor-flaky-timeouts
Oct 28, 2026
Merged

norrietaylor merged 1 commit into
mainfrom
fix/box-actor-flaky-timeouts

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes #2609

Neither flake was a timeout that was too short. Each had a race behind it, so this PR changes no bound and changes no assertion.

a_mint_whose_host_dies_before_attach_reports_a_spawn_failure (channel did not close within the timeout; got [])

Cause: a race in the test seam. mint_attach_seam seeds HostDiedBeforeAttach. It ends the host it just launched by calling launched.1.abort() and awaiting the handle, and only then does the mint's attach send. Since #2570, landed 2026-10-25 (the flakes started the next day), the handle of a host_ip box is the watch_quic443_exit watcher, not the host loop. Aborting the watcher reaches the loop only through AbortLoopOnDrop, which asks the loop to abort. So the awaited handle can resolve while the loop still holds its mailbox. The attach send then queues into that mailbox and mint_session_host returns Ok(()). The client gets no refusal and its channel never closes. Under load the loop's cancellation loses this race more often.

Fix: the seam now also awaits the event the attach depends on, the mailbox closing. It uses a new test-only HostHandle::closed() that wraps mpsc::Sender::closed(). The HostWedged branch gets the same wait, so nothing of the real host outlives the swap. No production path awaits an aborted host task and then expects a closed mailbox, so only the seam changes.

abrupt_client_loss_keeps_task (10.4 s)

Cause: the daemon logged a real hangup under the wrong label. The CI log captured in the failure shows that the loss was logged, but as WARN ... session ended with error error=Protocol error: Connection reset by peer (os error 104) and not as connection closed by peer. The test's 10 s wait for that line could never succeed. A client that closes its UDS socket while daemon output (here, the last echo) is still unread in its receive queue makes the daemon's next read fail with ECONNRESET and not with an end of file. The accept loop matched only early eof, Broken pipe and Disconnected in the rendered message, so it logged a routine abrupt hangup as an incident. That also affects field logs, not only the test.

Fix: a closed_by_peer helper in server.rs also reads the io error kind. ConnectionReset, UnexpectedEof and BrokenPipe now count as a peer hangup. The old message matches stay for errors that russh reports as its own. A unit test, server::tests::a_reset_connection_was_closed_by_its_peer, pins the classification.

Diff

  • crates/minimald/src/box_actor.rs: the seam waits on closed() in both branches.
  • crates/minimald/src/box_runtime/handle.rs: adds the test-only HostHandle::closed().
  • crates/minimald/src/server.rs: adds closed_by_peer, and the accept loop uses it.
  • crates/minimald/src/server/tests.rs: adds the unit test.

Testing

Host: Linux x86_64, 4 vCPUs. The load was stress-ng --cpu 4 (--cpu 8 where noted). All runs used --retries 0 or ran the libtest binary directly.

Mint test, under stress-ng --cpu 4:

build runner runs pass fail
before (origin/main 4152396) test binary, --exact 40 31 9, each channel did not close within the timeout; got [] at tests.rs:1660 after 5.9–6.9 s
after cargo nextest run -p minimald --retries 0 40 40 0 (0.45 s min, 0.56 s median, 1.76 s max)
after test binary, --exact (same harness as before) 40 40 0

Abrupt test:

  • Before the fix, the test itself did not fail locally in 70 loaded runs: 30 runs under stress-ng --cpu 4, plus 40 runs as 4 parallel loops of 10 under stress-ng --cpu 8. The window is narrow.
  • To show the mechanism, I ran a throwaway variant (not committed). It drops the client right after it sends one more line, so the echo is still in flight. On the pre-fix daemon, 3 of 10 runs logged exactly CI's line, session ended with error error=Protocol error: Connection reset by peer (os error 104).
  • With the fix, 30 of 30 runs of that variant logged connection closed by peer, and 8 of those carried Connection reset by peer (os error 104).
  • The real test after the fix, cargo nextest run -p minimald --retries 0 under stress-ng --cpu 4: 30/30 passed (5.54–5.82 s).

Suites and gates:

  • cargo nextest run -p minimald --retries 0: 1139 passed, 18 skipped.
  • just fix: clean.
  • cargo clippy -p minimald --all-targets --locked -- -D warnings: clean.
  • cargo fmt --all -- --check: clean.
  • just clippy-strict: clean.

Checklist

  • Docs updated if behavior changed (no docs change: only log framing changes, and the connection closed by peer line already exists)
  • BREAKING CHANGE: footer present if this is a breaking change (not a breaking change)

Generated by Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Connections closed by the remote peer are now reported at informational level rather than as warnings. Other session errors continue to be reported as warnings.
    • Host shutdown handling now waits for the mailbox to close in affected test scenarios, improving consistency when verifying host cleanup.

@norrietaylor
norrietaylor requested a review from a team as a code owner October 26, 2026 21:36
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: ad1805e1-2bdf-42e0-af5c-64f73f419d65
Commits

Reviewing files that changed from the base of the PR and between 116bdff and 485b4c5.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: d62fdb40-1213-42aa-a413-8aa2191c309c
Commits

Reviewing files that changed from the base of the PR and between 2ced764 and 7b80342.

Files selected for processing: 4
  • crates/minimald/src/box_actor.rs
  • crates/minimald/src/box_runtime/handle.rs
  • crates/minimald/src/server.rs
  • crates/minimald/src/server/tests.rs

Included review availability: This review used your included allowance. 3 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

Walkthrough

The test seam now waits for host mailbox closure in two cases. The server now classifies selected connection errors as peer closures and logs those sessions at informational level.

Changes: Mailbox Shutdown Synchronization

Layer / File(s) Summary
Wait for host mailbox closure
crates/minimald/src/box_runtime/handle.rs, crates/minimald/src/box_actor.rs
A test-only method waits for host mailbox closure. The HostDiedBeforeAttach and HostWedged cases call it after aborting and joining the host task.

Peer Closure Classification

Layer / File(s) Summary
Classify connection errors
crates/minimald/src/server.rs, crates/minimald/src/server/tests.rs
closed_by_peer recognizes selected I/O error kinds and rendered messages. Server::serve uses the helper to select informational logging, and a unit test checks recognized and excluded errors.

Priority: Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: bryan-minimal

Merge Risk: Minimal · previous review 7b803

No actionable merge-blocking risk is established for this change.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title uses the required Conventional Commit format and clearly identifies the two minimald flake fixes: the box_actor mint seam race and reset hangups.
Description check Passed The description includes complete Summary, Testing, and Checklist sections. It explains the causes, fixes, linked issue, test evidence, and validation commands.
Linked Issues check Passed The changes satisfy the coding requirements in [#2609]. The mint seam now awaits HostHandle::closed() after aborting the host in both HostDiedBeforeAttach and HostWedged, so the attach send wait…
Out of Scope Changes check Passed All changed files support [#2609]. The test-only mailbox wait fixes the mint test seam race. The closed_by_peer helper and its unit test fix the log classification that prevented the abrupt-loss tes…
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 4 files.
Full details: Linked Issues check

Explanation

The changes satisfy the coding requirements in [#2609]. The mint seam now awaits HostHandle::closed() after aborting the host in both HostDiedBeforeAttach and HostWedged, so the attach send waits for the mailbox-closure event. The server now classifies protocol-wrapped ConnectionReset, UnexpectedEof, and BrokenPipe as peer hangups, which allows abrupt_client_loss_keeps_task to observe the expected log event. The change adds a_reset_connection_was_closed_by_its_peer to test the classification. The PR reports repeated loaded runs for both tests and a passing minimald suite.

Full details: Out of Scope Changes check

Explanation

All changed files support [#2609]. The test-only mailbox wait fixes the mint test seam race. The closed_by_peer helper and its unit test fix the log classification that prevented the abrupt-loss test from observing its expected event. No unrelated production feature, API change, or unrelated test change is present.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

CI annotation context is incomplete: annotation collection unavailable or limited

A rabbit checks the mailbox door,
Then waits until it shuts once more.
A broken pipe, a reset sound,
Gets tagged as peer-closed, safe and sound.
The tests keep watch on each small seam,
While carrots hop across the stream.

Comment @coderabbitai help to get the list of available commands.

@minimal-foundry

minimal-foundry Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Approved: no findings at 485b4c574735.

Nothing to do.

Clobber: none seen.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Oct 26, 2026
@CLAassistant

CLAassistant commented Oct 26, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both diagnoses hold up against the code: aborting the host_ip watcher only asks the loop to abort, so waiting on the mailbox's close is the right seam; and ECONNRESET on a peer that closed with output unread is the same hangup as EOF, which closed_by_peer names once instead of matching rendered text. No bound changed, no assertion changed. This is also the flake that hit #2650's CI yesterday.

…eset hangups

- wait for the host mailbox to close in the mint seam
- log a reset connection as closed by its peer
@norrietaylor
norrietaylor force-pushed the fix/box-actor-flaky-timeouts branch from 116bdff to 485b4c5 Compare October 28, 2026 05:40

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@norrietaylor
norrietaylor added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 4066c3b Oct 28, 2026
36 checks passed
@norrietaylor
norrietaylor deleted the fix/box-actor-flaky-timeouts branch October 28, 2026 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky minimald box_actor timing tests in the merge queue

3 participants