Skip to content

fix(minvmd,switch): the per-box BEP connection cap has a default that one Pulumi preview does not - #3396

Merged
mitodrummer merged 7 commits into
mainfrom
agent/3385-the-per-box-bep-connection-cap-has-a-default-tha
Nov 17, 2026
Merged

mitodrummer merged 7 commits into
mainfrom
agent/3385-the-per-box-bep-connection-cap-has-a-default-tha

Conversation

@minimal-foundry

Copy link
Copy Markdown
Contributor

Defect

Fixes #3385

Task marker
### D3180 — The per-box BEP connection cap has a default that one Pulumi preview does not hit, and an operator can set it in minvmd config
  kind: task
  size: M
  est: 110 code, 90 tests
  executor: agent
  defect: bug
  epic: 1437
  layers:
    - switch:
      - crates/switch/src/bep_host.rs: raises DEFAULT_PER_SOURCE_CAP from 16 to the tuned value (64 to 128) and updates its doc comment and the cap tests that assume 16
      test: cargo test -p switch connections_past_the_per_source_cap_are_reset
    - minvmd:
      - crates/minvmd/src/config.rs: adds an optional bep_per_box_cap field to ResourceConfig, with a round-trip test
      - crates/minvmd/src/cmd/mod.rs: adds an env override and an effective_bep_per_box_cap resolver: env, then config.toml, then DEFAULT_PER_SOURCE_CAP
      - crates/minvmd/src/cmd/config.rs: run_set validates and saves the cap (must be at least 1), and run_show prints the effective cap and its source
      - crates/minvmd/src/main.rs: adds a --bep-per-box-cap argument to the config set subcommand and passes it to run_set
      - crates/minvmd/src/cmd/run.rs: passes the effective cap to with_per_source_cap at the boot wiring in place of the constant
      - crates/minvmd/src/net/bep_stub.rs: makes the stub tests read the wire cap in place of the constant where they assert the pool size
      test: cargo test -p minvmd effective_bep_per_box_cap
      test: cargo test -p minvmd round_trips_through_toml
    - spec:
      - docs/specs/18-spec-box-networking/18-spec-box-networking.md: records the tuned per-source cap at NET-132 and says that minvmd config can set it
      test: cargo test -p minvmd bep
  proof: After minvmd config set --bep-per-box-cap 128 and a reboot, 60 parallel CONNECTs from one box to the proxy address all get through, and minvmd config show prints the cap and its source.

Proof

The default per-box cap on Box Egress Proxy connections is now 128, not 16. So 60 parallel CONNECTs from one box get through with no configuration. An operator can run minvmd config set --bep-per-box-cap 128. That command saves bep_per_box_cap to config.toml, refuses 0, and prints saved: vcpus=..., ram_mib=..., bep_per_box_cap=128 (takes effect on next boot). At the next boot the BEP wire takes the cap in this order: MINVMD_BEP_PER_BOX_CAP, then config.toml, then the default. The boot logs the cap and its source. minvmd config show prints bep_per_box_cap = 128 (config), and --json gives bep_per_box_cap and bep_per_box_cap_source. I did not do the live 60-CONNECT run, because this box has no libkrun or VM.

Notes from the agent: 1) The cargo test -p minvmd bep failure is not from this change. bep_attach::tests::one_line_per_attachment_issued_and_withdrawn captures logs with a thread-local tracing subscriber (set_default), and sibling tests on other threads hit the same callsites. On the tree without my minvmd changes, cargo test -p minvmd bep_attach failed 4 times in 8 runs, and that filter runs none of my tests. Before my change, the bep filter ran 9 tests and happened to pass. It now runs 14, because the new tests contain bep in their names as the plan requires. That changes the timing, and the race now loses every time on this 4-core box. The test passes alone and with --test-threads=1 (14 passed). The fix belongs in crates/minvmd/src/bep_attach.rs, which is outside this task's layers. Suggested fix: a process-global capture subscriber, or serialise the log tests. 2) I chose 128, the top of the 64 to 128 range. That gives headroom over the 60 connections in the proof. The cost is about 128 pool slots for each registered box. No switch test assumed 16: the 16s in the refusal-quota test count refusals, not the cap. So only the constant and its doc comment changed in bep_host.rs. 3) The boot wiring in cmd/run.rs is under #[cfg(minvmd_l…[truncated]

Tests

line exit tail
cargo test -p switch connections_past_the_per_source_cap_are_reset 0 running 1 test test bep_host::tests::connections_past_the_per_source_cap_are_reset ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 65 filtered out; finished in 0.20s warning: unknown fields in task runtime-implement-test-0: network warning: unknown fields in task runtime-implem
cargo test -p minvmd effective_bep_per_box_cap 0
cargo test -p minvmd round_trips_through_toml 0
cargo test -p minvmd bep 0

Verify

min task run agent-verify passed ([tasks.verify]).

Out of scope

These files are outside the task's layers:, so this PR is a draft:

  • crates/minvmd/src/bep_attach.rs

One Pulumi preview opens dozens of parallel CONNECTs from one box, and
the working value of 16 reset them.
minvmd config set --bep-per-box-cap saves the cap to config.toml, and
the boot wiring resolves it from MINVMD_BEP_PER_BOX_CAP, then
config.toml, then the switch default. minvmd config show prints the
effective cap and its source.
The cap is 128, and minvmd config set --bep-per-box-cap overrides it.
Another test thread can cache a never interest for the issue and
withdraw callsites before this thread's subscriber counts. Rebuild the
cache before each logged call.
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1a51bdc0-444d-4715-83b6-184531f77b3e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@github-actions

Copy link
Copy Markdown
Contributor

Spec lint

docs/specs/18-spec-box-networking/18-spec-box-networking.md — 10 finding(s)

docs/specs/18-spec-box-networking/18-spec-box-networking.md:171: P004 NET-009 names test 'native_resolution_without_proxy_env', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:378: P004 NET-040 names test 'fresh_install_own_ip_ingress_publishes_loopback', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:388: P004 NET-041 names test 'installer_switch_binary_executable', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:435: P004 NET-048 names test 'linux_amd64_manifest_ships_vm_stack', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:440: P004 NET-049 names test 'fresh_linux_kvm_activate_local_minvmd', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:445: P004 NET-050 names test 'linux_arm64_manifest_ships_vm_stack', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:450: P004 NET-051 names test 'fresh_arm64_kvm_activate_local_minvmd', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:936: P004 NET-132 names test 'proxy_sees_each_vm_box_by_its_switch_address', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:962: P004 NET-134 names test 'deny_all_box_reaches_proxy_and_no_other_host_port', which was not found in the repository
docs/specs/18-spec-box-networking/18-spec-box-networking.md:986: P004 NET-107 names test 'session_outbound_request', which was not found in the repository

Advisory. None of this blocks the merge; the review decides.

Advisory, and never a merge gate. Linter: gominimal/foundry@5ffd41cf, spec format: How Minimal Writes Specs.

@norrietaylor

Copy link
Copy Markdown
Member

/revise CI test-kvm fails on this head (run 43406501188), in the group "stock install runs VM boxes end to end (amd64, installed pair, KVM)". The installed pair (minimal + minvmd) installs and verifies, then:

##[error]'min ls --raw' does not list the stock-installed VM session 01a1d46f-fe82-7de5-82fd-c83bcd1f4d07

No other PR fails this way, and main is green. This branch changes how the installed minvmd starts and reads its config (crates/minvmd/src/cmd/run.rs, cmd/config.rs, cmd/mod.rs, config.rs, main.rs). Find why a VM session activated through the installed minvmd no longer shows in min ls --raw, for example a boot path or config default that changed for an installed (non-dev) minvmd. Fix it, and add a test that pins it. Do not change crates/minvmd/src/bep_attach.rs. Its log-capture race is outside this task.

@norrietaylor

Copy link
Copy Markdown
Member

/revise CI test-kvm fails on this head (run 43406501188), in the group "stock install runs VM boxes end to end (amd64, installed pair, KVM)". The installed pair (minimal + minvmd) installs and verifies, then:

##[error]'min ls --raw' does not list the stock-installed VM session 01a1d46f-fe82-7de5-82fd-c83bcd1f4d07

This repeats the /revise from 2026-11-13. Its two rounds (jobs 165508a0, f1554b76) ran, then lost their result to an HTTP 401 when the round was posted. That was a runtime bug, now fixed (agent-runtime#501). Nothing reached the branch.

When that failure was seen, no other PR failed this way and main was green. This branch changes how the installed minvmd starts and reads its config (crates/minvmd/src/cmd/run.rs, cmd/config.rs, cmd/mod.rs, config.rs, main.rs). Find why a VM session activated through the installed minvmd no longer shows in min ls --raw, for example a boot path or config default that changed for an installed (non-dev) minvmd. Fix it, and add a test that pins it. Do not change crates/minvmd/src/bep_attach.rs. Its log-capture race is outside this task.

@minimal-foundry

Copy link
Copy Markdown
Contributor Author

Round 3 of 10 on a /revise note: the coder changed nothing. Revise round 3: no change, on purpose. I did not find a cause on this branch for the test-kvm failure "'min ls --raw' does not list the stock-installed VM session". I also could not reproduce it: this box has no /dev/kvm, so the KVM lane cannot run here.

What I checked: (1) The boot wiring in cmd/run.rs keeps the same order. The only change is the cap value it passes to with_per_source_cap. effective_bep_per_box_cap is a pure read of config.toml (ResourceConfig::read, a missing file yields the default) plus an env var. It writes nothing and takes no lock. (2) Nothing in the diff branches on an installed versus a lane-built minvmd. Both resolve the same default (128) when no config.toml and no MINVMD_BEP_PER_BOX_CAP exist. So the lane-pair groups in the same job booted with the same cap and the same code. (3) The cap only sizes the BEP listener pool: 128 slots, two in-memory smoltcp sockets each, per registered box. The pool runs on the BEP peer's own thread. The SocketSet …[truncated]

@norrietaylor

Copy link
Copy Markdown
Member

/revise merge origin/main into this branch, and change nothing else.

Round 3 found no cause on this branch for the 2026-11-13 test-kvm failure ("'min ls --raw' does not list the stock-installed VM session"), and KVM cannot run in this sandbox. That CI run was against a main from three days ago. A merge of today's main gives CI a fresh test-kvm run, which shows whether the failure is real on this branch.

Do:

  1. Merge origin/main (a merge commit, no rebase, no force-push). Resolve any conflict so both sides' behaviour stays.
  2. Run this task's test: lines.
  3. Change no other file.

@minimal-foundry

Copy link
Copy Markdown
Contributor Author

Round 4 of 10 on a /revise note: the coder changed nothing. every commit this round put on the branch above 183e300 is a merge of the base branch, so it carries no work of its own

@norrietaylor
norrietaylor marked this pull request as ready for review November 16, 2026 21:33
@norrietaylor
norrietaylor requested a review from a team as a code owner November 16, 2026 21:33

@norrietaylor norrietaylor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read at 602c634c against D3180. DEFAULT_PER_SOURCE_CAP is 128, inside the plan's 64–128. The cap resolves env (MINVMD_BEP_PER_BOX_CAP), then config.toml, then the default (resolve_bep_per_box_cap, pure and tested per layer). A zero is refused by config set and read as unset at resolve. config show prints the cap and its source, and the boot wiring logs both and passes the cap to with_per_source_cap. Spec NET-132 records the tuned value and the knob. Cost: each slot holds two smoltcp sockets with 4 KiB rx and 4 KiB tx (POOL_BUFFER_LEN), so 128 slots is about 2 MiB per registered box, up from 256 KiB at 16. That is acceptable. Outside layers:: bep_attach.rs gets two rebuild_interest_cache() calls in one test, a test-only fix for a log-capture race. Required CI is green on today's main, including the test-kvm lane that failed on 2026-11-13.

@minimal-foundry

minimal-foundry Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Approved: every review thread resolved at c6a067edf568.

@norrietaylor resolved the last thread, which releases the approval this review had withheld.

Nothing to do.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — advisory review

Verdict COMMENT — 1 finding, 1 inline.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

Comment thread crates/minvmd/src/cmd/mod.rs Resolved
Each slot of a box's share holds listening sockets and buffers, so an
unbounded value can exhaust memory when a box registers. config set
refuses a value above the maximum, and the boot resolver clamps an env
or config.toml value to it with a warning in the log.
@minimal-foundry

Copy link
Copy Markdown
Contributor Author

Round 5 of 10 on the runtime's review findings, pushed as c6a067edf568.

line exit tail
cargo test -p switch connections_past_the_per_source_cap_are_reset 0 running 1 test test bep_host::tests::connections_past_the_per_source_cap_are_reset ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 65 filtered out; finished in 0.20s warning: unknown fields in task runtime-implement-test-0: network warning: unknown fields in task runtime-implem
cargo test -p minvmd effective_bep_per_box_cap 0
cargo test -p minvmd round_trips_through_toml 0
cargo test -p minvmd bep 0

Verify min task run agent-verify passed.

Outside the task's layers::

  • crates/minvmd/src/bep_attach.rs

Converted to a draft: something above needs a person before review.

Review threads the coder judged:

  • 👍 valid — crates/minvmd/src/cmd/mod.rs:303: Added MAX_BEP_PER_BOX_CAP = 1024. config set refuses a larger value. The boot resolver clamps an env or config.toml value to 1024 and logs a warning. Tests pin both.

After minvmd config set --bep-per-box-cap 128 and a reboot, each box can hold 128 connections to the proxy address, so 60 parallel CONNECTs from one box all get through. minvmd config show prints the cap and its source (env, config or default). This round, minvmd config set --bep-per-box-cap refuses a value above 1024 with the message "--bep-per-box-cap must be at most 1024". An env or config.toml value above 1024 is clamped to 1024 at boot.

Notes from the agent: Revise round 5: added MAX_BEP_PER_BOX_CAP = 1024 in cmd/mod.rs. validate_bep_per_box_cap refuses a larger value. resolve_bep_per_box_cap clamps env and config.toml values to it, and effective_bep_per_box_cap logs a warning when it clamps. The --bep-per-box-cap help text now says "(1 to 1024)". I did not run the 60-CONNECT proof on a booted VM on this box. The unit tests cover the resolver and the validation only. The task and spec give no maximum, so 1024 is my choice: it is 8 times the 128 default.

@minimal-foundry
minimal-foundry Bot marked this pull request as draft November 16, 2026 21:48

@minimal-arch minimal-arch Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arch-review: PASS · the full ruling

The per-box Box Egress Proxy cap now has an upper bound of 1024 and a lower bound of 1. minvmd config set refuses a value outside it, and the boot resolver clamps an env or config.toml value to it. Section 9 line 333 requires per-box connection caps on the BEP plane, and this head meets that requirement. Only the host can set the cap, so the guest cannot reach it (section 7.1 line 245). The change conforms. The NET-132 note does not yet record the 1024 maximum. Add it in a later spec edit.

Ruled at c6a067e against gominimal/arch@327e1b57d16b by claude-opus-5-5.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approval withheld

Verdict COMMENT — APPROVE withheld: 1 unresolved thread — minimal-foundry[bot] crates/minvmd/src/cmd/mod.rs:303. No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@norrietaylor
norrietaylor marked this pull request as ready for review November 16, 2026 21:56

@norrietaylor norrietaylor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-read at c6a067ed. Round 5 bounds the knob: MAX_BEP_PER_BOX_CAP = 1024. config set refuses a value above it, and resolve_bep_per_box_cap clamps an env or config.toml value to it, with a warning at boot that names the source. config show reads the same resolver. Tests cover the refusal and the clamp from each layer. The rest is the main merge. bep_attach.rs (test-only, outside layers:) is accepted at this head.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — every review thread is resolved; @norrietaylor closed the last one. The review of c6a067edf568 withheld this approval on the threads open then. Posted as a comment: the App cannot approve its own pull request; the check and the label carry it.

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Nov 16, 2026
@norrietaylor
norrietaylor added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@norrietaylor
norrietaylor added this pull request to the merge queue Nov 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026
@mitodrummer
mitodrummer added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit ff59066 Nov 17, 2026
41 checks passed
@mitodrummer
mitodrummer deleted the agent/3385-the-per-box-bep-connection-cap-has-a-default-tha branch November 17, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

D3180 — The per-box BEP connection cap has a default that one Pulumi preview does not hit, and an operator can set it in minvmd config

2 participants