Skip to content

test(minimal): wait for the daemon spans in the commands-trace test - #3401

Merged
norrietaylor merged 2 commits into
mainfrom
fix/3398-otel-trace-wait
Nov 16, 2026
Merged

norrietaylor merged 2 commits into
mainfrom
fix/3398-otel-trace-wait

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Root cause

exec_run_attach_and_task_run_share_the_commands_trace (crates/minimal/tests/otel_cli.rs) runs the verbs, then reads the in-process span log once (in_trace(&log, TRACE)) and looks for each verb's daemon request span under its cmd span. SpanLog records a span when it opens and fills its fields as they are recorded. A verb's answer can reach the CLI before the daemon's request span carries the trace fields the lookup matches, so a single read can miss it.

In CI the test passes. In agent-verify's full-suite runs on a loaded box it failed in every run today: #3361, #3363, #3372, #3386, #3391, #3392, #3394, #3395. Each one turned an agent PR into a draft.

Fix

  • Poll the span log every 20 ms, bounded at 10 s, until every verb's request span is in it. Then assert on that read, as before. A real regression still fails, after the bound, with the same panic and span dump.
  • Move the per-verb lookup into daemon_request and request_span, so the wait and the assertion use one predicate.

Test only. No product code changes.

Test plan

  • CI tests (Linux) runs otel_cli green. The file is #![cfg(target_os = "linux")], so it does not run on macOS. A local cross-check for Linux was not possible here (aws-lc-sys needs a target C toolchain).
  • cargo fmt --all -- --check

Fixes #3398

Summary by CodeRabbit

  • Tests
    • Improved reliability of telemetry trace checks by waiting up to 10 seconds for matching daemon request spans before validation.
    • Added polling for opt-out telemetry spans, verifying that four daemon requests are marked as opted out.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The test maps session verbs to daemon requests and matches daemon spans by name, parent command span, and optional command prefix. It polls the trace for up to 10 seconds, at 20 ms intervals, before checking command spans and opted-out spans.

Changes: Daemon span test

Layer / File(s) Summary
Map, match, and poll for daemon spans
crates/minimal/tests/otel_cli.rs
Helpers map session verbs to daemon requests and match spans by name, parent span ID, and optional command prefix. The test polls for up to 10 seconds, at 20 ms intervals, for matching daemon spans and four additional opted-out exec or attach spans.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: bryan-minimal

Merge Risk: Minimal · up to 88cd0

This only makes a flaky telemetry test wait for daemon spans before asserting. It has no production impact and is safe to merge.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title uses a Conventional Commit form (type(scope): summary), is imperative and lower-case, and names the main change: waiting for daemon spans in the commands-trace test. The raw summary confirms…
Description check Passed The description follows the repository template in substance. It has a Summary-equivalent section (Root cause and Fix) explaining what changes and why, and it links the issue (Fixes #3398). It has a T…
Linked Issues check Passed Issue #3398 asks that exec_run_attach_and_task_run_share_the_commands_trace stop failing when a daemon request span is not yet under the cmd span at the time the test reads the span log. The diff in…
Out of Scope Changes check Passed The whole-PR diff against the merge base changes only crates/minimal/tests/otel_cli.rs. No product code changes. The helper functions, the traced-verb wait loop, and the verb-to-span mapping serve #…
Full details: Title check

Explanation

The title uses a Conventional Commit form (type(scope): summary), is imperative and lower-case, and names the main change: waiting for daemon spans in the commands-trace test. The raw summary confirms the test now polls up to 10 seconds for each verb's daemon request span before asserting, so the title matches the changeset.

Full details: Description check

Explanation

The description follows the repository template in substance. It has a Summary-equivalent section (Root cause and Fix) explaining what changes and why, and it links the issue (Fixes #3398). It has a Testing-equivalent section (Test plan) that reports cargo fmt passing and explains that the Linux-only test could not be run locally. The Checklist items are not addressed: docs updates are not mentioned, and the BREAKING CHANGE footer is not mentioned. The description omits the template's explicit Checklist, which is a non-critical gap.

Resolution

Optionally add the template's Checklist section, stating that docs need no update (test-only change) and that no BREAKING CHANGE footer applies. Optionally paste the relevant test output as evidence, as the Testing section requests.

Full details: Linked Issues check

Explanation

Issue #3398 asks that exec_run_attach_and_task_run_share_the_commands_trace stop failing when a daemon request span is not yet under the cmd span at the time the test reads the span log. The diff in crates/minimal/tests/otel_cli.rs implements the hypothesis in the issue. The test now reads in_trace(&log, TRACE) in a loop. It polls every 20 ms, bounded at 10 s, until each verb's request span resolves through request_span. It then asserts on that read. The lookup moved into daemon_request and request_span, so the wait and the assertion use one predicate. If a span never appears, the test still panics with the same message and span dump. The issue calls the cause a hypothesis, so the diff cannot prove the flake is gone. The Linux CI run is not yet shown as passing, and the test could not be run locally. These are evidence limits, not unmet requirements.

Full details: Out of Scope Changes check

Explanation

The whole-PR diff against the merge base changes only crates/minimal/tests/otel_cli.rs. No product code changes. The helper functions, the traced-verb wait loop, and the verb-to-span mapping serve #3398. The diff also adds a second bounded wait, in the telemetry opt-out test, that polls the telemetry=opt-out marks before counting them. The current PR description covers only the traced-verb test and does not mention this wait, although the PR objectives say it does. The second wait uses the same race rationale in the same file, so it is related test-only hardening, not an unrelated change. The PR description should name this second wait.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks the trace at dawn
Each span appears before it’s gone
Twenty milliseconds, then see
Four opt-out spans join the spree
The command paths now match just right
The rabbit hops into the night

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 44acf0e9-9e42-4794-8c68-14c746e67538
Commits

Reviewing files that changed from the base of the PR and between 2946b3f and 88cd079.

Files selected for processing: 1

  • crates/minimal/tests/otel_cli.rs

Included review availability: This review used your included allowance. Your plan provides up to 5 included reviews per hour; 1 remain after this review.

exec_run_attach_and_task_run_share_the_commands_trace read the span log
once, right after the verbs returned. A verb's answer can reach the CLI
before the daemon records the trace fields on its request span, and in
agent-verify's loaded full-suite runs the test failed every time while
CI passed. Poll the log, bounded at 10s, until every verb's request
span is there, then assert on that read as before. The lookup moves
into daemon_request and request_span so the wait and the assertion use
one predicate.

Fixes #3398
@norrietaylor
norrietaylor force-pushed the fix/3398-otel-trace-wait branch from 9af363d to 2946b3f Compare November 13, 2026 16:23
@norrietaylor
norrietaylor marked this pull request as ready for review November 13, 2026 18:46
@norrietaylor
norrietaylor requested a review from a team as a code owner November 13, 2026 18:46
@minimal-foundry

minimal-foundry Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Approved: every review thread resolved at 88cd079920ed.

@norrietaylor resolved the last thread, which releases the approval this review had withheld.

Nothing to do.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — advisory review

Verdict COMMENT — 1 finding, 1 inline.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

Comment thread crates/minimal/tests/otel_cli.rs Resolved

@bryan-minimal bryan-minimal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read at 2946b3f. The diagnosis matches the code: the daemon's request spans declare their trace fields empty and record them later, so one read of the span log right after the verbs return can miss a span that is about to match. Polling the same predicate the assertion uses, with a bound, is the right repair. A real regression still fails with the same panic and span dump, ten seconds later. Test only; the two helpers keep the wait and the assertion from drifting apart.

Two comments, neither blocking:

  • The telemetry-off half has the same single read. After the second run_the_verbs the test counts the opt-out marks once and asserts exactly 4. That mark is recorded on the daemon span the same way, after the answer may already have reached the CLI, so on a loaded box this count can come up short for the same reason. The review bot's thread says this and I agree with it. The same bounded poll, waiting for the count to reach marked_before + 4, would close it here, or in a follow-up if you want this one in quickly since it unblocks the agent PRs.
  • The wait's predicate does not include span_id, which the loop reads next for the exec span and unwraps. If the trace fields are recorded in separate record calls, a read can land between parent_span_id and span_id. I have not seen it fail; adding s.field("span_id").is_some() to request_span would make the wait cover what the assertions read.

The negative assertion (in_trace(&log, INHERITED).is_empty()) is also a single read, but an early read only makes it weaker, not flaky, so I would leave it.

The telemetry-off half counted the daemon spans' opt-out marks with one
read right after the verbs ran. The daemon records the mark on its request
span after it answers, so that read raced as the first one did (#3398).
Poll for all four marks with the same 10 s bound, then assert.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approval withheld

Verdict COMMENT — APPROVE withheld: 1 unresolved thread — minimal-foundry[bot] crates/minimal/tests/otel_cli.rs:784. No findings.

Panel: 3 of 3 personas ran (correctness, security, supply-chain).

@norrietaylor

Copy link
Copy Markdown
Member Author

@bryan-minimal one change since your approval at 2946b3f: 88cd079 applies the same bounded poll to the telemetry-off opt-out count (the LOW thread at otel_cli.rs:784). CI is green and the test ran for real (PASS [3.759s]). Could you re-approve at 88cd079?

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — every review thread is resolved; @norrietaylor closed the last one. The review of 88cd079920ed withheld this approval on the threads open then.

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Nov 16, 2026
@norrietaylor

Copy link
Copy Markdown
Member Author

@bryan-minimal on your second point: I'm adding s.field("span_id").is_some() to request_span so the wait covers everything the loop unwraps. It's one line, and you're re-reviewing anyway, so I'd rather close it here than leave a known gap in a flake fix. I'll ping you with the new sha.

@norrietaylor
norrietaylor added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 364eb1a Nov 16, 2026
40 checks passed
@norrietaylor
norrietaylor deleted the fix/3398-otel-trace-wait branch November 16, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: exec_run_attach_and_task_run_share_the_commands_trace misses a daemon span under the cmd span

2 participants