Skip to content

fix(orchestrator): carry the current span into spawned deliverable tasks - #2123

Closed
bryan-minimal wants to merge 1 commit into
mainfrom
fix/orchestrator-span-propagation
Closed

bryan-minimal wants to merge 1 commit into
mainfrom
fix/orchestrator-span-propagation

Conversation

@bryan-minimal

@bryan-minimal bryan-minimal commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

JoinSet::spawn doesn't inherit the caller's tracing span. So each build, cache-fill and subset task the orchestrator spawns starts outside the request's trace, and so does every span inside it (download_with_verification_and_caching, download_web_with_verification_and_caching, materialize).

Seen in build-servers' Cloud Trace: over 24 h, about 740 root spans from res-server with no parent (454 + 244 downloads, 45 materialize), so a run's trace stops at grpc.server. build-servers#309 fixed the same thing for res-server's own spawns; this is the layer below.

Fix: .in_current_span() on the three spawned futures. No behaviour change otherwise.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Build, cache-fill, and subset task activity is now associated with the current tracing context for clearer diagnostics.

JoinSet::spawn does not inherit the caller's tracing span, so every
build, cache-fill and subset task (and the source downloads and
materialize spans inside it) started a new root trace. In build-servers'
Cloud Trace that is ~740 orphan roots a day from res-server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bryan-minimal
bryan-minimal requested a review from a team as a code owner October 9, 2026 19:59
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: c8e04fdc-990b-4541-a4f5-1a7d6c5f4fca

📥 Commits

Reviewing files that changed from the base of the PR and between ea1b015 and e02d083.


📒 Files selected for processing (1)
  • crates/orchestrator/src/lib.rs

Included review availability: This review used your included allowance. 1 included review remains after this review. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.



📝 Walkthrough

Walkthrough

The orchestrator now instruments spawned build, cache-fill, and subset-materialization tasks with the current tracing span. A comment notes that JoinSet::spawn does not inherit the current span.

Changes

Task Span Propagation

Layer / File(s) Summary
Instrument spawned tasks
crates/orchestrator/src/lib.rs
The orchestrator imports tracing::Instrument, documents that JoinSet::spawn does not inherit the current span, and instruments build, cache-fill, and subset-materialization tasks with that span.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~4 minutes

Change: Bug fix

Suggested reviewers: 0chroma


Merge Risk: ⚪ Minimal · up to e02d0

The change propagates tracing context to the build, cache-fill, and subset tasks, addressing orphaned traces. No actionable merge-blocking risk remains.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: propagating the current tracing span into spawned orchestrator tasks. It follows the required Conventional Commit format.
Description check Passed The description provides a clear summary, motivation, scope, and implementation details. It does not include the template's Testing section or Checklist, but the main required change information is co…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.


  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit hops where build tasks run
With cache-fill close behind
Subsets join the tracing span
Their task trails now aligned
The bunny thumps: “One span for all!”

Comment @coderabbitai help to get the list of available commands.

@minimal-foundry

minimal-foundry Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Approved: no findings at e02d08307817.

Nothing to do.

Clobber: none: git log on crates/orchestrator/src/lib.rs shows no recent fix reverted; the head commit is this PR's own.

@minimal-foundry minimal-foundry Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — approved

Verdict APPROVE — No findings.

@minimal-foundry minimal-foundry Bot added the review:approved minimal-review approved the current head label Oct 9, 2026

@norrietaylor norrietaylor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architecture review: span propagation only; conforms to architecture.md and the arch specs. Approved.

@bryan-minimal

Copy link
Copy Markdown
Member Author

Superseded by #1995 (98dbe05, 2026-10-13), which wraps each spawned deliverable in tracing::Instrument::instrument(..., span.clone()) with span = tracing::Span::current(), the same fix. This branch now conflicts with main only because both sides changed those three spawns. build-servers picked up #1995 with its pin bump to 6fb3c55 (build-servers#373, deployed 2026-10-16).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved minimal-review approved the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants