Skip to content

docs,ci: land the CI strategy and test-extension contract - #742

Merged
norrietaylor merged 1 commit into
mainfrom
ci/docs-contract-refresh
Jul 14, 2026
Merged

norrietaylor merged 1 commit into
mainfrom
ci/docs-contract-refresh

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 14, 2026 •

Copy link
Copy Markdown
Member

Lands the CI-strategy documentation and the test-extension contract for
contributors and agents — the final docs piece of the CI migration
(#687). Docs-mostly, plus a
small test-evidence plumbing addition.

What

  • CONTRIBUTING.md — the contract: adding a test never requires a CI edit.
    A table maps each test kind to where it lives and how the lane finds it, with
    the discriminator between an integration harness and a full-CLI e2e
    script
    : does it drive the minimal CLI through the whole system?

    Test kind Where How CI runs it
    Unit / in-process integration / doctest next to the code / tests/*.rs / /// core-tests suite
    Integration harness (real VM / kernel / netns) crates/<crate>/tests/*_integration.rs (*_root_integration.rs if it needs root) auto-discovered by binary(/_integration$/) — no CI edit
    End-to-end (drives the CLI) a scripts/ script (e.g. session-e2e.sh) the lane calls the script
  • docs/ci-strategy.md — the full CI design, with a preamble mapping the
    strategy's terms to this repo (five lane workflows; justfile + scripts/,
    not xtask; _integration binary-suffix filtersets, not a vm_e2e_ prefix).
    The §10 contract table is the shipped convention.

  • CLAUDE.md — a pointer so agents inherit the "do not edit
    .github/workflows/" rule.

  • .config/nextest.toml + core-tests — JUnit output and a
    $GITHUB_STEP_SUMMARY pass/fail table, so per-PR test evidence is visible
    without downloading artifacts.

Reconciliation

This supersedes the older parked ci/docs-contract branch, which had drifted:

  • nightly-tests.yml, soak-session-e2e.sh, and the promote-timeout already
    merged with #720 — dropped.
  • A repo-wide CODEOWNERS merged with #728
    (* @gominimal/minimalists), which already gates .github/. The parked
    branch's /.github/ @two-people rule would have narrowed ownership, so it's
    dropped — the freeze is already enforced.
  • The contract is written for the _integration naming that
    #732 shipped, not the old
    _e2e draft.

Sequencing

Per the plan, this pairs with the ruleset flip (which turns on code-owner
review that this contract references); the flip is gated on the current soak
re-maturing. This PR itself is docs-only and can merge independently.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added automated CI test summaries, including total tests, failures, errors, and execution time.
    • Test summaries are published even when earlier test steps fail.
  • Documentation

    • Added comprehensive CI strategy documentation covering test tiers, supported environments, caching, and execution practices.
    • Clarified contributor guidance for CI changes, test discovery, test extensions, and commit message formatting.

@coderabbitai

coderabbitai Bot commented Jul 14, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 81659509-b6fd-486e-b9e8-0ca521b97b00

📥 Commits

Reviewing files that changed from the base of the PR and between 864d0c3 and b948df9.

📒 Files selected for processing (5)
  • .config/nextest.toml
  • .github/actions/core-tests/action.yml
  • CLAUDE.md
  • CONTRIBUTING.md
  • docs/ci-strategy.md
📝 Walkthrough

Walkthrough

This change adds CI JUnit report publication and documents the repository’s CI architecture, test tiers, platform requirements, build and cache model, workflow constraints, and test-extension conventions.

Changes

CI strategy and test reporting

Layer / File(s) Summary
JUnit report generation and publication
.config/nextest.toml, .github/actions/core-tests/action.yml
Nextest writes a CI JUnit report, and the core-tests action always publishes its parsed results to the GitHub step summary.
CI topology and test taxonomy
docs/ci-strategy.md
Defines CI lanes, test tiers, platform constraints, VM execution, artifact handling, and build/cache rules.
Build, cache, and pipeline execution model
docs/ci-strategy.md
Documents PR and nightly jobs, runner requirements, local/CI parity, coverage matrices, and cache-write behavior.
Contributor CI and test-extension contract
CLAUDE.md, CONTRIBUTING.md, docs/ci-strategy.md
Documents commit formatting, workflow edit restrictions, test discovery, harness naming, and extension through scripts or the justfile.
CI strategy references
docs/ci-strategy.md
Adds appendices covering prior-art CI patterns and external references.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Nextest
  participant JUnitReport
  participant CoreTestsAction
  participant GitHubStepSummary
  Nextest->>JUnitReport: write CI test results
  CoreTestsAction->>JUnitReport: parse aggregate attributes
  CoreTestsAction->>GitHubStepSummary: append Markdown summary
Loading

Poem

I’m a rabbit with a CI carrot bright,
JUnit hops through the test-night.
Scripts guide paws, workflows stay still,
Summaries bloom at the runner’s will.
Hop, hop—green checks on the hill!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: documenting the CI strategy and the test-extension contract.

Comment @coderabbitai help to get the list of available commands.

@norrietaylor
norrietaylor force-pushed the ci/docs-contract-refresh branch from b217938 to 9afbcc6 Compare July 14, 2026 00:58

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
CONTRIBUTING.md (1)

18-24: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Keep the validation command consistent with the repository contract.

The documented Clippy command omits --allow-dirty --fix, unlike CLAUDE.md and the required repository command. This can cause contributors to skip the mandated fix pass.

Proposed fix
- cargo clippy --all-targets -- -D warnings
+ cargo clippy --allow-dirty --fix --all-targets -- -D warnings

As per coding guidelines, files matching **/* must run cargo fmt && cargo test -- --include-ignored and cargo clippy --allow-dirty --fix --all-targets -- -D warnings.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CONTRIBUTING.md` around lines 18 - 24, Update the Clippy validation command
in the “Before opening a PR” checklist to include the required `--allow-dirty
--fix` flags, preserving the existing `--all-targets -- -D warnings` arguments
and keeping the documented command consistent with the repository contract.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CONTRIBUTING.md`:
- Around line 46-47: Update the integration harness discovery documentation to
exclude *_root_integration binaries from the generic binary(/_integration$/)
filter, and document a separate binary(/_root_integration$/) match for the
privileged lane. Apply the same correction to the duplicate CI strategy guidance
in docs/ci-strategy.md.

In `@docs/ci-strategy.md`:
- Around line 302-306: Update the branch-protection guidance in the CI strategy
documentation to match the repository’s actual required-check topology: verify
whether the five lane aggregators or only ci-ok is required, then revise the
conflicting statements consistently throughout the document while preserving the
described preflight and join-job behavior.
- Around line 48-53: Update the tiered fan-out paragraph to limit archive
creation and replay claims to the macOS and Linux-KVM lanes, which use cargo
nextest run with --archive-file. Explicitly note that core-tests and the
Linux-native, CI, release, and nightly lanes still run cargo nextest run
directly, and remove the claim that every downstream test job reuses archived
artifacts.
- Around line 157-160: Update the “Tagging” guidance to explicitly use cargo
nextest run for the default and VM test commands, since default-filter and the
referenced filtersets apply only to nextest. Preserve the existing VM test
exclusion and explicit selection behavior, and clarify that plain cargo test
does not honor this filtering.
- Around line 361-376: Update the “Local/CI parity: cargo xtask” section to use
the repository’s existing justfile and scripts/ extension points instead of
cargo xtask commands or tests/smoke/ paths. Replace the example commands and
test-location guidance with the actual documented commands and directories used
by this repository, while preserving the goal of keeping local and CI execution
aligned.
- Around line 446-452: Clarify the naming-convention sentence so the per-crate
meta-test applies only to VM harness candidates, not every top-level file under
crates/<crate>/tests/. Preserve the existing suffix requirements and allowlist,
while distinguishing ordinary integration tests from harness files.

---

Outside diff comments:
In `@CONTRIBUTING.md`:
- Around line 18-24: Update the Clippy validation command in the “Before opening
a PR” checklist to include the required `--allow-dirty --fix` flags, preserving
the existing `--all-targets -- -D warnings` arguments and keeping the documented
command consistent with the repository contract.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 60194908-af2c-4c40-b30d-1a21e3fd1d4e

📥 Commits

Reviewing files that changed from the base of the PR and between eec07c6 and 9afbcc6.

📒 Files selected for processing (5)
  • .config/nextest.toml
  • .github/actions/core-tests/action.yml
  • CLAUDE.md
  • CONTRIBUTING.md
  • docs/ci-strategy.md

Comment thread CONTRIBUTING.md Resolved
Comment thread docs/ci-strategy.md Resolved
Comment thread docs/ci-strategy.md Resolved
Comment thread docs/ci-strategy.md Resolved
Comment thread docs/ci-strategy.md Resolved
Comment thread docs/ci-strategy.md Resolved
@norrietaylor
norrietaylor marked this pull request as ready for review July 14, 2026 15:58
@norrietaylor
norrietaylor force-pushed the ci/docs-contract-refresh branch from 9afbcc6 to 864d0c3 Compare July 14, 2026 16:04

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CONTRIBUTING.md`:
- Line 45: Update the Doctest row in CONTRIBUTING.md to use the workspace-wide
command cargo test --workspace --doc, matching CI and covering doctests across
all crates.

In `@docs/ci-strategy.md`:
- Around line 488-493: Correct the local-proof artifact claim in the “Provide
proofs” section: either document the local nextest invocation that produces
JUnit XML, or remove JUnit XML from the artifacts attributed to just up and
scripts/session-e2e.sh. Keep the existing CI artifact description accurate and
unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2f22772d-3d61-4943-bfd6-8d687914bad7

📥 Commits

Reviewing files that changed from the base of the PR and between 9afbcc6 and 864d0c3.

📒 Files selected for processing (5)
  • .config/nextest.toml
  • .github/actions/core-tests/action.yml
  • CLAUDE.md
  • CONTRIBUTING.md
  • docs/ci-strategy.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/actions/core-tests/action.yml
  • .config/nextest.toml

Comment thread CONTRIBUTING.md Resolved
Comment thread docs/ci-strategy.md Resolved
Documents the frozen-CI contract for contributors and agents (N8):

- CONTRIBUTING.md — the test-extension table (unit / in-process integration /
  doctest / *_integration.rs harness / scripts/ e2e) with the discriminator
  between an integration harness and a full-CLI e2e script; adding a test never
  requires a workflow edit.
- docs/ci-strategy.md — the full CI design, with a preamble mapping the
  strategy's terms to this repo (five lane workflows; justfile + scripts, not
  xtask; _integration binary-suffix filtersets, not a vm_e2e_ test-name prefix)
  and the section 10 contract table updated to the shipped convention.
- CLAUDE.md — a pointer so agents inherit the "do not edit .github/workflows/"
  rule.
- .config/nextest.toml + core-tests — JUnit output plus a $GITHUB_STEP_SUMMARY
  pass/fail table, so per-PR test evidence is visible without artifact
  downloads.

Reconciled against what already merged: the nightly tier, soak script, and
promote timeout landed with #720, and a repo-wide CODEOWNERS landed with #728
(already gating .github/ via the minimalists team), so the parked branch's
copies of those are dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HbXL6jxHTY2YuLPaMxc1yU
@norrietaylor
norrietaylor force-pushed the ci/docs-contract-refresh branch from 864d0c3 to b948df9 Compare July 14, 2026 16:10
@norrietaylor
norrietaylor merged commit 4b81c73 into main Jul 14, 2026
28 checks passed
@norrietaylor
norrietaylor deleted the ci/docs-contract-refresh branch July 14, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants