Repository navigation
Closed
Labels
performanceBenchmark performance regressionsBenchmark performance regressions
Milestone
Description
Follow-up of #841. The manual shape sweeps (report, merged in #860) proposed six small representatives. Each guards one behavior the sweep found, at a cost that looks compatible with a nightly route. None was added to nightly CI by #841, and none is qualified yet.
Candidates
| Candidate | Behavior it guards | Bench-host warmed median |
|---|---|---|
| 100k facts, 90%-skewed running sum, committed | Partition-at-a-time window retention follows the largest partition | 718.680 ms |
10k entities, K=256 skewed set membership, committed |
Set membership over large value-page sets | 1,124.625 ms |
| 100k facts, descending half offset + 1,000, committed | Sort heap of offset + limit rows, paired with ascending input-order Sort omission |
62.810 ms |
1,110-node tree ancestors, k=1, pending |
Small in-memory ancestor alternative (5.2× slower without it) | 0.205 ms |
1,110-node tree ancestors, k=16, committed |
Bound-value AVET probes (16× slower with attribute scans) | 54.969 ms |
| 10k facts, one held cursor across the churn schedule | Quarantine of freed pages, release and reuse | 0.86–0.94 s of checkpoints per run |
Work
For each candidate:
- Qualify it on CI hardware. Measure fixture construction, warm-up, process start-up and run-to-run variance on the nightly runner. The figures above come from an idle bench host and don't transfer.
- Pin more than time. Assert the answer (row count and checksum) and the relevant path counters (e.g.
bound_value_avet_range,ancestor_alternative_selected,entity_probe_cache_hit), plus charged query memory where that is the behavior guarded (window retention,Sortheap). A path regression must fail distinctly from a slower runner. - Choose the route. Decide between Criterion and the envelope profile (
profile.envelope, opt 3; the bench profile is opt-level z). Follow the alternating main/v3 schedule and per-suite alerting. - Reject candidates that fail. Record why for any candidate that doesn't qualify (cost, variance, or a fixture too large for the runner).
Keep conclusions distinct from #841: a nightly route detects regressions in one representative, and it doesn't re-establish the sweep's envelope.
Acceptance criteria
- Each candidate is qualified with a recorded cost and variance on the nightly runner, or rejected with a reason. All six qualified (evidence, test: qualify the #841 nightly candidates (#863) #876): six runner runs, cost ≤ 204 s, spread 3.5–13.1% after per-run speed normalization (bar amended, see the README).
- Qualified candidates run nightly, asserting answers, path counters and (where relevant) charged memory as well as time. Six Criterion IDs under
^window/,^query/,^recursion/ran and uploaded in v3 Nightly Benchmarks 42401252969 and 42510869989 (0 alerts, 0 errors);v3 acceptance / nightly-candidates(ci: run the #863 nightly candidate acceptance suite (v3) #878, ci: run the #863 nightly candidate acceptance suite (main) #879) passed in scheduled Smoke Suite (v3) 42412666900 and 42523112507. -
docs/bench-coverage-v3.mdlists the new routes and marks the Bench: large-scale shape sweeps and recursive optimization isolation #841 proposals as resolved (test: qualify the #841 nightly candidates (#863) #876).
Metadata
Metadata
Assignees
Labels
performanceBenchmark performance regressionsBenchmark performance regressions
Activity
docs: #863 qualification evidence; remove the temporary workflow
Merge pull request #876 from project-minigraf/test/863-nightly-candid…
Merge pull request #878 from project-minigraf/ci/863-acceptance
ci: run the #863 nightly candidate acceptance suite on v3 (#879)
adityamukho commentedon Nov 9, 2026
Done on v3: #876 (candidates, hooks, evidence), #878 (v3) and #879 (main) (nightly acceptance). Verified by two scheduled v3 nightly benchmark runs and two scheduled v3 smoke runs; links in the ticked criteria above.