Repository navigation
ci(bench): compare each run with the last 30 runs, not all history - #568
Merged
Merged
Conversation
Bencher's t-test used every sample since May, so a correct step change kept alerting for weeks: query/predicate_pushdown/100k since #445 (#512, #495), and checkpoint/* and open/wal_replay/100k since #476's backport stopped the drop-save that had made those fixtures no-ops (#561, #562). Replaying main's history, a 30-run window still alerts on two runs in a row for a lasting change, then takes it as the baseline after about five runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QAt9R1aSPQG7rV5ouFDMU
This was referenced Oct 13, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
adityamukho
added a commit
that referenced
this pull request
Oct 13, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016QAt9R1aSPQG7rV5ouFDMU
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
--threshold-max-sample-size 30to the Bencher t-test inbench.yml.Why. The t-test compared each run with every sample since May, so a correct step change kept alerting for weeks:
query/predicate_pushdown/100k(Benchmark regression: query (main) #512 main, Benchmark regression: query (v3) #495 v3) has alerted every night since 2026-10-06. That step is Queries return wrong results with more than 65,535 uncheckpointed facts #445 (v2.0.3): the fixture holds 200k pending facts, and before Queries return wrong results with more than 65,535 uncheckpointed facts #445 the query returned 0 rows (see Benchmark regression: query #487/Benchmark regression: query #501).checkpoint/*(Benchmark regression: checkpoint (main) #562) andopen/wal_replay/100k(Benchmark regression: open (main) #561) on main stepped at0da66eb3..a861e551. Bisected locally to fix(db): stale WAL removed by checkpoint; usize::MAX threshold skips close-time save (#457, #476, v2.x backport) #540, the v2.x backport of Close-time save ignores wal_checkpoint_threshold(usize::MAX) #476. Before it, dropping awal_checkpoint_threshold(usize::MAX)handle still saved the.graph. Sopopulate_file_no_checkpointleft a fully checkpointed file,checkpointtook the nothing-to-do early return, andwal_replayapplied 0 entries.checkpoint/1k: 109 µs → 11.2 ms;open/wal_replay/1k: 5.09 ms → 0.39 ms. The benches now measure the work they are named for. This is not a code regression.Window size. I replayed main's Bencher history through the same t-test. With all history, predicate_pushdown alerts indefinitely. With 30 runs, a lasting step still alerts on two nights in a row (so the issue is opened), then becomes the baseline after about five runs. Noise-only benches (insert_file) produced no alert pairs.
--threshold-max-sample-sizeis in the pinned CLI (v0.4.25).Touches different lines from #566 (issue titles), so the two should merge cleanly.
🤖 Generated with Claude Code
https://claude.ai/code/session_016QAt9R1aSPQG7rV5ouFDMU