Skip to content

Benchmark regression - 2026-04-17 #148

Closed as not planned
Closed as not planned

Description

@github-actions

Bencher detected a performance regression in the nightly benchmark run (insert_file).

Run: https://github.com/adityamukho/minigraf/actions/runs/24564482102

Please review the Bencher dashboard for details on which benchmarks regressed.

Activity

  1. adityamukho commented on Apr 17, 2026

    @adityamukho
    Collaborator

    Analysis: False positive — runner variability, not a code regression

    After investigation, this is not a genuine performance regression. The alert was triggered by GitHub Actions runner jitter, not by any code change.

    Evidence

    The benchmark numbers themselves are incoherent:

    • insert_file/single_fact/1k: confidence interval [426 µs … 1.6 ms] — a 4× spread with 1 high-severe outlier flagged by Criterion
    • insert_file/single_fact/1k measured slower than insert_file/single_fact/10k — physically impossible for the same operation at smaller scale
    • insert_file/single_fact/1m measured faster than insert_file/single_fact/100k — same issue

    This pattern is classic OS/runner scheduling noise, not a workload characteristic.

    No code change affects insert performance:

    The Bencher baseline was established from run 24553303618, which ran before PRs #144, #145, #146 were merged. The commits merged since then are:

    There is no code path change between the passing baseline and the current main that could affect insert_file latency.

    Conclusion

    The Bencher threshold was crossed because the baseline run happened to land on a faster runner, and this run hit scheduling jitter (evidenced by multiple high-severe outliers across measurements). The next nightly run should return to baseline levels.

    Closing as false positive. The Bencher alerts can be dismissed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceBenchmark performance regressions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions