Skip to content

test: long-running soak test at scale (millions of facts, many checkpoints, reopen cycles) #392

Closed

Description

@adityamukho

Problem

The nightly smoke suite has a 15-minute timeout. No test runs for hours, or at a size like a real agent-memory or audit-log deployment, for example:

  • 10M+ facts
  • thousands of checkpoints
  • heavy retract and reassert on hot entities
  • many open and close cycles

Bugs like #370 appeared only after several checkpoints on a real workload.

Ask

Add a weekly soak workflow, for example 4–6 hours on one runner. It should:

  • grow a file-backed database to 10M+ facts, in batches of mixed sizes, with multi-valued attributes and retracts;
  • checkpoint on a schedule and reopen at random;
  • every N minutes, check a sample of entities against a reference log through EAVT, AEVT and full-scan paths;
  • record file size, open time, checkpoint time, peak RSS and p50/p99 query latency, and upload them as artifacts (these also feed the performance envelope issue);
  • fail if anything differs, or if memory grows without bound.

Activity

  1. added this to the v3.0.0 milestone on Sep 26, 2026
  2. added 3 commits that reference this issue on Oct 10, 2026
  3. adityamukho commented on Oct 10, 2026

    @adityamukho
    CollaboratorAuthor

    Done: soak test merged on v3 (#519) and main (#520). .github/workflows/soak.yml runs weekly (Saturday main, Sunday v3). Finding: #518.

  4. added a commit that references this issue on Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceBenchmark performance regressionsstabilityStability/panic issue

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions