Skip to content

durability: a .wal from a later checkpoint is replayed onto an older .graph (silent hole after a restore) #535

Closed

Description

@adityamukho

Problem

Opening a .graph file next to a .wal written against a later checkpoint replays the WAL on top of the older file and reports success. The transactions committed between the two checkpoints are missing, and the WAL's later transactions are applied on top, so the database has a silent hole in its history. The next checkpoint makes it permanent.

The usual way to get here is a restore: an operator copies a backup .graph back into place and leaves the current .wal beside it.

Found while writing the durability guide (#398), on v3 (2fea6a3):

let opts = || OpenOptions::new().wal_checkpoint_threshold(usize::MAX);
// gen A: [:a :n 1], checkpoint; copy a.graph to backup
// then: [:a :n 2] checkpoint, [:a :n 3] checkpoint, [:a :n 4] (left in a.graph.wal)
// restore backup over a.graph, keep a.graph.wal, Minigraf::open
// -> opens; tx count 4; [:find ?v :where [?e :n ?v]] returns 1 and 4. Transactions 2 and 3 are gone.

Cause

replay_wal skips entries at or below the file's last_checkpointed_tx_count and applies the rest. The WAL v2 header records base_generation, the generation that was active when the WAL was created, but meta selection uses it only when one meta slot is valid (check_single_valid). When both slots are valid and the WAL's base generation is newer than the chosen meta, nothing refuses the open.

Ask

On open, refuse (new STG code, file and WAL untouched) when the WAL's base_generation is greater than the chosen meta's generation: the WAL belongs to a later state of the file. A WAL with a lower base generation stays valid (a crash between a checkpoint's commit and its WAL delete, #457). Same for read-only opens. Add the case to the golden corpus or the integrity tests.

v2.x WALs (version 1) have no base generation, so v2.x cannot detect this; the durability guide tells operators to remove any .wal before restoring a backup.

Activity

  1. added this to the v3.0.0 milestone on Oct 11, 2026
  2. added
    bugSomething isn't working
    durabilityData durability concern
    known-issueBug present in a published release and not yet fixed in one; listed in the pinned known-issues issue
    on Oct 11, 2026
  3. added 2 commits that reference this issue on Oct 11, 2026
  4. adityamukho commented on Oct 11, 2026

    @adityamukho
    CollaboratorAuthor

    Fixed on v3 in PR #537 (200baca): open refuses with the new STG-047 when the WAL's base generation is newer than every meta page in the file. Stays open until v3.0.0 is released. No v2.x backport: v1 WALs record no base generation.

  5. adityamukho commented on Oct 14, 2026

    @adityamukho
    CollaboratorAuthor

    Fixed on v3 in #537 (open refuses with STG-047 when the WAL's base generation is newer than every meta page). Closed under the close-on-merge rule for v3 (#585). It stays listed in #421 for v2.x (no backport: v1 WALs record no base generation), with fix version v3.0.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdata-integrityData integrity issuedurabilityData durability concernknown-issueBug present in a published release and not yet fixed in one; listed in the pinned known-issues issue

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions