Skip to content

durability: a commit appended after a torn WAL entry is lost on recovery #513

Closed

Description

@adityamukho

Problem

A WAL entry appended after a torn entry is lost on recovery, although its commit was fsynced and reported as successful.

WalReader::read_entries stops at the first entry that is truncated or fails its CRC. WalWriter::open_or_create_at seeks to the end of the file and appends there. So when the WAL ends in a partial entry, every entry written after it is unreachable on the next replay.

Two ways to get there:

  1. Crash during an append. A kill or power loss in the middle of append_entry leaves a partial entry. The next open replays the entries before it, then appends new commits after the partial bytes. If the process stops again before a checkpoint, those commits are gone on the next open.
  2. A failed append in a running process. write_all fails partway (short write, ENOSPC, EIO). The commit returns an error and is not applied, but the partial bytes stay in the file, and the next commit on the same handle is appended after them.

Found while working on #390.

Repro (v3 at 0aa813d and main at v2.0.4)

Three processes, each ending in std::process::exit(0) so nothing checkpoints:

  1. Open a new file, (transact [[:a :p/n 1]]), exit.
  2. Append 7 bytes to <db>.wal (a torn entry). Open: 1 fact. (transact [[:b :p/n 2]]) returns Ok. 2 facts. Exit.
  3. Open: 1 fact. The acknowledged commit from step 2 is lost.

Affected

Every release with the WAL, v0.5.0 through v2.0.4, file-backed databases only. A clean close checkpoints and deletes the WAL, so data is lost only when a second crash (or a kill) follows a first one before any checkpoint, or after a failed append in a running process.

Fix

Fix on v3, then a backport to main for a v2.x patch release.

Activity

  1. added this to the v3.0.0 milestone on Oct 10, 2026
  2. added
    bugSomething isn't working
    durabilityData durability concern
    known-issueBug present in a published release and not yet fixed in one; listed in the pinned known-issues issue
    on Oct 10, 2026
  3. added 3 commits that reference this issue on Oct 10, 2026
  4. adityamukho commented on Oct 10, 2026

    @adityamukho
    CollaboratorAuthor

    Fixed on v3 in #514 and on main (v2.x) in #516. The fault matrix from #390 (#515) also covers it. This issue stays open until the fix ships in a published release (the next v2.x patch).

  5. added a commit that references this issue on Oct 11, 2026
  6. adityamukho commented on Oct 11, 2026

    @adityamukho
    CollaboratorAuthor

    Fixed in v2.0.5 (released 2026-10-11, PR #516); also fixed on v3 (PR #514) for v3.0.0. Moved to "Fixed in v2.0.5" in #421.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdata-integrityData integrity issuedurabilityData durability concernknown-issueBug present in a published release and not yet fixed in one; listed in the pinned known-issues issue

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions