diff --git a/CHANGELOG.md b/CHANGELOG.md index fd60b305b..0add30393 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,14 +41,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Queries no longer lose or duplicate facts when more than 65,535 facts are waiting for a checkpoint (#445; also fixed in v2.0.3). - **A file shorter than its meta page count is refused at open with `STG-044`** (#497). Open never compared the meta page's `page_count` with the file's length. A truncated file (an interrupted copy, a partial download) opened, and its queries and checkpoints then failed with `INT-049` after the WAL had accepted writes. A meta with a `page_count` far past the end opened too: its next checkpoint wrote at that page id, leaving a sparse file of up to 4 EiB, and `verify()` scanned to that count and never finished. Open now refuses both, read-write and read-only, without modifying the file. It does not fall back to the other meta page, because a valid newer meta means that checkpoint happened. A checkpoint now fails with `INT-049` instead of writing a meta that counts pages the backend does not hold, and `verify()` reports a `page_count` past the end as `STG-044` and stops its page scan at the end of the file. `BrowserDb` is not checked: it keeps only live pages (#440), so its store has no length to compare. New code `STG-044`. - **A WAL from a later checkpoint is refused at open with `STG-047`** (#535). Opening an older `.graph` next to a `.wal` created after a later checkpoint, as happens when a backup is restored and the current WAL is left in place, replayed the WAL on top of the older file and reported success: the later transactions were applied and the ones between the two checkpoints were silently missing, and the next checkpoint made the gap permanent. The WAL's header records the generation it was created at; open now refuses, read-write and read-only, when that is newer than every meta page in the file, and leaves both files unmodified. A WAL with an older base (a crash between a checkpoint's commit and its WAL delete) replays as before. v2.x WALs record no generation, so v2.x cannot detect this. -- **A file shorter than one page is refused at open with `STG-045` unless it is a torn first write** (#506). Open treated any file under 4 KiB as a new database, because it holds no whole page, and a read-write open wrote a new empty database over it. A database cut below one page by an interrupted copy, or an unrelated file at the path, was silently replaced. Now such a file opens as a new database only when its bytes are zeros or a prefix of the first meta page a new database writes (what a crash during creation can leave, the rule a torn full page 0 already followed); anything else is `STG-045`, read-write and read-only, and the file is not modified. A zero-length file is still a new database. New code `STG-045`. +- **A file shorter than one page is refused at open with `STG-045` unless it is a torn first write** (#506). Open treated any file under 4 KiB as a new database, because it holds no whole page, and a read-write open wrote a new empty database over it. A database cut below one page by an interrupted copy, or an unrelated file at the path, was silently replaced. Now such a file opens as a new database only when its bytes are zeros or a prefix of the first meta page a new database writes (what a crash during creation can leave, the rule a torn full page 0 already followed); anything else is `STG-045`, read-write and read-only, and the file is not modified. A zero-length file is still a new database. New code `STG-045`. Also fixed in v2.0.5. - **A failed checkpoint is never retried on the same handle** (#390). A checkpoint that failed after it may have written (a failed write, a full disk, a failed `fsync`) could be retried by the next checkpoint. After a failed `fsync` the operating system may have dropped the written pages, and the checkpoint's meta page may have reached the disk although the call failed; a retry could then overwrite pages that meta references, and a crash during the retry left a file that failed to read with `STG-029`. Now every later write, checkpoint and index rebuild through that handle fails with the new `STG-046`, queries keep working, and the close-time checkpoint is skipped. Reopening the database recovers everything committed: it is all in the WAL. A `rebuild_indexes()` refused with `STG-041` writes nothing and does not count as failed. A fault-injection matrix now covers EIO, torn and short writes, a full disk and failed fsyncs at every write and sync of transacts, checkpoints, the close-time checkpoint and index rebuilds. -- **A commit made after a torn WAL entry is no longer lost on recovery** (#513). Recovery replays the WAL up to its first partial or damaged entry, but the writer appended after the end of the file. After a crash during an append, the next session's commits went in after the partial entry, so a second crash before any checkpoint lost them, although each had been fsynced and reported as committed. A failed append (short write, full disk, I/O error) did the same to the next commit through that handle. Opening a database for writing now cuts the WAL back to the end of its last valid entry, and syncs it, before anything is appended. After a failed append, every later write through that handle fails with the new `WAL-007` until the database is reopened or a `checkpoint()` succeeds. The transaction whose append failed is not applied; after a reopen it is either there in full (its entry reached the disk) or not at all. Affected every release since v0.5.0. +- **A commit made after a torn WAL entry is no longer lost on recovery** (#513). Recovery replays the WAL up to its first partial or damaged entry, but the writer appended after the end of the file. After a crash during an append, the next session's commits went in after the partial entry, so a second crash before any checkpoint lost them, although each had been fsynced and reported as committed. A failed append (short write, full disk, I/O error) did the same to the next commit through that handle. Opening a database for writing now cuts the WAL back to the end of its last valid entry, and syncs it, before anything is appended. After a failed append, every later write through that handle fails with the new `WAL-007` until the database is reopened or a `checkpoint()` succeeds. The transaction whose append failed is not applied; after a reopen it is either there in full (its entry reached the disk) or not at all. Affected every release since v0.5.0. Also fixed in v2.0.5. - **A v7 fact page with a damaged type byte fails the upgrade with `STG-014`** (#496). Opening a v7 file skipped, without an error, any page in its fact-page range whose type byte was not the fact-page type. The read-write open then upgraded the file to v8 and freed the old v7 pages, so that page's facts were lost for good; the read-only open returned the same answer with them missing. A v2.x writer fills that range with fact pages only, so any other type there is damage: open now fails with `STG-014`, read-write and read-only, and leaves the v7 file as it was. v7 fact pages have no checksum of their own, so damage elsewhere in a page can still decode to wrong facts or fail with another code; keep a copy of a v7 file until it has opened under v3.0.0. -- **Rule queries no longer fail with `INT-020` on databases with 1,000,000 or more live facts** (#528). The recursive evaluator counted the database's own facts against `max_results`, so once a database held that many live facts, every rule query failed at its first derived fact, whatever the size of its answer. `max_results` now counts only the facts the rules derive, so the `:max-results 3000000` workaround is no longer needed. Affected every release since v0.18.0. Also fixed on the v2.x line (`main`, #538), not yet released. +- **Rule queries no longer fail with `INT-020` on databases with 1,000,000 or more live facts** (#528). The recursive evaluator counted the database's own facts against `max_results`, so once a database held that many live facts, every rule query failed at its first derived fact, whatever the size of its answer. `max_results` now counts only the facts the rules derive, so the `:max-results 3000000` workaround is no longer needed. Affected every release since v0.18.0. Also fixed in v2.0.5 (#538). - Reopening next to a WAL whose entries were all already checkpointed no longer rewinds the transaction counter, so later transactions no longer reuse `tx_count` values and `:as-of` history stays correct (#447; also fixed in v2.0.3). -- `checkpoint()` removes a WAL whose entries are all already in the file (#457). When it had no facts to flush, it returned early and left the WAL until a later write was checkpointed. That WAL is left by a crash between a checkpoint's commit and its WAL delete, or by a failed delete (`WAL-006`). Its contents were always skipped on replay, so no data was affected; the sidecar file just stayed. Also fixed on the v2.x line (`main`, #540), not yet released. -- Closing a handle opened with `wal_checkpoint_threshold(usize::MAX)` no longer writes its pending facts to the `.graph` file (#476). The sentinel was documented to suppress the close-time checkpoint, and `Minigraf`'s close did skip it, but the storage layer still saved on drop, leaving a WAL whose entries were all checkpointed. The file now stays as the last explicit `checkpoint()` left it, and the WAL keeps every pending fact for the next open to replay, so bulk loaders and benchmarks see the pending WAL they ask for. No data is lost either way. Also fixed on the v2.x line (`main`, #540), not yet released. +- `checkpoint()` removes a WAL whose entries are all already in the file (#457). When it had no facts to flush, it returned early and left the WAL until a later write was checkpointed. That WAL is left by a crash between a checkpoint's commit and its WAL delete, or by a failed delete (`WAL-006`). Its contents were always skipped on replay, so no data was affected; the sidecar file just stayed. Also fixed in v2.0.5 (#540). +- Closing a handle opened with `wal_checkpoint_threshold(usize::MAX)` no longer writes its pending facts to the `.graph` file (#476). The sentinel was documented to suppress the close-time checkpoint, and `Minigraf`'s close did skip it, but the storage layer still saved on drop, leaving a WAL whose entries were all checkpointed. The file now stays as the last explicit `checkpoint()` left it, and the WAL keeps every pending fact for the next open to replay, so bulk loaders and benchmarks see the pending WAL they ask for. No data is lost either way. Also fixed in v2.0.5 (#540). - **A query on another thread sees each transaction whole** (#554). A commit wrote its WAL entry, then added the transaction's facts to the in-memory store one fact at a time, each under its own lock, so a query running on another thread could return part of the transaction. `execute("(transact ...)")`, `execute("(retract ...)")` and `WriteTransaction::commit` all did this. They now add the whole transaction under one lock, as does WAL replay. Stored data was never affected. Affected `WriteTransaction` since v0.5.0 and `execute` since v0.6.0. - **All patterns of one query read the same state of the database** (#555). A query whose patterns name entities or attributes read each of them with its own lookup, each under its own lock, so a transaction that committed between two lookups was seen by some patterns and not others, and the query could return rows that match no state of the database (an entity with one attribute but not another that was always written with it). The lookups now run under one lock. Stored data was never affected. Affected every release since v1.1.0.