Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A file shorter than one page is refused at open with `STG-045` unless it is a torn first write (#506).** Open treated any file under 4 KiB as a new database and wrote a new header over it, so a database cut below one page by an interrupted copy, or an unrelated file at the path, was silently replaced. Such a file now opens as a new database only when its bytes are zeros or a prefix of the header a new database writes (what a crash during creation can leave); anything else is `STG-045` and the file is not modified. A zero-length file is still a new database. New code `STG-045`, numbered as in v3.0.0, which also fixes this.
- **A commit made after a torn WAL entry is no longer lost on recovery (#513).** Recovery replays the WAL up to its first partial or damaged entry, but the writer appended after the end of the file. After a crash during an append, the next session's commits went in after the partial entry, so a second crash before any checkpoint lost them, although each had been fsynced and reported as committed. A failed append (short write, full disk, I/O error) did the same to the next commit through that handle. Opening a database for writing now cuts the WAL back to the end of its last valid entry, and syncs it, before anything is appended. After a failed append, every later write through that handle fails with the new `WAL-007` until the database is reopened or a `checkpoint()` succeeds. The transaction whose append failed is not applied; after a reopen it is there in full or not at all. Affected every release since v0.5.0. Also fixed in v3.0.0.
- **Rule queries no longer fail with `INT-020` on databases with 1,000,000 or more live facts (#528).** The recursive evaluator counted the database's own facts against `max_results`, so once a database held that many live facts, every rule query failed at its first derived fact, whatever the size of its answer. `max_results` now counts only the facts the rules derive, so raising it with `OpenOptions::max_results` or `:max-results` is no longer needed. Affected every release since v0.18.0. Also fixed in v3.0.0.
- `checkpoint()` removes a WAL whose entries are all already in the file (#457). When it had no facts to flush, it returned early and left the WAL until a later write was checkpointed. That WAL is left by a crash between a checkpoint's commit and its WAL delete, or by a failed delete (`WAL-006`). Its contents were always skipped on replay, so no data was affected; the sidecar file just stayed. Also fixed in v3.0.0.
- Closing a handle opened with `wal_checkpoint_threshold(usize::MAX)` no longer writes its pending facts to the `.graph` file (#476). The sentinel was documented to suppress the close-time checkpoint, and `Minigraf`'s close did skip it, but the storage layer still saved on drop, leaving a WAL whose entries were all checkpointed. The file now stays as the last explicit `checkpoint()` left it, and the WAL keeps every pending fact for the next open to replay, so bulk loaders and benchmarks see the pending WAL they ask for. No data is lost either way. Also fixed in v3.0.0.

### Tests

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.

## Test Coverage

**1238 tests** (1229 passing, 9 ignored; unit + integration + doc).
**1241 tests** (1232 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1238 tests
cargo test # run 1241 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

**Verified**: 2026-10-10 with `cargo test`

**Result**: 1,229 passing tests and 9 ignored tests (1,238 total), including the golden-file corpus (#391)
**Result**: 1,232 passing tests and 9 ignored tests (1,241 total), including the golden-file corpus (#391)

Since v2.0.4, `tests/edge_cases_test.rs` checks that a file of 1 to 4,095 bytes is `STG-045`, unchanged and with no WAL, when it is a database cut below one page or foreign bytes, and opens as a new database that takes a write and reads it back after reopen when it is a prefix of the initial header or all zeros (#506).

Expand All @@ -16,7 +16,7 @@ Since v2.0.4, torn-WAL-tail tests (#513) check that opening a WAL writer cuts of

Not counted above: the `BrowserDb` tests run only on wasm32 with `wasm-bindgen-test` in a browser. Two of them check that the IndexedDB flush returns every page a write dirtied, and that a dirty page that cannot be read is an error rather than left out.

The v2.0.4 suite covers the public database API, the Datalog engine, storage and recovery paths, and the v2.0.0 locking and structured-error-code changes, plus v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Since v2.0.1, `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a cyclic leaf chain or damaged slot directory is rejected rather than copied (#315). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389). Disjunction tests check that `or-join` returns an empty result, not INT-031, when no rows reach it: on an empty database, after a pattern that matches nothing, after a `not` that removes every row, and in a rule body (#405). A stratification unit test builds the same two-rule negative cycle 64 times and checks that the INT-054 message names the same predicate pair every time (#410). Magic-sets tests check that a recursive rule with a literal start that recurses through another rule returns the same results as without the literal (#297). API tests check that `execute()` and `WriteTransaction::execute()` reject `$slot` bind tokens in every position with API-010 (#407). The REPL `run()` test now runs in a child process with null stdin, so it no longer hangs when the test runner's stdin is an open non-TTY stream (#415). A fault-injection test crashes `save()` at each of its 47 page writes and syncs and checks that every reopen finds exactly the old or the new checkpointed facts (#374). For v2.0.3, storage unit tests and two `edge_cases_test.rs` tests check that more than 65,535 uncheckpointed facts each resolve to their own fact through entity lookups and attribute scans, in memory and file-backed, before and after a checkpoint, and that the widened `FactRef` encodes to the same bytes as before (#445). A WAL test reopens a database next to a WAL whose entries are all checkpointed, writes again, and checks that `:as-of` history is unchanged before and after another reopen; a unit test checks that restoring the transaction counter never lowers it (#447). Since v2.0.3, `tests/golden_corpus_test.rs` opens a copy of each committed golden file in `tests/golden/` (v7 files written by minigraf 2.0.3: one checkpoint, several checkpoints, indexes rebuilt on open, a crash-left WAL, an already-checkpointed WAL, same-transaction multi-values) and checks its manifest's CRC32, `tx_count` and query results after open, after a checkpoint and reopen, and after one more write; a second test checks that every golden file has a manifest (#391). A recursive-rules test runs rule queries on a database with more live facts than `max_results` and checks that they answer, and that the limit still stops a rule that derives more facts than it allows (#528).
The v2.0.4 suite covers the public database API, the Datalog engine, storage and recovery paths, and the v2.0.0 locking and structured-error-code changes, plus v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Since v2.0.1, `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a cyclic leaf chain or damaged slot directory is rejected rather than copied (#315). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389). Disjunction tests check that `or-join` returns an empty result, not INT-031, when no rows reach it: on an empty database, after a pattern that matches nothing, after a `not` that removes every row, and in a rule body (#405). A stratification unit test builds the same two-rule negative cycle 64 times and checks that the INT-054 message names the same predicate pair every time (#410). Magic-sets tests check that a recursive rule with a literal start that recurses through another rule returns the same results as without the literal (#297). API tests check that `execute()` and `WriteTransaction::execute()` reject `$slot` bind tokens in every position with API-010 (#407). The REPL `run()` test now runs in a child process with null stdin, so it no longer hangs when the test runner's stdin is an open non-TTY stream (#415). A fault-injection test crashes `save()` at each of its 47 page writes and syncs and checks that every reopen finds exactly the old or the new checkpointed facts (#374). For v2.0.3, storage unit tests and two `edge_cases_test.rs` tests check that more than 65,535 uncheckpointed facts each resolve to their own fact through entity lookups and attribute scans, in memory and file-backed, before and after a checkpoint, and that the widened `FactRef` encodes to the same bytes as before (#445). A WAL test reopens a database next to a WAL whose entries are all checkpointed, writes again, and checks that `:as-of` history is unchanged before and after another reopen; a unit test checks that restoring the transaction counter never lowers it (#447). Since v2.0.3, `tests/golden_corpus_test.rs` opens a copy of each committed golden file in `tests/golden/` (v7 files written by minigraf 2.0.3: one checkpoint, several checkpoints, indexes rebuilt on open, a crash-left WAL, an already-checkpointed WAL, same-transaction multi-values) and checks its manifest's CRC32, `tx_count` and query results after open, after a checkpoint and reopen, and after one more write; a second test checks that every golden file has a manifest (#391). A recursive-rules test runs rule queries on a database with more live facts than `max_results` and checks that they answer, and that the limit still stops a rule that derives more facts than it allows (#528). A WAL test restores a WAL whose entries are all checkpointed and checks that an explicit `checkpoint()` removes it, keeps the fact, and that the next write does not reuse tx 1; the golden-corpus test now requires `v7_stale_wal` to have no WAL after a checkpoint too (#457). A WAL test checks that dropping a handle opened with `wal_checkpoint_threshold(usize::MAX)` leaves the `.graph` file byte for byte unchanged under `Full` and `Normal` sync, keeps the WAL, and that a reopen replays it (#476).

### Covered Areas

Expand Down
37 changes: 36 additions & 1 deletion src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,11 @@ pub struct OpenOptions {
/// A checkpoint's cost grows with the total size of the database (see
/// [`Minigraf::checkpoint`]), while durability does not depend on it: the WAL
/// is crash-durable. Raise this for write-heavy workloads on large graphs.
///
/// `usize::MAX` turns off every implicit checkpoint: no auto-checkpoint and
/// nothing written to the `.graph` file on close. Only an explicit
/// [`Minigraf::checkpoint`] writes it; until then the WAL holds the pending
/// facts, and the next open replays them.
pub wal_checkpoint_threshold: usize,
/// Number of pages to hold in the LRU page cache. Default: 256 (= 1MB at 4KB pages).
///
Expand Down Expand Up @@ -305,8 +310,14 @@ impl Drop for Inner {
// On clean close, perform a best-effort checkpoint to reduce WAL size.
// Errors are silently ignored (can't propagate from Drop).
// Skip if wal_checkpoint_threshold is usize::MAX — that sentinel suppresses
// all checkpointing (used by benchmarks to keep WAL entries pending).
// all checkpointing (used by benchmarks to keep WAL entries pending),
// including the storage's own save on drop (#476). The WAL holds every
// pending fact, and the next open replays it.
if self.options.wal_checkpoint_threshold == usize::MAX {
#[cfg(not(target_arch = "wasm32"))]
if let Ok(WriteContext::File { pfs, .. }) = self.write_lock.get_mut() {
pfs.discard();
}
return;
}
if let Ok(mut ctx) = self.write_lock.lock() {
Expand Down Expand Up @@ -759,7 +770,18 @@ impl Minigraf {
// second handle in this one — #304/#314). The guard still matters where
// the filesystem cannot lock (e.g. NFSv3 without lockd, some FUSE
// mounts) and the caller set `allow_unlocked`.
//
// A WAL can still exist here with every entry already in the file
// (a crash between a checkpoint's commit and its WAL delete, or a
// failed delete): replay counts only the entries it applies. Each
// of those entries is at or below the committed tx count, so
// removing the file loses nothing (#457).
if *wal_entry_count == 0 && !pfs.is_dirty() {
let wal_path = Self::wal_path_for(db_path);
if wal_path.exists() {
*wal = None;
WalWriter::delete_file(&wal_path)?;
}
return Ok(());
}
// `force_dirty` is needed for the WAL-replay case: facts were loaded
Expand Down Expand Up @@ -1879,6 +1901,19 @@ mod tests {

// ── file-backed: sync mode is plumbed through to both WalWriter call sites ──

/// #476: an in-memory handle under the `usize::MAX` sentinel closes
/// without a checkpoint and without touching any file storage.
#[test]
fn test_max_threshold_in_memory_drop() {
let db = Minigraf::in_memory_with_options(
OpenOptions::new().wal_checkpoint_threshold(usize::MAX),
)
.unwrap();
db.execute("(transact [[:e0 :a0 1]])").unwrap();
assert_eq!(db.current_tx_count(), 1, "write applied");
drop(db);
}

#[test]
fn test_normal_sync_mode_reaches_both_wal_writer_call_sites() {
let dir = tempfile::tempdir().unwrap();
Expand Down
8 changes: 8 additions & 0 deletions src/storage/persistent_facts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1084,6 +1084,14 @@ impl<B: StorageBackend + 'static> PersistentFactStorage<B> {
self.dirty
}

/// Forget unsaved changes, so dropping this storage commits nothing. Used
/// on close under `wal_checkpoint_threshold(usize::MAX)`, whose WAL keeps
/// the pending facts (#476).
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn discard(&mut self) {
self.dirty = false;
}

/// Run a closure with read access to the underlying storage backend.
///
/// Used by the browser WASM layer to read pages after `save()` without
Expand Down
6 changes: 3 additions & 3 deletions tests/golden_corpus_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -171,9 +171,9 @@ fn check_manifest(m: &Manifest, failures: &mut Vec<String>) {
db.checkpoint().unwrap();
}

// 4. Persist: reopen after the checkpoint. v2.x leaves a WAL whose entries
// were all checkpointed already (`v7_stale_wal`) in place; v3 removes it.
if READER >= 3 && wal_path(&path).exists() {
// 4. Persist: reopen after the checkpoint, which removes the WAL, including
// one whose entries are all checkpointed already (`v7_stale_wal`, #457).
if wal_path(&path).exists() {
failures.push(format!("{}: WAL still present after checkpoint", m.name));
}
{
Expand Down
Loading
Loading