diff --git a/CHANGELOG.md b/CHANGELOG.md index 872fe42b83..641d71f908 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened; while one is open, checkpoints are deferred (writes stay in the WAL, automatic checkpoints wait, and `checkpoint()` or `rebuild_indexes()` with work to do fail with the new `API-013`). Close, drop or finish the log to release it. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow. +- **Read-only open: `OpenOptions::read_only`** (#429). Opens a database so that nothing done through the handle writes to the `.graph` file or its WAL. The file is locked in shared mode: any number of read-only handles, in this process or others, can hold it at once, and they exclude a read-write open (and it excludes them) with the existing STG-025/STG-026. The handle shows the same data as a read-write open: a WAL left by an earlier session is applied in memory but not checkpointed or deleted, and a format v7 file is read into memory instead of migrated. A missing file is the new `STG-042` (nothing is created). `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` fail with the new `API-014`; queries, cursors, `fact_log`, `verify` and rule registration work, and dropping the handle does not checkpoint. `page_cache_size` applies as usual, so a tool can keep the whole file resident. Rust API only for now; the bindings follow. - **Cursor API: `Minigraf::query()` and `PreparedQuery::query()`** (#432). Both return a `Cursor` that hands a query's rows back in batches: `vars()`, `next_batch(max_rows) -> Result, _>`, `close()`, and an iterator over rows. The answer is fixed when the cursor opens, so writes that commit while it is open do not change it. A cursor borrows nothing from the database handle: it can outlive it and move to another thread. For now the answer is still computed when the cursor opens, under the same limits as `execute()`; the API is shaped so the engine can later produce rows as they are pulled without breaking callers. `query()` rejects `transact`, `retract` and `rule` with the new `API-012`, and a query with bind slots with `API-010`. `execute()` is unchanged. Rust API only for now; the bindings follow. - **`Minigraf::verify()` and `Minigraf::rebuild_indexes()`** (#373). Page checksums turn a torn or rotted page into an error, but they cannot see a page that is valid and says the wrong thing. `verify()` walks every committed page and returns an `IntegrityReport` with one structured error per finding: an index that holds different facts than EAVT or a different count than the meta page (new `STG-038`), keys out of order or a page reached twice (new `STG-039`), dictionary maps that disagree (new `STG-040`), ids with no dictionary entry (`STG-036`), and leaked, doubly used or wrongly free pages (`STG-035`). It keeps no per-fact state: indexes are compared by an order-independent digest. `rebuild_indexes()` rebuilds EAVT, AEVT, AVET and VAET from an intact index that agrees with another one (or the only intact one), derives the free list again, and commits like a checkpoint, including uncheckpointed writes; a crash at any point leaves the previous checkpoint. If no index can serve as the source, or the dictionary is damaged, it fails with new `STG-041` and writes nothing. Both are Rust API only for now; the bindings follow. diff --git a/CLAUDE.md b/CLAUDE.md index 32b3b4f3e2..98780a9887 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -138,7 +138,7 @@ cargo run < demos/demo_negation.txt 5. **`src/repl.rs`** — Interactive REPL; TTY-aware (suppresses prompts/banner for piped input) -6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size` +6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`, `OpenOptions::read_only` (#429: shared lock, nothing written, writes are `API-014`) - `src/cursor.rs`: `Cursor` / `Batch` — owned, `Send` cursor returned by `Minigraf::query` and `PreparedQuery::query` (#432); answer computed at open today, shaped for streaming operators - `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation, deferring checkpoints (`API-013`) @@ -196,7 +196,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are ## Test Coverage -**1309 tests** (1300 passing, 9 ignored; unit + integration + doc). +**1329 tests** (1320 passing, 9 ignored; unit + integration + doc). See `docs/TEST_COVERAGE.md` for the full per-file breakdown. **Testing conventions** — see the Testing Conventions section below before writing any tests. diff --git a/README.md b/README.md index 1cdc5c9d43..bbff5e3da7 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,10 @@ while let Some(batch) = log.next_batch(1000)? { for rec in batch { /* rec.tx_count, rec.entity, rec.attribute, rec.value, rec.asserted */ } } +// Read-only open — for backups and offline tools. Read-only handles share a file; +// a read-write handle (like `db` on myapp.graph) excludes them. +let archive = Minigraf::open_with_options("archive.graph", OpenOptions::new().read_only(true))?; + // Explicit transaction — all-or-nothing let mut tx = db.begin_write()?; tx.execute(r#"(transact [[:alice :person/age 31]])"#)?; @@ -117,7 +121,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit ```bash cargo run # interactive Datalog REPL -cargo test # run 1309 tests +cargo test # run 1329 tests cargo run < demos/demo_recursive.txt # recursive rules demo ``` diff --git a/ROADMAP.md b/ROADMAP.md index a427d680d8..b93cd58cab 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -68,7 +68,8 @@ Scope — format and storage integrity: Scope — query API: - Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`, with the current evaluator computing the answer at open). The binding cursor objects follow. Streaming operators replace the evaluator behind the cursor in 3.x without further API changes; how operators bound their memory (spill, scratch pages in the `.graph` file, or a clear error) is still open. -- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). Read-only open (#429) and a raw log writer (#431) complete the extract-transform-load path; the binding objects follow with #462. +- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462. +- Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows. Scope — release-gate testing and documentation: diff --git a/docs/ERROR_REFERENCE.md b/docs/ERROR_REFERENCE.md index b29d281c9c..95d2bc29f1 100644 --- a/docs/ERROR_REFERENCE.md +++ b/docs/ERROR_REFERENCE.md @@ -150,6 +150,7 @@ with no `CodedError` anywhere in its chain. | STG-039 | Malformed tree | Storage | | STG-040 | Dictionary inconsistency | Storage | | STG-041 | Cannot rebuild indexes | Storage | +| STG-042 | Database file not found (read-only open) | Storage | | WAL-001 | Invalid WAL magic number | WAL | | WAL-002 | Unsupported WAL version | WAL | | WAL-003 | Value size exceeds maximum | WAL | @@ -169,6 +170,7 @@ with no `CodedError` anywhere in its chain. | API-011 | Two valid-time windows for one fact in one transaction | Database API | | API-012 | Non-query command passed to query() | Database API | | API-013 | Checkpoint while a fact log is open | Database API | +| API-014 | Write on a read-only handle | Database API | | INT-000 | Unclassified internal error | Internal | | INT-001 | WriteTransaction already in progress on this thread | Internal | | INT-002 | Invalid entity (API layer) | Internal | @@ -225,6 +227,7 @@ with no `CodedError` anywhere in its chain. | INT-053 | Header checksum mismatch: possible file corruption | Internal | | INT-054 | Unstratifiable negative recursion cycle | Internal | | INT-055 | Rule predicate disappeared during rollback | Internal | +| INT-056 | Write to a read-only storage backend | Internal | Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. They stay registered and documented, and error codes are never removed or reused. @@ -1974,6 +1977,8 @@ See the [file format section in README](../README.md#file-format) for version hi - Reuse the handle you already have. `Minigraf` is cheap to clone and all clones share one database. - If you cannot find the other handle, it is usually held by a longer-lived object than you expect — a cache, a registry, or a background task. +Read-only handles (`OpenOptions::read_only`) share a file with each other, but not with a read-write handle: either kind refuses the other with this error. + **Scenario**: A request handler calls `Minigraf::open` per request while a connection pool already holds the same path open. --- @@ -1989,6 +1994,8 @@ See the [file format section in README](../README.md#file-format) for version hi - There is no lock file to delete. If you find a stale `.graph.lock`, it is a leftover from a version before 2.0 and has no effect — it is not read or deleted, and should not be removed by hand in case an old process still depends on it. - If two services genuinely need the same file, put one in front of the other. Minigraf is embedded, not a server. +A read-only handle (`OpenOptions::read_only`) in another process also refuses a read-write open, and a read-write handle refuses a read-only one. Read-only handles do not refuse each other. + **Scenario**: A rolling deployment starts the new pod before the old one has exited, and both mount the same volume. --- @@ -2180,6 +2187,20 @@ See the [file format section in README](../README.md#file-format) for version hi --- +### STG-042 Database file not found (read-only open) + +**Error text**: `database file not found ({}); a read-only open does not create one` + +**Cause**: The database was opened with `OpenOptions::read_only` and no file exists at the path. A read-write open creates a new, empty database; a read-only open never creates or writes anything. + +**Resolution**: +- Check the path. The `{}` is the path as given. +- To create the database, open it once without `read_only`. + +**Scenario**: A migration tool is pointed at `memory.graph` from the wrong working directory. + +--- + ## WAL — Write-Ahead Log Errors WAL errors relate to the sidecar `.wal` file written alongside the `.graph` file. @@ -2492,6 +2513,24 @@ db.checkpoint()?; **Scenario**: A backup job streams the fact log on a thread while the application calls `checkpoint()` on a timer. +### API-014 Write on a read-only handle + +**Error text**: `database is open read-only; {} is not allowed` + +**Cause**: The database was opened with `OpenOptions::read_only`, and the call would write to the file or its WAL. The `{}` names the call: `transact`, `retract`, `begin_write`, `checkpoint` or `rebuild_indexes`. Nothing is written and no transaction number is used. + +**Resolution**: +- Open the database without `read_only` to write. A read-write open waits for no one: it fails with STG-025 or STG-026 while any read-only handle holds the file, so close those first. +- Queries, cursors, `fact_log()`, `verify()` and rule registration work on a read-only handle. + +```rust +let src = Minigraf::open_with_options("old.graph", OpenOptions::new().read_only(true))?; +let mut log = src.fact_log(&FactFilter::new())?; +// ... copy the records elsewhere; src.execute("(transact ...)") would be API-014 +``` + +**Scenario**: An offline migration tool opens its source read-only and, by mistake, runs a cleanup `retract` against it instead of against the destination. + --- ## INT — Internal Errors @@ -3189,3 +3228,12 @@ this transaction, but the rule registry no longer contained it. **Resolution**: - File a bug report — this indicates a lifecycle bug in `WriteTransaction`'s rule-staging/rollback logic. + +### INT-056 Write to a read-only storage backend + +**Error text**: `write to a read-only storage backend: {}` + +**Cause**: A page write or sync reached the file backend of a handle opened with `OpenOptions::read_only`. The public API refuses every write on such a handle first (API-014), so this is a missed check, not a usage error. The `{}` is the page or `sync`. Nothing was written. + +**Resolution**: +- File a bug report with the call that returned it. diff --git a/docs/TEST_COVERAGE.md b/docs/TEST_COVERAGE.md index 248dad91e2..9e3e88cfd1 100644 --- a/docs/TEST_COVERAGE.md +++ b/docs/TEST_COVERAGE.md @@ -4,9 +4,9 @@ **Verified**: 2026-10-07 with `cargo test --quiet` -**Result**: 1,300 passing tests and 9 ignored tests (1,309 total) +**Result**: 1,320 passing tests and 9 ignored tests (1,329 total) -This covers the unreleased file-format-v8 work (#287, #371): the index-key value-collision fix, the v7→v8 migration, and the drop of formats v1–v6. It otherwise still covers the public database API, the Datalog engine, storage and recovery paths, v2.0.0's locking and structured-error-code changes, and v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Merged from the v2.x line: `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a damaged slot directory is rejected rather than copied (#315). B+tree cursor tests check that every scan walks the tree, not the leaf `next_leaf` chain: full scans and start positions match the input, `seek` from the current position lands where a fresh descent from the root would (randomized, plus within a leaf, to a sibling, across subtrees, past the end, into the gap after a leaf, and backwards as a no-op), a seek to the sibling leaf reads exactly one page, reads are unchanged with every `next_leaf` zeroed, a child pointer into a non-B+tree page is STG-013, and a cycle in child pointers is an error, not a hang (v8 storage format PR 1, #374). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389). Disjunction tests check that `or-join` returns an empty result, not INT-031, when no rows reach it: on an empty database, after a pattern that matches nothing, after a `not` that removes every row, and in a rule body (#405). A stratification unit test builds the same two-rule negative cycle 64 times and checks that the INT-054 message names the same predicate pair every time (#410). Magic-sets tests check that a recursive rule with a literal start that recurses through another rule returns the same results as without the literal (#297). API tests check that `execute()` and `WriteTransaction::execute()` reject `$slot` bind tokens in every position with API-010 (#407). The REPL `run()` test now runs in a child process with null stdin, so it no longer hangs when the test runner's stdin is an open non-TTY stream (#415). v8 page-format tests (PR 2, #374, #388) check the 24-byte page header (a flipped bit, a wrong page id and a future generation each give their own STG code, legacy page types fail before the CRC, and a failed page is never cached), meta-page encoding, the free-list chain, WAL v2's base generation, and every row of the meta-selection table: both slots valid, a torn newest slot with the WAL's base generation (opens at the older one), a WAL newer than the valid meta, pages of the next generation on the free list or past `page_count` with no WAL (all STG-033, with the file unmodified), an unknown feature bit (STG-034), and a pre-release v8 single header (STG-032). Fault-injection tests crash `save()` at every page write (cleanly and with torn pages, including the meta write) and every sync, and crash a v7 migration at every write; each reopen finds exactly the old or the new checkpoint, and every reachable page plus the free list covers the file exactly. Twenty-five consecutive saves never write a page the previous meta references. Covering-key tests (PR 3, #433) check that byte order equals logical order for every key component and value type on random tuples (negative, zero and large integers, NaN and ±0.0, strings with 0x00, 64- and 65-byte strings, FOREVER, newest-transaction-first) and that every component round-trips; that prefix-compressed leaves and shortest-separator internal nodes round-trip, agree with a linear scan, keep `max(left) < s ≤ min(right)`, and reject an overflow; that the byte-key B+tree's builds, incremental rebuilds, cursor and `seek` match their expected sets; that dictionary ids are assigned once, never reused, shared between a keyword attribute and value, and deterministic; that a re-asserted or retracted long value writes no new value page and a forced hash collision still finds the right value; that one transaction with two timestamps is STG-037 and a missing dictionary entry STG-036; that committed reads by entity, entity and attribute, attribute, and in full equal an in-memory model across eight checkpoints and a reopen; that short-valued reads touch no value page; that a v7 file with every value type, long strings, refs and retractions migrates to exactly the same facts; that a pre-covering-key v8 file is STG-032; and that string values over 4,068 bytes and attributes or keywords over 1,024 bytes are WAL-003. A WAL test checks that reopening next to a fully checkpointed WAL does not reuse transaction numbers (#447). Copy-on-write checkpoint tests (PR 4, #434) check that lazy free-list pops and pushes conserve every page across page boundaries, partial head pages and chain exhaustion, never hand out a page freed in the same generation, and share unread chain pages unchanged; that a copy-on-write insert equals a sorted merge for random batches (including keys below the first and above the last, many into one leaf, root growth to depth 3, and 1 KB separators that split internal nodes), leaves the old tree readable and unchanged, frees exactly the replaced nodes, writes only new pages, writes exactly one leaf and two internal nodes for one entry in a depth-3 tree, and does not fragment leaves under single inserts; that a checkpoint after 1 or 100 right-edge facts writes at most the tree-depth difference more pages at 100,000 facts than at 10,000, and at most one path per new entry per tree for random keys; and that four reader threads querying throughout 40 checkpoints never fail and never see the fact count go backwards. Hardening tests (PR 5) damage an index leaf, an internal node, a dictionary leaf, a value page and a free-list page in turn. Every committed read or checkpoint that reaches the damaged page fails with STG-029, never with wrong or fewer rows. A query through `Minigraf` on a file with damaged leaves fails with STG-029, and the file still opens at its last checkpoint. A size test measures at most 200 bytes per fact (142 measured) on 144k facts in #433's shape; its 1.15M-fact variant is ignored locally and runs in the nightly smoke workflow. A shared dictionary-cache test checks that cached ident and timestamp lookups return the same facts and that the cache stays bounded. Merged from the v2.x line: storage and `edge_cases_test.rs` tests check that more than 65,535 uncheckpointed facts each resolve to their own fact through entity lookups and attribute scans, in memory and file-backed (#445). Merged from the v2.x line: `tests/golden_corpus_test.rs` opens a copy of each committed golden file in `tests/golden/` (v7 files written by minigraf 2.0.3: one checkpoint, several checkpoints, indexes rebuilt on open, a crash-left WAL, an already-checkpointed WAL, same-transaction multi-values) and checks its manifest's CRC32, `tx_count` and query results after open, after a checkpoint and reopen, and after one more write; a second test checks that every golden file has a manifest (#391). On `v3` the same test checks that each v7 file has a v8 meta page after its first open and gives the same results after reopening the migrated file, and adds seven v8 golden files (one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values at the size limits); the #371 queries, skipped on v2.x, run and pass. Integrity tests (#373) check that intact files verify with no problems and an exact fact and page count, and that `verify` reports each kind of damage with its code: an EAVT, AEVT, AVET or VAET leaf missing an entry (STG-038), keys out of order and a page reached from two trees (STG-039), a rotted index leaf (STG-029), a free page still in use and a leaked page (STG-035), a missing transaction entry (STG-036), and an entity id whose UUID maps to another id (STG-040). After each index or free-list damage, `rebuild_indexes` leaves a file that verifies clean, reads back every fact and accounts for every page. The source is an index that agrees with another one, or the only intact one; three intact indexes that all disagree, or a damaged dictionary, give STG-041 with the file unchanged. A rebuild commits pending facts, including new ids and long values. A crash at every write (clean and torn) and every sync of a rebuild reopens either at the old generation with only its old damage or at the new one with none, never losing a fact. Through `Minigraf` (`tests/integrity_test.rs`), a rebuild commits WAL entries and deletes the WAL, and flipping a byte in each leaf of a file in turn is always reported, then either repaired (index leaves) or refused with STG-041 and the file unchanged (dictionary leaves). Live-read tests (#379) check that `triple_len` ends exactly where `tx↓` begins for every value type and index; that on random histories (several valid-time windows, retractions, re-assertions, several values per attribute, long values, 12 transactions per checkpoint) the committed live read for every scan kind and `:as-of` point is exactly the net-asserted set, that it stays equivalent with pending facts on top, and after reopen; and that a point read on a triple asserted and retracted 2,000 times reads under a third of the pages a history read does, with or without `:as-of`. `tests/live_reads_test.rs` runs the same history in a checkpointed file database and in memory and checks that point, attribute, full-scan, join, valid-time, `:as-of` and rule queries return the same rows, before and after reopen. One-window tests (#435) check that `net_asserted_facts` keeps each triple's newest transaction only (a later window replaces an earlier one, closing an open window, a retraction in the newest transaction hides it, records with two windows in one transaction keep both, once each) against a per-triple model on random histories; that the on-disk key walk agrees at every `:as-of` before and after reopen; and that `API-011` rejects two windows of one fact in one transaction without taking a `tx_count`. `tests/valid_time_window_test.rs` checks replacing, closing, extending, reopening and retract-then-reassert through entity-bound and attribute queries in memory, on a checkpointed file and after reopen, and `API-011` from `transact` and from `WriteTransaction::commit`. Cursor tests (#432, `tests/cursor_test.rs`) check that `query()` batches partition the `execute()` answer for several batch sizes (never empty, never over `max_rows`, `max_rows == 0` still progresses, `None` stays `None`), that the iterator and aggregates, rules, `:as-of` and `:any-valid-time` match `execute()`, that a cursor opened before a `transact`, a `WriteTransaction` commit, a retract or a checkpoint returns the pre-write answer to the end (in memory and file-backed), that `transact`/`retract`/`rule` are `API-012` and bind slots `API-010`, that a `WriteTransaction` thread gets `INT-001`, that rule limits give `INT-020` as from `execute()`, that `PreparedQuery::query` matches `PreparedQuery::execute`, and that a cursor is `Send` and outlives its database handle. Fact-log tests (#430, `tests/fact_log_test.rs`) check that `fact_log()` returns exactly the records written (assertions and retractions, with `tx_count` and `tx_id`) in memory, with everything pending, half checkpointed, fully checkpointed and after reopen, in both orders, with windows of 1 to 1,000 records and batch sizes of 1 to 1,000, in non-decreasing `tx_count` in Tx order; that every layout gives identical records, including valid-time windows and long string values; that a transaction larger than the window is split with no loss or duplicate; that attribute, attribute-prefix, entity and tx-range filters and their combinations match a model, and unknown names match nothing; that writes after open are not in the log; that `checkpoint()` and `rebuild_indexes()` are `API-013` while a log is open (a checkpoint with nothing to write is allowed), that automatic checkpoints from `execute()` and `WriteTransaction::commit` wait for the log, and that closing, dropping or finishing the log releases the database; that a `WriteTransaction` thread gets `INT-001`; and that a log is `Send`, outlives its database handle, and the deferred writes replay on the next open. +This covers the unreleased file-format-v8 work (#287, #371): the index-key value-collision fix, the v7→v8 migration, and the drop of formats v1–v6. It otherwise still covers the public database API, the Datalog engine, storage and recovery paths, v2.0.0's locking and structured-error-code changes, and v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Merged from the v2.x line: `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a damaged slot directory is rejected rather than copied (#315). B+tree cursor tests check that every scan walks the tree, not the leaf `next_leaf` chain: full scans and start positions match the input, `seek` from the current position lands where a fresh descent from the root would (randomized, plus within a leaf, to a sibling, across subtrees, past the end, into the gap after a leaf, and backwards as a no-op), a seek to the sibling leaf reads exactly one page, reads are unchanged with every `next_leaf` zeroed, a child pointer into a non-B+tree page is STG-013, and a cycle in child pointers is an error, not a hang (v8 storage format PR 1, #374). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389). Disjunction tests check that `or-join` returns an empty result, not INT-031, when no rows reach it: on an empty database, after a pattern that matches nothing, after a `not` that removes every row, and in a rule body (#405). A stratification unit test builds the same two-rule negative cycle 64 times and checks that the INT-054 message names the same predicate pair every time (#410). Magic-sets tests check that a recursive rule with a literal start that recurses through another rule returns the same results as without the literal (#297). API tests check that `execute()` and `WriteTransaction::execute()` reject `$slot` bind tokens in every position with API-010 (#407). The REPL `run()` test now runs in a child process with null stdin, so it no longer hangs when the test runner's stdin is an open non-TTY stream (#415). v8 page-format tests (PR 2, #374, #388) check the 24-byte page header (a flipped bit, a wrong page id and a future generation each give their own STG code, legacy page types fail before the CRC, and a failed page is never cached), meta-page encoding, the free-list chain, WAL v2's base generation, and every row of the meta-selection table: both slots valid, a torn newest slot with the WAL's base generation (opens at the older one), a WAL newer than the valid meta, pages of the next generation on the free list or past `page_count` with no WAL (all STG-033, with the file unmodified), an unknown feature bit (STG-034), and a pre-release v8 single header (STG-032). Fault-injection tests crash `save()` at every page write (cleanly and with torn pages, including the meta write) and every sync, and crash a v7 migration at every write; each reopen finds exactly the old or the new checkpoint, and every reachable page plus the free list covers the file exactly. Twenty-five consecutive saves never write a page the previous meta references. Covering-key tests (PR 3, #433) check that byte order equals logical order for every key component and value type on random tuples (negative, zero and large integers, NaN and ±0.0, strings with 0x00, 64- and 65-byte strings, FOREVER, newest-transaction-first) and that every component round-trips; that prefix-compressed leaves and shortest-separator internal nodes round-trip, agree with a linear scan, keep `max(left) < s ≤ min(right)`, and reject an overflow; that the byte-key B+tree's builds, incremental rebuilds, cursor and `seek` match their expected sets; that dictionary ids are assigned once, never reused, shared between a keyword attribute and value, and deterministic; that a re-asserted or retracted long value writes no new value page and a forced hash collision still finds the right value; that one transaction with two timestamps is STG-037 and a missing dictionary entry STG-036; that committed reads by entity, entity and attribute, attribute, and in full equal an in-memory model across eight checkpoints and a reopen; that short-valued reads touch no value page; that a v7 file with every value type, long strings, refs and retractions migrates to exactly the same facts; that a pre-covering-key v8 file is STG-032; and that string values over 4,068 bytes and attributes or keywords over 1,024 bytes are WAL-003. A WAL test checks that reopening next to a fully checkpointed WAL does not reuse transaction numbers (#447). Copy-on-write checkpoint tests (PR 4, #434) check that lazy free-list pops and pushes conserve every page across page boundaries, partial head pages and chain exhaustion, never hand out a page freed in the same generation, and share unread chain pages unchanged; that a copy-on-write insert equals a sorted merge for random batches (including keys below the first and above the last, many into one leaf, root growth to depth 3, and 1 KB separators that split internal nodes), leaves the old tree readable and unchanged, frees exactly the replaced nodes, writes only new pages, writes exactly one leaf and two internal nodes for one entry in a depth-3 tree, and does not fragment leaves under single inserts; that a checkpoint after 1 or 100 right-edge facts writes at most the tree-depth difference more pages at 100,000 facts than at 10,000, and at most one path per new entry per tree for random keys; and that four reader threads querying throughout 40 checkpoints never fail and never see the fact count go backwards. Hardening tests (PR 5) damage an index leaf, an internal node, a dictionary leaf, a value page and a free-list page in turn. Every committed read or checkpoint that reaches the damaged page fails with STG-029, never with wrong or fewer rows. A query through `Minigraf` on a file with damaged leaves fails with STG-029, and the file still opens at its last checkpoint. A size test measures at most 200 bytes per fact (142 measured) on 144k facts in #433's shape; its 1.15M-fact variant is ignored locally and runs in the nightly smoke workflow. A shared dictionary-cache test checks that cached ident and timestamp lookups return the same facts and that the cache stays bounded. Merged from the v2.x line: storage and `edge_cases_test.rs` tests check that more than 65,535 uncheckpointed facts each resolve to their own fact through entity lookups and attribute scans, in memory and file-backed (#445). Merged from the v2.x line: `tests/golden_corpus_test.rs` opens a copy of each committed golden file in `tests/golden/` (v7 files written by minigraf 2.0.3: one checkpoint, several checkpoints, indexes rebuilt on open, a crash-left WAL, an already-checkpointed WAL, same-transaction multi-values) and checks its manifest's CRC32, `tx_count` and query results after open, after a checkpoint and reopen, and after one more write; a second test checks that every golden file has a manifest (#391). On `v3` the same test checks that each v7 file has a v8 meta page after its first open and gives the same results after reopening the migrated file, and adds seven v8 golden files (one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values at the size limits); the #371 queries, skipped on v2.x, run and pass. Integrity tests (#373) check that intact files verify with no problems and an exact fact and page count, and that `verify` reports each kind of damage with its code: an EAVT, AEVT, AVET or VAET leaf missing an entry (STG-038), keys out of order and a page reached from two trees (STG-039), a rotted index leaf (STG-029), a free page still in use and a leaked page (STG-035), a missing transaction entry (STG-036), and an entity id whose UUID maps to another id (STG-040). After each index or free-list damage, `rebuild_indexes` leaves a file that verifies clean, reads back every fact and accounts for every page. The source is an index that agrees with another one, or the only intact one; three intact indexes that all disagree, or a damaged dictionary, give STG-041 with the file unchanged. A rebuild commits pending facts, including new ids and long values. A crash at every write (clean and torn) and every sync of a rebuild reopens either at the old generation with only its old damage or at the new one with none, never losing a fact. Through `Minigraf` (`tests/integrity_test.rs`), a rebuild commits WAL entries and deletes the WAL, and flipping a byte in each leaf of a file in turn is always reported, then either repaired (index leaves) or refused with STG-041 and the file unchanged (dictionary leaves). Live-read tests (#379) check that `triple_len` ends exactly where `tx↓` begins for every value type and index; that on random histories (several valid-time windows, retractions, re-assertions, several values per attribute, long values, 12 transactions per checkpoint) the committed live read for every scan kind and `:as-of` point is exactly the net-asserted set, that it stays equivalent with pending facts on top, and after reopen; and that a point read on a triple asserted and retracted 2,000 times reads under a third of the pages a history read does, with or without `:as-of`. `tests/live_reads_test.rs` runs the same history in a checkpointed file database and in memory and checks that point, attribute, full-scan, join, valid-time, `:as-of` and rule queries return the same rows, before and after reopen. One-window tests (#435) check that `net_asserted_facts` keeps each triple's newest transaction only (a later window replaces an earlier one, closing an open window, a retraction in the newest transaction hides it, records with two windows in one transaction keep both, once each) against a per-triple model on random histories; that the on-disk key walk agrees at every `:as-of` before and after reopen; and that `API-011` rejects two windows of one fact in one transaction without taking a `tx_count`. `tests/valid_time_window_test.rs` checks replacing, closing, extending, reopening and retract-then-reassert through entity-bound and attribute queries in memory, on a checkpointed file and after reopen, and `API-011` from `transact` and from `WriteTransaction::commit`. Cursor tests (#432, `tests/cursor_test.rs`) check that `query()` batches partition the `execute()` answer for several batch sizes (never empty, never over `max_rows`, `max_rows == 0` still progresses, `None` stays `None`), that the iterator and aggregates, rules, `:as-of` and `:any-valid-time` match `execute()`, that a cursor opened before a `transact`, a `WriteTransaction` commit, a retract or a checkpoint returns the pre-write answer to the end (in memory and file-backed), that `transact`/`retract`/`rule` are `API-012` and bind slots `API-010`, that a `WriteTransaction` thread gets `INT-001`, that rule limits give `INT-020` as from `execute()`, that `PreparedQuery::query` matches `PreparedQuery::execute`, and that a cursor is `Send` and outlives its database handle. Fact-log tests (#430, `tests/fact_log_test.rs`) check that `fact_log()` returns exactly the records written (assertions and retractions, with `tx_count` and `tx_id`) in memory, with everything pending, half checkpointed, fully checkpointed and after reopen, in both orders, with windows of 1 to 1,000 records and batch sizes of 1 to 1,000, in non-decreasing `tx_count` in Tx order; that every layout gives identical records, including valid-time windows and long string values; that a transaction larger than the window is split with no loss or duplicate; that attribute, attribute-prefix, entity and tx-range filters and their combinations match a model, and unknown names match nothing; that writes after open are not in the log; that `checkpoint()` and `rebuild_indexes()` are `API-013` while a log is open (a checkpoint with nothing to write is allowed), that automatic checkpoints from `execute()` and `WriteTransaction::commit` wait for the log, and that closing, dropping or finishing the log releases the database; that a `WriteTransaction` thread gets `INT-001`; and that a log is `Send`, outlives its database handle, and the deferred writes replay on the next open. Read-only open tests (#429, `tests/read_only_open_test.rs`) check that a read-only handle on a file with checkpointed facts and a pending WAL returns the same query results, `tx_count` and fact log as a read-write open; that queries, a cursor, a fact log, `verify`, `prepare`, rules and a UDF leave the `.graph` and WAL bytes and the `.graph` mtime unchanged, and no WAL is created; that `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` are `API-014` without taking a `tx_count`, also on a read-only in-memory database and through the builder path; that a missing file is `STG-042` and nothing is created; that an empty file is an empty database and stays 0 bytes; that read-only handles share a file while a read-write one is refused with `STG-025`, both ways; that a file without write permission opens read-only; and that small and whole-file page caches work. File-backend unit tests check that a shared-mode backend refuses page writes and syncs with `INT-056`, that the in-process registry counts readers and admits readers or one writer (also without kernel locks), and that a reader in another process shares the file with a reader here and refuses a writer here with `STG-026`. Storage unit tests check that a read-only open of a v7 file at every point of an interrupted migration (clean and torn writes) sees every fact and writes nothing, and that an empty backend stays empty. `tests/golden_corpus_test.rs` opens a copy of every golden file read-only first and checks its `tx_count` and queries, and that neither the file nor its WAL changes. ### Covered Areas diff --git a/docs/superpowers/specs/2026-10-07-read-only-open-design.md b/docs/superpowers/specs/2026-10-07-read-only-open-design.md new file mode 100644 index 0000000000..0f92bba295 --- /dev/null +++ b/docs/superpowers/specs/2026-10-07-read-only-open-design.md @@ -0,0 +1,111 @@ +# Read-only open (#429, v3.0.0) — design + +**Issue:** #429 (`OpenOptions::read_only` for offline tooling). +**Branch:** `v3`. **Related:** #430 (fact log, merged), #431 (raw log writer), +#462 (binding cursors), temporal_reasoning#378 (offline migration). + +## Goal + +Open a database so that nothing done through the handle can write to the `.graph` +file or its WAL, and so that several such handles (a migration reader, a backup +step, an inspector) can hold the file at the same time. + +## API + +```rust +pub struct OpenOptions { // #[non_exhaustive], so adding a field is not a break + /* existing fields */ + pub read_only: bool, // default false +} +impl OpenOptions { + pub fn read_only(self, read_only: bool) -> Self; +} +``` + +No other API changes. `Minigraf::open_with_options(path, OpenOptions::new().read_only(true))` +and `OpenOptions::new().read_only(true).path(p).open()` both work. + +## Decisions + +1. **Shared lock.** A read-only open takes the kernel lock in shared mode + (`File::try_lock_shared`); a read-write open keeps the exclusive lock. Read-only + handles coexist, in this process or others. A writer and a reader exclude each + other, so a read-only handle sees a file and WAL that cannot change under it. The + writer side gets the existing STG-025/STG-026 errors with the same retry budget. + The in-process registry (`OPEN_PATHS`) becomes a map from path to holders (one + writer, or any number of readers) so that STG-025 keeps meaning "this process + already holds a conflicting handle", and `allow_unlocked` on a filesystem without + locks admits many readers or one writer, never both. +2. **No file is created or modified.** The `.graph` file is opened for reading only: + - Missing file: new **STG-042**, "database file not found ({}); a read-only open + does not create one". Nothing is created. + - Empty file, or a torn initial meta (a crash during creation): an empty database. + Nothing is initialised. + - Torn page 0 after a v7 → v8 migration: the backup meta is used, and page 0 is not + restored. The next read-write open restores it. + - Format v7 file: not migrated. Its facts are read into memory, as uncheckpointed + facts on an empty committed state, and the migration happens on the next + read-write open. Memory is O(facts) for such a file; a v8 file is read through + the page cache as usual. + - WAL: read and applied in memory, exactly as a read-write open does (same skip of + already-checkpointed entries, same tx counter restore). The WAL file is not + opened for writing, not recreated, and not deleted. + The backend refuses `write_page` and `sync` with INT-056 (an invariant: no code path + reaches it), so a missed check fails closed instead of writing. +3. **Writes are refused, reads are not.** New **API-014**, "database is open + read-only; {} is not allowed", from: + - `execute` of `transact` or `retract` (before a tx count is allocated), + - `begin_write`, + - `checkpoint` and `rebuild_indexes`. + Queries, `query` cursors, `prepare`, `fact_log`, `verify`, `current_tx_count` and + UDF registration work. Rule registration also works: rules live in the handle's + memory and are never written to the file. + Dropping the handle does not checkpoint. +4. **In-memory databases.** `read_only` on `open_memory`/`in_memory_with_options` + gives an empty database that refuses writes with API-014, the same rules as above. + It has no use, but it is consistent and needs no special case. +5. **Page cache.** `page_cache_size` already sizes the cache that serves every + committed page read, so it applies unchanged. A tool that wants the whole file + resident passes `file_size / 4096` pages. Docs say so. +6. **Bindings: out of scope here.** The bindings live in their own repositories. A + follow-up issue (like #462 for cursors) exposes `open_with_options` with + `read_only`, `page_cache_size` and `allow_unlocked` in every binding, Python first + (Tier 1). + +## Implementation + +- `FileBackend::open_with(path, allow_unlocked, mode: LockMode)` with + `LockMode::{Exclusive, Shared}`. `Shared` opens read-only without `create`, maps + `NotFound` to STG-042, uses `try_lock_shared`, never syncs the parent directory, and + sets `read_only` so `write_page`/`sync` fail with INT-056. +- `OPEN_PATHS: HashMap`; `PathGuard` removes its own claim. +- `PersistentFactStorage::open_read_only(backend, cache, wal_base)`: the same meta + selection with a `read_only` flag that skips the page-0 restore and `init_empty`, and + loads a v7 file into memory instead of migrating. +- `WriteContext::ReadOnly { pfs }` holds the backend (and with it the lock) and serves + `verify`. `do_checkpoint` and `wal_write_stamped_batch` return API-014 for it. + `Minigraf` checks `options.read_only` up front in `execute`, `begin_write`, + `checkpoint` and `rebuild_indexes`. `Inner::drop` skips the checkpoint. +- API-014, STG-042 and INT-056 are registered in `src/error.rs` and + `docs/ERROR_REFERENCE.md`. + +## Tests (`tests/read_only_open_test.rs`, unit tests in `file.rs`) + +- Same view: a file with checkpointed facts and uncheckpointed WAL entries returns the + same query results, `current_tx_count` and fact log read-only as read-write. +- Nothing written: the `.graph` and `.wal` bytes and mtimes are unchanged after a + read-only session with queries, a cursor, a fact log, `verify` and drop; no WAL is + created when none existed. +- Refusals: `transact`, `retract`, `begin_write`, `checkpoint`, `rebuild_indexes` give + API-014 and `current_tx_count` is unchanged; a rule and a query using it work. +- Missing file: STG-042 and no file created. Empty file: empty database, file still + 0 bytes. +- Lock: two read-only handles at once (same process); read-write while read-only is + open is refused (STG-025) and vice versa; both work after the other drops. A child + process holding read-only does not block a read-only open here but does block a + read-write one (STG-026). +- v7 file (golden corpus fixture): read-only open shows the same facts as a + read-write open, and the file bytes are unchanged (still v7). +- Read-only file permissions (Unix, skipped as root): opens and queries. +- `page_cache_size` reaches the cache in read-only mode. +- In-memory `read_only`: writes give API-014. diff --git a/src/db.rs b/src/db.rs index 2a4ea18811..a174aac775 100644 --- a/src/db.rs +++ b/src/db.rs @@ -32,7 +32,7 @@ use crate::query::datalog::rules::RuleRegistry; use crate::query::datalog::types::{AttributeSpec, DatalogCommand, Transaction}; use crate::storage::backend::MemoryBackend; #[cfg(not(target_arch = "wasm32"))] -use crate::storage::backend::file::FileBackend; +use crate::storage::backend::file::{FileBackend, LockMode}; use crate::storage::persistent_facts::PersistentFactStorage; #[cfg(not(target_arch = "wasm32"))] use crate::wal::WalWriter; @@ -126,6 +126,10 @@ pub struct OpenOptions { /// **File-backed databases only.** In-memory databases (`Minigraf::in_memory`, /// `open_memory`) ignore this option — `MemoryBackend` serves every read and /// write directly from RAM and never consults the page cache. + /// + /// Every read of a committed page goes through this cache, including on a + /// [`read_only`](Self::read_only) handle. A tool that reads the whole file + /// can keep it resident with `file_size / 4096` pages. pub page_cache_size: usize, /// Maximum facts that can be derived per recursive rule iteration. /// Defaults to 1_000_000. Use to prevent runaway recursive rules. @@ -151,6 +155,23 @@ pub struct OpenOptions { /// /// No effect on in-memory databases, which have no WAL. pub synchronous: SyncMode, + /// Open the database read-only. Defaults to `false`. + /// + /// Nothing done through the handle writes to the `.graph` file or its WAL: + /// a missing file is an error (`STG-042`) rather than created, a WAL left + /// by a previous session is applied in memory but not checkpointed or + /// deleted, and a format v7 file is read into memory instead of migrated. + /// The handle shows the same data a read-write open would. + /// + /// The file is locked in shared mode, so any number of read-only handles, + /// in this process or others, can hold it at once; a read-write open + /// excludes them and they exclude it (`STG-025`/`STG-026`). + /// + /// `transact`, `retract`, [`Minigraf::begin_write`], + /// [`Minigraf::checkpoint`] and [`Minigraf::rebuild_indexes`] fail with + /// `API-014`. Queries, cursors, [`Minigraf::fact_log`], + /// [`Minigraf::verify`] and rule registration (which lives in memory) work. + pub read_only: bool, } impl Default for OpenOptions { @@ -162,6 +183,7 @@ impl Default for OpenOptions { max_results: DEFAULT_MAX_RESULTS, allow_unlocked: false, synchronous: SyncMode::Full, + read_only: false, } } } @@ -224,6 +246,13 @@ impl OpenOptions { self } + /// Open the database read-only. See [`OpenOptions::read_only`]. + #[must_use] + pub fn read_only(mut self, read_only: bool) -> Self { + self.read_only = read_only; + self + } + /// Set the path for a file-backed database. #[cfg(not(target_arch = "wasm32"))] pub fn path(self, path: impl AsRef) -> OpenOptionsWithPath { @@ -281,6 +310,12 @@ enum WriteContext { /// Count of WAL entries written since the last checkpoint (or since open). wal_entry_count: usize, }, + /// File-backed database opened with [`OpenOptions::read_only`]: the + /// storage (and with it the shared file lock), and no WAL writer. + #[cfg(not(target_arch = "wasm32"))] + ReadOnly { + pfs: PersistentFactStorage, + }, } // ─── IntegrityReport ────────────────────────────────────────────────────────── @@ -327,7 +362,7 @@ impl Drop for Inner { // Errors are silently ignored (can't propagate from Drop). // Skip if wal_checkpoint_threshold is usize::MAX — that sentinel suppresses // all checkpointing (used by benchmarks to keep WAL entries pending). - if self.options.wal_checkpoint_threshold == usize::MAX { + if self.options.wal_checkpoint_threshold == usize::MAX || self.options.read_only { return; } if let Ok(mut ctx) = self.write_lock.lock() { @@ -429,13 +464,22 @@ impl Minigraf { // Open the main .graph file. The WAL's base generation (if a WAL exists) // decides which meta page to trust when only one is valid. - let backend = FileBackend::open_with(&db_path, opts.allow_unlocked)?; + let mode = if opts.read_only { + LockMode::Shared + } else { + LockMode::Exclusive + }; + let backend = FileBackend::open_with(&db_path, opts.allow_unlocked, mode)?; let wal_base = if wal_path.exists() { crate::wal::WalReader::open(&wal_path)?.base_generation() } else { None }; - let pfs = PersistentFactStorage::open(backend, opts.page_cache_size, wal_base)?; + let pfs = if opts.read_only { + PersistentFactStorage::open_read_only(backend, opts.page_cache_size, wal_base)? + } else { + PersistentFactStorage::open(backend, opts.page_cache_size, wal_base)? + }; // Share the fact storage let fact_storage = pfs.storage().clone(); @@ -443,6 +487,15 @@ impl Minigraf { // Replay any existing WAL entries before opening the writer let wal_entry_count = Self::replay_wal(&wal_path, &fact_storage, &pfs)?; + if opts.read_only { + // The WAL was applied in memory; it is neither written nor deleted. + return Ok(Self::from_parts( + fact_storage, + WriteContext::ReadOnly { pfs }, + opts, + )); + } + // Open the WAL writer only if the WAL file already exists from a previous session. // Otherwise, create it lazily on the first write. let wal = if wal_path.exists() { @@ -462,15 +515,27 @@ impl Minigraf { wal_entry_count, }; - Ok(Minigraf { + Ok(Self::from_parts(fact_storage, ctx, opts)) + } + + fn from_parts(fact_storage: FactStorage, ctx: WriteContext, options: OpenOptions) -> Self { + Minigraf { inner: Arc::new(Inner { fact_storage, rules: Arc::new(RwLock::new(RuleRegistry::new())), functions: Arc::new(RwLock::new(FunctionRegistry::with_builtins())), write_lock: Mutex::new(ctx), - options: opts, + options, }), - }) + } + } + + /// Fail with API-014 if this handle was opened read-only. + fn reject_if_read_only(&self, op: &str) -> Result<()> { + if self.inner.options.read_only { + bail_coded!(ErrorCode::Api014, op); + } + Ok(()) } /// Create an in-memory database (no WAL, no persistence). Suitable for tests and REPL. @@ -503,15 +568,7 @@ impl Minigraf { // we just use the shared FactStorage directly. drop(pfs); - Ok(Minigraf { - inner: Arc::new(Inner { - fact_storage, - rules: Arc::new(RwLock::new(RuleRegistry::new())), - functions: Arc::new(RwLock::new(FunctionRegistry::with_builtins())), - write_lock: Mutex::new(WriteContext::Memory), - options: opts, - }), - }) + Ok(Self::from_parts(fact_storage, WriteContext::Memory, opts)) } // ── WAL replay helper ──────────────────────────────────────────────────── @@ -600,6 +657,11 @@ impl Minigraf { ); if is_write { + match &cmd { + DatalogCommand::Transact(_) => self.reject_if_read_only("transact")?, + DatalogCommand::Retract(_) => self.reject_if_read_only("retract")?, + _ => {} + } let mut ctx = self .inner .write_lock @@ -788,6 +850,7 @@ impl Minigraf { if is_write_tx_active() { bail_coded!(ErrorCode::Int001); } + self.reject_if_read_only("begin_write")?; let guard = self .inner .write_lock @@ -830,6 +893,7 @@ impl Minigraf { } fn checkpoint_inner(&self) -> Result<()> { + self.reject_if_read_only("checkpoint")?; let mut ctx = self .inner .write_lock @@ -873,7 +937,7 @@ impl Minigraf { match &*ctx { WriteContext::Memory => Ok(IntegrityReport::default()), #[cfg(not(target_arch = "wasm32"))] - WriteContext::File { pfs, .. } => { + WriteContext::File { pfs, .. } | WriteContext::ReadOnly { pfs } => { let f = pfs.verify()?; Ok(IntegrityReport { facts: f.facts, @@ -910,6 +974,7 @@ impl Minigraf { } fn rebuild_indexes_inner(&self) -> Result<()> { + self.reject_if_read_only("rebuild_indexes")?; let mut ctx = self .inner .write_lock @@ -927,6 +992,8 @@ impl Minigraf { pfs.rebuild_indexes()?; Self::delete_wal(wal, db_path, wal_entry_count) } + #[cfg(not(target_arch = "wasm32"))] + WriteContext::ReadOnly { .. } => bail_coded!(ErrorCode::Api014, "rebuild_indexes"), } } @@ -976,6 +1043,8 @@ impl Minigraf { pfs.save()?; Self::delete_wal(wal, db_path, wal_entry_count)?; } + #[cfg(not(target_arch = "wasm32"))] + WriteContext::ReadOnly { .. } => bail_coded!(ErrorCode::Api014, "checkpoint"), } Ok(()) } @@ -1502,6 +1571,8 @@ impl<'a> WriteTransaction<'a> { Ok(*wal_entry_count >= opts.wal_checkpoint_threshold) } + #[cfg(not(target_arch = "wasm32"))] + WriteContext::ReadOnly { .. } => bail_coded!(ErrorCode::Api014, "transact"), } } @@ -2046,6 +2117,7 @@ mod tests { max_results: 1_000_000, allow_unlocked: false, synchronous: SyncMode::Full, + read_only: false, }; let db = Minigraf::open_with_options(&path, opts).unwrap(); assert_eq!(db.inner.options.wal_checkpoint_threshold, 5); @@ -2128,7 +2200,7 @@ mod tests { "lazy-reopen call site should honor opts.synchronous" ); } - WriteContext::Memory => panic!("expected file-backed WriteContext"), + _ => panic!("expected read-write file-backed WriteContext"), } drop(ctx); // Handle drops here without checkpointing: wal_checkpoint_threshold == @@ -2167,7 +2239,7 @@ mod tests { "initial-open call site should honor opts.synchronous" ); } - WriteContext::Memory => panic!("expected file-backed WriteContext"), + _ => panic!("expected read-write file-backed WriteContext"), } } diff --git a/src/error.rs b/src/error.rs index fe4232e2a8..93d569f752 100644 --- a/src/error.rs +++ b/src/error.rs @@ -182,6 +182,7 @@ pub(crate) enum ErrorCode { Stg039, Stg040, Stg041, + Stg042, Wal001, Wal002, Wal003, @@ -201,6 +202,7 @@ pub(crate) enum ErrorCode { Api011, Api012, Api013, + Api014, Int001, Int002, Int003, @@ -256,6 +258,7 @@ pub(crate) enum ErrorCode { Int053, Int054, Int055, + Int056, } /// Single source of truth: (code, code string, message template, category). @@ -1043,6 +1046,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[ "Cannot rebuild indexes: {}", ErrorCategory::Storage, ), + ( + ErrorCode::Stg042, + "STG-042", + "database file not found ({}); a read-only open does not create one", + ErrorCategory::Storage, + ), ( ErrorCode::Wal001, "WAL-001", @@ -1157,6 +1166,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[ "checkpoint deferred while {} fact log(s) are open; close them and checkpoint again", ErrorCategory::Api, ), + ( + ErrorCode::Api014, + "API-014", + "database is open read-only; {} is not allowed", + ErrorCategory::Api, + ), ( ErrorCode::Int001, "INT-001", @@ -1487,6 +1502,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[ "rule predicate '{}' disappeared during rollback", ErrorCategory::Internal, ), + ( + ErrorCode::Int056, + "INT-056", + "write to a read-only storage backend: {}", + ErrorCategory::Internal, + ), ]; pub(crate) fn registry_entry(code: ErrorCode) -> (&'static str, &'static str, ErrorCategory) { @@ -1889,6 +1910,7 @@ mod tests { | ErrorCode::Stg039 | ErrorCode::Stg040 | ErrorCode::Stg041 + | ErrorCode::Stg042 | ErrorCode::Wal001 | ErrorCode::Wal002 | ErrorCode::Wal003 @@ -1908,6 +1930,7 @@ mod tests { | ErrorCode::Api011 | ErrorCode::Api012 | ErrorCode::Api013 + | ErrorCode::Api014 | ErrorCode::Int001 | ErrorCode::Int002 | ErrorCode::Int003 @@ -1962,7 +1985,8 @@ mod tests { | ErrorCode::Int052 | ErrorCode::Int053 | ErrorCode::Int054 - | ErrorCode::Int055 => assert_has_registry_entry(code), + | ErrorCode::Int055 + | ErrorCode::Int056 => assert_has_registry_entry(code), } } @@ -2097,6 +2121,7 @@ mod tests { ErrorCode::Stg039, ErrorCode::Stg040, ErrorCode::Stg041, + ErrorCode::Stg042, ErrorCode::Wal001, ErrorCode::Wal002, ErrorCode::Wal003, @@ -2116,6 +2141,7 @@ mod tests { ErrorCode::Api011, ErrorCode::Api012, ErrorCode::Api013, + ErrorCode::Api014, ErrorCode::Int001, ErrorCode::Int002, ErrorCode::Int003, @@ -2171,6 +2197,7 @@ mod tests { ErrorCode::Int053, ErrorCode::Int054, ErrorCode::Int055, + ErrorCode::Int056, ] { exhaustive(code); } diff --git a/src/storage/backend/file.rs b/src/storage/backend/file.rs index 2674c2fe2b..7b68b012b2 100644 --- a/src/storage/backend/file.rs +++ b/src/storage/backend/file.rs @@ -51,7 +51,7 @@ fn classify(result: Result<(), std::fs::TryLockError>, allow_unlocked: bool) -> } } -/// Canonicalised paths this process currently has open. +/// Canonicalised paths this process currently has open, with who holds each. /// /// Correctness comes from the kernel lock, not this registry -- but this is /// no longer *purely* diagnostic. Two things read it: which error message to @@ -63,10 +63,40 @@ fn classify(result: Result<(), std::fs::TryLockError>, allow_unlocked: bool) -> /// `already_open_here` ever wrongly returned `true`, `open_with` would both /// report the wrong error *and* skip the retry that would have let it /// succeed. -static OPEN_PATHS: std::sync::LazyLock>> = - std::sync::LazyLock::new(|| std::sync::Mutex::new(std::collections::HashSet::new())); +static OPEN_PATHS: std::sync::LazyLock< + std::sync::Mutex>, +> = std::sync::LazyLock::new(|| std::sync::Mutex::new(std::collections::HashMap::new())); + +/// How a backend holds its file: one writer, or any number of readers (#429). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum LockMode { + /// Read-write. Exclusive kernel lock; creates the file if missing. + Exclusive, + /// Read-only. Shared kernel lock; never creates or writes the file. + Shared, +} + +/// Who in this process holds a path: one writer, or `n` readers. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Holders { + Writer, + Readers(usize), +} + +/// Record a holder of `path`, which the kernel lock already admitted. +fn register(open: &mut std::collections::HashMap, path: &Path, mode: LockMode) { + match (mode, open.get_mut(path)) { + (LockMode::Shared, Some(Holders::Readers(n))) => *n = n.saturating_add(1), + (LockMode::Shared, _) => { + open.insert(path.to_path_buf(), Holders::Readers(1)); + } + (LockMode::Exclusive, _) => { + open.insert(path.to_path_buf(), Holders::Writer); + } + } +} -/// Removes this backend's path from `OPEN_PATHS` when the backend drops. +/// Removes this backend's claim on its path from `OPEN_PATHS` when the backend drops. struct PathGuard(Option); impl Drop for PathGuard { @@ -74,7 +104,12 @@ impl Drop for PathGuard { if let Some(path) = self.0.take() && let Ok(mut open) = OPEN_PATHS.lock() { - open.remove(&path); + match open.get_mut(&path) { + Some(Holders::Readers(n)) if *n > 1 => *n = n.saturating_sub(1), + _ => { + open.remove(&path); + } + } } } } @@ -83,16 +118,17 @@ impl Drop for PathGuard { /// /// Not diagnostic-only: besides choosing which error message to print, this /// also gates whether `open_with` retries a `WouldBlock` at all (see -/// `OPEN_PATHS` above). +/// `OPEN_PATHS` above). A `WouldBlock` always means a conflicting holder (a +/// writer, or readers when we want to write), so any entry is one. fn already_open_here(path: &Path) -> bool { OPEN_PATHS .lock() - .map(|open| open.contains(path)) + .map(|open| open.contains_key(path)) .unwrap_or(false) } -/// Atomically test-and-set `path` into `OPEN_PATHS`, admitting only the -/// first caller. +/// Atomically test-and-set `path` into `OPEN_PATHS`, admitting only callers +/// that do not conflict with a holder: one writer, or any number of readers. /// /// Used by the `ProceedUnlocked` arm, where there is no kernel lock and this /// registry is the only thing preventing two handles on one unlockable path @@ -104,10 +140,19 @@ fn already_open_here(path: &Path) -> bool { /// /// Split out from `open_with` so the race can be exercised directly (#330) /// instead of needing a genuinely lock-incapable filesystem. -fn claim_unlocked_path(path: &Path) -> bool { +fn claim_unlocked_path(path: &Path, mode: LockMode) -> bool { OPEN_PATHS .lock() - .map(|mut open| open.insert(path.to_path_buf())) + .map(|mut open| { + let free = matches!( + (mode, open.get(path)), + (_, None) | (LockMode::Shared, Some(Holders::Readers(_))) + ); + if free { + register(&mut open, path, mode); + } + free + }) .unwrap_or(false) } @@ -162,6 +207,8 @@ pub struct FileBackend { file: File, /// Whole pages in the file. page_count: u64, + /// Opened with `LockMode::Shared`: `write_page` and `sync` refuse. + read_only: bool, } impl FileBackend { @@ -184,7 +231,7 @@ impl FileBackend { /// convenience for the ~40 in-crate test call sites. #[cfg(test)] pub fn open>(path: P) -> Result { - Self::open_with(path, false) + Self::open_with(path, false, LockMode::Exclusive) } /// As `FileBackend::open` (test-only), but `allow_unlocked` permits opening on a @@ -192,21 +239,47 @@ impl FileBackend { /// /// `allow_unlocked` does NOT override a lock held by someone else. It /// applies only when the filesystem rejects locking outright. - pub fn open_with>(path: P, allow_unlocked: bool) -> Result { + /// + /// `LockMode::Shared` opens an existing file for reading only (STG-042 if + /// it is missing) under a shared lock, which other shared holders do not + /// conflict with, and the backend then refuses every write (#429). + pub fn open_with>( + path: P, + allow_unlocked: bool, + mode: LockMode, + ) -> Result { let path = path.as_ref().to_path_buf(); + let read_only = mode == LockMode::Shared; - let file = OpenOptions::new() - .read(true) - .write(true) - .create(true) - .truncate(false) - .open(&path)?; + let file = if read_only { + match File::open(&path) { + Ok(f) => f, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + bail_coded!(ErrorCode::Stg042, path.display()); + } + Err(e) => return Err(e.into()), + } + } else { + OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&path)? + }; + let try_lock = || { + if read_only { + file.try_lock_shared() + } else { + file.try_lock() + } + }; // Canonicalise only after the file exists. Used for the diagnostic // registry; if it fails we fall back to the generic message. let canonical = std::fs::canonicalize(&path).unwrap_or_else(|_| path.clone()); - let mut lock_result = file.try_lock(); + let mut lock_result = try_lock(); // Retry a `WouldBlock` with backoff, but ONLY when the conflict might // be with another process. If this process already has the path @@ -222,7 +295,7 @@ impl FileBackend { let mut delay = LOCK_RETRY_INITIAL_DELAY; for _ in 0..LOCK_RETRY_ATTEMPTS { std::thread::sleep(delay); - lock_result = file.try_lock(); + lock_result = try_lock(); if !matches!(lock_result, Err(std::fs::TryLockError::WouldBlock)) { break; } @@ -233,7 +306,7 @@ impl FileBackend { let path_guard = match classify(lock_result, allow_unlocked) { LockOutcome::Acquired => { if let Ok(mut open) = OPEN_PATHS.lock() { - open.insert(canonical.clone()); + register(&mut open, &canonical, mode); } PathGuard(Some(canonical)) } @@ -253,7 +326,7 @@ impl FileBackend { // second handle in this process opens the same unlockable // path. See `claim_unlocked_path` for why the check and // insert must be one atomic operation. - if !claim_unlocked_path(&canonical) { + if !claim_unlocked_path(&canonical, mode) { bail_coded!(ErrorCode::Stg025, path.display()); } PathGuard(Some(canonical)) @@ -266,7 +339,7 @@ impl FileBackend { // storage layer's business. let file_len = file.metadata()?.len(); let page_count = file_len / PAGE_SIZE as u64; - if page_count == 0 { + if page_count == 0 && !read_only { // Make the new file's directory entry durable. Without the directory // sync a power loss can lose the whole file despite later fsyncs // (#389). An empty file left by a crash takes this branch again. @@ -277,6 +350,7 @@ impl FileBackend { path, file, page_count, + read_only, _path_guard: path_guard, }) } @@ -290,6 +364,9 @@ impl FileBackend { impl StorageBackend for FileBackend { fn write_page(&mut self, page_id: u64, data: &[u8]) -> Result<()> { + if self.read_only { + bail_coded!(ErrorCode::Int056, format!("page {page_id}")); + } if data.len() != PAGE_SIZE { bail_coded!(ErrorCode::Int051, data.len(), PAGE_SIZE); } @@ -342,6 +419,9 @@ impl StorageBackend for FileBackend { } fn sync(&mut self) -> Result<()> { + if self.read_only { + bail_coded!(ErrorCode::Int056, "sync"); + } self.file.sync_all()?; Ok(()) } @@ -351,6 +431,9 @@ impl StorageBackend for FileBackend { } fn close(&mut self) -> Result<()> { + if self.read_only { + return Ok(()); + } self.sync() } @@ -440,7 +523,12 @@ mod tests { .parse() .expect("hold millis is a number"); - let backend = FileBackend::open(&db_path).expect("child holds lock"); + let mode = if std::env::var("MINIGRAF_HOLD_SHARED").is_ok() { + LockMode::Shared + } else { + LockMode::Exclusive + }; + let backend = FileBackend::open_with(&db_path, false, mode).expect("child holds lock"); std::fs::write(&ready_marker, "").expect("write ready marker"); std::thread::sleep(std::time::Duration::from_millis(millis)); drop(backend); @@ -763,7 +851,7 @@ mod tests { let path = path.clone(); std::thread::spawn(move || { barrier.wait(); - claim_unlocked_path(&path) + claim_unlocked_path(&path, LockMode::Exclusive) }) }) .collect(); @@ -782,6 +870,150 @@ mod tests { ); } + // ── #429: shared (read-only) mode ─────────────────────────────────────── + + fn code_of(e: anyhow::Error) -> String { + crate::error::MinigrafError::from(e).code().to_string() + } + + fn open_err(path: &Path, mode: LockMode) -> String { + match FileBackend::open_with(path, false, mode) { + Ok(_) => panic!("open must be refused"), + Err(e) => code_of(e), + } + } + + #[test] + fn test_shared_refuses_writes_and_sync() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("ro.graph"); + { + let mut w = FileBackend::open(&path).unwrap(); + w.write_page(0, &[7u8; PAGE_SIZE]).unwrap(); + w.sync().unwrap(); + } + let mut r = FileBackend::open_with(&path, false, LockMode::Shared).unwrap(); + assert_eq!(r.page_count().unwrap(), 1); + assert_eq!(r.read_page(0).unwrap(), vec![7u8; PAGE_SIZE]); + assert_eq!( + code_of(r.write_page(0, &[0u8; PAGE_SIZE]).unwrap_err()), + "INT-056" + ); + assert_eq!( + code_of(r.write_page(1, &[0u8; PAGE_SIZE]).unwrap_err()), + "INT-056" + ); + assert_eq!(code_of(r.sync().unwrap_err()), "INT-056"); + r.close().unwrap(); + drop(r); + assert_eq!(std::fs::read(&path).unwrap(), vec![7u8; PAGE_SIZE]); + } + + #[test] + fn test_shared_missing_file_is_stg_042() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("missing.graph"); + assert_eq!(open_err(&path, LockMode::Shared), "STG-042"); + assert!(!path.exists()); + } + + #[test] + fn test_shared_holders_coexist_and_exclude_a_writer() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("shared.graph"); + drop(FileBackend::open(&path).unwrap()); + let canonical = std::fs::canonicalize(&path).unwrap(); + + let a = FileBackend::open_with(&path, false, LockMode::Shared).unwrap(); + let b = FileBackend::open_with(&path, false, LockMode::Shared).unwrap(); + assert_eq!( + OPEN_PATHS.lock().unwrap().get(&canonical).copied(), + Some(Holders::Readers(2)) + ); + let start = std::time::Instant::now(); + assert_eq!(open_err(&path, LockMode::Exclusive), "STG-025"); + assert!( + start.elapsed() < std::time::Duration::from_millis(200), + "a same-process conflict is not retried" + ); + drop(a); + assert_eq!( + OPEN_PATHS.lock().unwrap().get(&canonical).copied(), + Some(Holders::Readers(1)) + ); + assert_eq!(open_err(&path, LockMode::Exclusive), "STG-025"); + drop(b); + assert!(!already_open_here(&canonical)); + + let w = FileBackend::open(&path).unwrap(); + assert_eq!(open_err(&path, LockMode::Shared), "STG-025"); + drop(w); + assert!(!already_open_here(&canonical)); + drop(FileBackend::open_with(&path, false, LockMode::Shared).unwrap()); + assert!(!already_open_here(&canonical)); + } + + #[test] + fn test_claim_unlocked_path_admits_readers_or_one_writer() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("unlocked.graph"); + assert!(claim_unlocked_path(&path, LockMode::Shared)); + assert!(claim_unlocked_path(&path, LockMode::Shared)); + assert!(!claim_unlocked_path(&path, LockMode::Exclusive)); + OPEN_PATHS.lock().unwrap().remove(path.as_path()); + assert!(claim_unlocked_path(&path, LockMode::Exclusive)); + assert!(!claim_unlocked_path(&path, LockMode::Shared)); + assert!(!claim_unlocked_path(&path, LockMode::Exclusive)); + OPEN_PATHS.lock().unwrap().remove(path.as_path()); + } + + /// A reader in another process shares the file with a reader here, and + /// refuses a writer here with the cross-process STG-026. + #[test] + fn test_cross_process_shared_lock() { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("cross_shared.graph"); + drop(FileBackend::open(&db_path).unwrap()); + let ready_marker = dir.path().join("cross_shared.ready"); + + let exe = std::env::current_exe().expect("test binary path"); + let mut holder = std::process::Command::new(&exe) + .args([ + "storage::backend::file::tests::hold_lock_entrypoint", + "--exact", + "--nocapture", + ]) + .env("MINIGRAF_HOLD_DB", &db_path) + .env("MINIGRAF_HOLD_READY_MARKER", &ready_marker) + .env("MINIGRAF_HOLD_MILLIS", "3000") + .env("MINIGRAF_HOLD_SHARED", "1") + .spawn() + .expect("spawn lock holder"); + let setup_deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while !ready_marker.exists() { + assert!( + std::time::Instant::now() < setup_deadline, + "lock-holding child never signalled readiness" + ); + std::thread::sleep(std::time::Duration::from_millis(5)); + } + + // The writer first: with a reader open here too, the conflict would + // be this process's own (STG-025). + let writer_code = open_err(&db_path, LockMode::Exclusive); + let reader = FileBackend::open_with(&db_path, false, LockMode::Shared); + holder.kill().expect("kill lock holder"); + holder.wait().expect("reap lock holder"); + + assert!(reader.is_ok(), "a second reader shares the file"); + assert_eq!( + writer_code, "STG-026", + "a writer is refused while a reader holds it" + ); + drop(reader); + FileBackend::open(&db_path).expect("writer opens once the readers are gone"); + } + // ── #389: parent-directory fsync ──────────────────────────────────────── #[test] diff --git a/src/storage/persistent_facts.rs b/src/storage/persistent_facts.rs index db7bb289fd..c818fc104f 100644 --- a/src/storage/persistent_facts.rs +++ b/src/storage/persistent_facts.rs @@ -64,14 +64,8 @@ impl PersistentFactStorage { page_cache_capacity: usize, wal_base_generation: Option, ) -> Result { - let mut pfs = PersistentFactStorage { - backend: Arc::new(Mutex::new(backend)), - page_cache: Arc::new(PageCache::new(page_cache_capacity)), - storage: FactStorage::new(), - dirty: false, - meta: MetaPage::empty(1), - }; - let meta = match pfs.select_meta(wal_base_generation)? { + let mut pfs = Self::unopened(backend, page_cache_capacity); + let meta = match pfs.select_meta(wal_base_generation, false)? { Opened::Fresh => pfs.init_empty()?, Opened::Meta(m) => { m.check_features()?; @@ -86,6 +80,72 @@ impl PersistentFactStorage { Ok(pfs) } + /// Open storage on `backend` without writing to it (#429). + /// + /// Meta selection is the same as [`open`](Self::open), but nothing is + /// repaired or created: an empty file (or a torn initial meta) is an empty + /// database, a torn page 0 after migration is read through the backup meta + /// without restoring page 0, and a format v7 file is read into memory as + /// uncheckpointed facts instead of being migrated. The caller must never + /// call [`save`](Self::save). + pub fn open_read_only( + backend: B, + page_cache_capacity: usize, + wal_base_generation: Option, + ) -> Result { + let mut pfs = Self::unopened(backend, page_cache_capacity); + let meta = match pfs.select_meta(wal_base_generation, true)? { + Opened::Fresh => MetaPage::empty(1), + Opened::Meta(m) => { + m.check_features()?; + m.check_layout()?; + m + } + Opened::LegacyV7(h) => pfs.load_v7_in_memory(&h)?, + }; + pfs.activate(meta); + pfs.storage + .restore_tx_counter_from(meta.last_checkpointed_tx_count); + Ok(pfs) + } + + fn unopened(backend: B, page_cache_capacity: usize) -> Self { + PersistentFactStorage { + backend: Arc::new(Mutex::new(backend)), + page_cache: Arc::new(PageCache::new(page_cache_capacity)), + storage: FactStorage::new(), + dirty: false, + meta: MetaPage::empty(1), + } + } + + /// Read a format v7 file's facts into memory as uncheckpointed facts, on an + /// empty committed state, for a read-only open. Returns the meta to + /// activate: empty, with the v7 header's tx counter so WAL replay skips + /// the entries the file already holds. + fn load_v7_in_memory(&mut self, header: &LegacyHeaderV7) -> Result { + let mut facts = { + let backend = self.lock()?; + crate::storage::packed_pages::read_all_v7( + &*backend, + 1, + header.fact_page_count_or_derived(), + )? + }; + // As in `migrate_v7`: never rewind past the header's counter. + let max_fact_tx = facts.iter().map(|f| f.tx_count).max().unwrap_or(0); + let max_tx = max_fact_tx.max(header.last_checkpointed_tx_count); + // Pending facts are kept in tx order (the fact log relies on it). + facts.sort_by_key(|f| f.tx_count); + for fact in facts { + let _ = self.storage.load_fact(fact)?; + } + Ok(MetaPage { + last_checkpointed_tx_count: max_tx, + ..MetaPage::empty(1) + }) + } + /// The LRU page cache capacity this storage was constructed with (for testing). #[allow(dead_code)] pub(crate) fn page_cache_capacity(&self) -> usize { @@ -99,8 +159,9 @@ impl PersistentFactStorage { } /// Choose the meta page to open at (spec §4.1.1). Never modifies the file, - /// except to restore page 0 from the migration backup meta (§9 step 5). - fn select_meta(&self, wal_base: Option) -> Result { + /// except to restore page 0 from the migration backup meta (§9 step 5), + /// which `read_only` skips. + fn select_meta(&self, wal_base: Option, read_only: bool) -> Result { let mut backend = self.lock()?; let page_count = backend.page_count()?; if page_count == 0 { @@ -139,8 +200,10 @@ impl PersistentFactStorage { && m.page_count == page_count { m.check_features()?; - backend.write_page(0, &m.encode())?; - backend.sync()?; + if !read_only { + backend.write_page(0, &m.encode())?; + backend.sync()?; + } return Ok(Opened::Meta(m)); } if page_count <= 2 && is_torn_initial_meta(&*backend, &page0)? { @@ -1387,6 +1450,54 @@ mod tests { assert!(points > 20, "expected many crash points"); } + /// A backend over `mem` whose every write and sync fails. + fn no_writes(mem: &MemoryBackend) -> FaultInjectingBackend { + let (backend, config) = FaultInjectingBackend::with_config(mem.clone()); + let mut cfg = config.lock().unwrap(); + cfg.fail_write_after = Some(0); + cfg.fail_sync_after = Some(0); + drop(cfg); + backend + } + + /// Read-only open (#429) of a v7 file, at every point of an interrupted + /// migration (including a torn page 0 that only the backup meta can + /// repair): every fact is there, and nothing is written. + #[test] + fn read_only_open_at_every_point_of_migration_writes_nothing() { + let facts = sample_facts(120); + for torn in [None, Some(0usize), Some(20), Some(2048)] { + for k in 0u64.. { + let mem = v7_file(&facts, 120); + let (backend, config) = FaultInjectingBackend::with_config(mem.clone()); + { + let mut cfg = config.lock().unwrap(); + cfg.fail_write_after = Some(k); + cfg.torn_write_bytes = torn; + } + let done = PersistentFactStorage::open(backend, 16, None).is_ok(); + let before = snapshot(&mem); + let pfs = PersistentFactStorage::open_read_only(no_writes(&mem), 16, None) + .expect("read-only open after an interrupted migration"); + assert_eq!(count_n(&pfs), 120, "every v7 fact is visible"); + assert!(pfs.storage().current_tx_count() >= 120); + drop(pfs); + assert!(snapshot(&mem) == before, "read-only open wrote"); + if done { + break; + } + } + } + } + + #[test] + fn read_only_open_of_an_empty_backend_writes_nothing() { + let mem = MemoryBackend::new(); + let pfs = PersistentFactStorage::open_read_only(no_writes(&mem), 16, None).unwrap(); + assert_eq!(count_n(&pfs), 0); + assert_eq!(mem.page_count().unwrap(), 0); + } + // ── crash atomicity (spec §11) ────────────────────────────────────────── /// Kill `save()` at every page write (cleanly or with a torn page) and every diff --git a/tests/golden_corpus_test.rs b/tests/golden_corpus_test.rs index 6dc899a030..f55db1a1d7 100644 --- a/tests/golden_corpus_test.rs +++ b/tests/golden_corpus_test.rs @@ -11,7 +11,7 @@ use std::collections::BTreeSet; use std::path::{Path, PathBuf}; -use minigraf::{Minigraf, QueryResult, Value}; +use minigraf::{Minigraf, OpenOptions, QueryResult, Value}; use serde_json::Value as Json; /// The major version of this reader, matched against a query's `min_reader`. @@ -170,6 +170,30 @@ fn check_manifest(m: &Manifest, failures: &mut Vec) { std::fs::write(wal_path(&path), b).unwrap(); } let tx_count = m.json["tx_count"].as_u64().expect("tx_count"); + + // Read-only first (#429): the same view, and neither file changes. + { + let opts = OpenOptions::new().read_only(true); + match Minigraf::open_with_options(&path, opts) { + Ok(db) => { + check_tx_count(m, &db, tx_count, "read-only", failures); + check_queries(m, &db, "read-only", failures); + } + Err(e) => failures.push(format!( + "{}: read-only open failed with {}", + m.name, + e.code() + )), + } + if std::fs::read(&path).unwrap() != src_bytes { + failures.push(format!("{}: read-only open modified {file}", m.name)); + } + let wal_now = std::fs::read(wal_path(&path)).ok(); + if wal_now != wal_bytes { + failures.push(format!("{}: read-only open changed the WAL", m.name)); + } + } + { let db = match Minigraf::open(&path) { Ok(db) => db, diff --git a/tests/read_only_open_test.rs b/tests/read_only_open_test.rs new file mode 100644 index 0000000000..0c8167b083 --- /dev/null +++ b/tests/read_only_open_test.rs @@ -0,0 +1,395 @@ +//! `OpenOptions::read_only` (#429): a handle that shows the same data as a +//! read-write open, writes nothing to the `.graph` file or its WAL, refuses +//! writes with API-014, and shares the file with other read-only handles. +#![cfg(not(target_arch = "wasm32"))] + +use minigraf::{FactFilter, Minigraf, OpenOptions, QueryResult}; +use std::path::{Path, PathBuf}; + +fn wal_path(path: &Path) -> PathBuf { + let mut p = path.as_os_str().to_owned(); + p.push(".wal"); + PathBuf::from(p) +} + +fn ro() -> OpenOptions { + OpenOptions::new().read_only(true) +} + +/// The error from an open that must fail (`Minigraf` is not `Debug`). +fn open_err( + result: Result, + what: &str, +) -> minigraf::MinigrafError { + match result { + Ok(_) => panic!("{what}: open must fail"), + Err(e) => e, + } +} + +fn open_ro(path: &Path) -> Minigraf { + Minigraf::open_with_options(path, ro()).expect("read-only open") +} + +const NAMES: &str = "(query [:find ?n :where [?e :person/name ?n]])"; + +fn names(db: &Minigraf) -> Vec { + match db.execute(NAMES).unwrap() { + QueryResult::QueryResults { results, .. } => { + let mut v: Vec = results + .iter() + .map(|r| match &r[0] { + minigraf::Value::String(s) => s.clone(), + _ => panic!("expected a string"), + }) + .collect(); + v.sort(); + v + } + _ => panic!("expected query results"), + } +} + +/// A file whose facts are all checkpointed, with no WAL: alice and carol +/// current, bob asserted then retracted. Returns its tx count. +fn checkpointed(path: &Path) -> u64 { + let db = Minigraf::open(path).unwrap(); + db.execute(r#"(transact [[:alice :person/name "Alice"]])"#) + .unwrap(); + db.execute(r#"(transact [[:bob :person/name "Bob"]])"#) + .unwrap(); + db.checkpoint().unwrap(); + db.execute(r#"(transact [[:carol :person/name "Carol"]])"#) + .unwrap(); + db.execute(r#"(retract [[:bob :person/name "Bob"]])"#) + .unwrap(); + // A clean close checkpoints and removes the WAL. + db.current_tx_count() +} + +/// The same facts, but the last two transactions stay in the WAL. +fn with_pending_wal(path: &Path) -> u64 { + { + let db = Minigraf::open(path).unwrap(); + db.execute(r#"(transact [[:alice :person/name "Alice"]])"#) + .unwrap(); + db.execute(r#"(transact [[:bob :person/name "Bob"]])"#) + .unwrap(); + } + let db = Minigraf::open_with_options( + path, + OpenOptions::new().wal_checkpoint_threshold(usize::MAX), + ) + .unwrap(); + db.execute(r#"(transact [[:carol :person/name "Carol"]])"#) + .unwrap(); + db.execute(r#"(retract [[:bob :person/name "Bob"]])"#) + .unwrap(); + db.current_tx_count() +} + +struct Snapshot { + graph: Vec, + wal: Option>, + graph_mtime: std::time::SystemTime, +} + +fn snapshot(path: &Path) -> Snapshot { + Snapshot { + graph: std::fs::read(path).unwrap(), + wal: std::fs::read(wal_path(path)).ok(), + graph_mtime: std::fs::metadata(path).unwrap().modified().unwrap(), + } +} + +fn assert_unchanged(path: &Path, before: &Snapshot) { + let after = snapshot(path); + assert!(after.graph == before.graph, ".graph bytes changed"); + assert!(after.wal == before.wal, "WAL changed, appeared or vanished"); + assert_eq!( + after.graph_mtime, before.graph_mtime, + ".graph mtime changed" + ); +} + +#[test] +fn shows_the_same_data_as_a_read_write_open() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("same.graph"); + let tx = with_pending_wal(&path); + assert!(wal_path(&path).exists(), "fixture needs a pending WAL"); + + let before = snapshot(&path); + let (ro_names, ro_tx, ro_log) = { + let db = open_ro(&path); + let log: Vec<_> = db + .fact_log(&FactFilter::new()) + .unwrap() + .map(|r| { + let r = r.unwrap(); + (r.tx_count, r.attribute, r.asserted) + }) + .collect(); + (names(&db), db.current_tx_count(), log) + }; + assert_unchanged(&path, &before); + + assert_eq!(ro_names, vec!["Alice".to_string(), "Carol".to_string()]); + assert_eq!(ro_tx, tx); + assert_eq!(ro_log.len(), 4, "3 assertions + 1 retraction"); + + let db = Minigraf::open(&path).unwrap(); + assert_eq!(names(&db), ro_names); + assert_eq!(db.current_tx_count(), ro_tx); + let rw_log: Vec<_> = db + .fact_log(&FactFilter::new()) + .unwrap() + .map(|r| { + let r = r.unwrap(); + (r.tx_count, r.attribute, r.asserted) + }) + .collect(); + assert_eq!(rw_log, ro_log); +} + +#[test] +fn reads_write_nothing() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("untouched.graph"); + with_pending_wal(&path); + let before = snapshot(&path); + { + let db = open_ro(&path); + names(&db); + let mut cursor = db.query(NAMES).unwrap(); + while cursor.next_batch(1).unwrap().is_some() {} + let log = db.fact_log(&FactFilter::new()).unwrap(); + assert_eq!(log.count(), 4); + let report = db.verify().unwrap(); + assert!(report.is_ok(), "verify finds no problems"); + let p = db.prepare("(query [:find ?n :where [?e :person/name ?n]])"); + p.unwrap(); + } + assert_unchanged(&path, &before); +} + +#[test] +fn no_wal_is_created() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("nowal.graph"); + checkpointed(&path); + assert!(!wal_path(&path).exists()); + let before = snapshot(&path); + { + let db = open_ro(&path); + assert_eq!(names(&db), vec!["Alice".to_string(), "Carol".to_string()]); + assert!( + db.execute(r#"(transact [[:dave :person/name "Dave"]])"#) + .is_err() + ); + } + assert!(!wal_path(&path).exists(), "no WAL created"); + assert_unchanged(&path, &before); +} + +#[test] +fn writes_are_api_014_and_change_nothing() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("refuse.graph"); + let tx = with_pending_wal(&path); + let before = snapshot(&path); + { + let db = open_ro(&path); + let cases: [(&str, Result<(), minigraf::MinigrafError>); 5] = [ + ( + "transact", + db.execute(r#"(transact [[:dave :person/name "Dave"]])"#) + .map(|_| ()), + ), + ( + "retract", + db.execute(r#"(retract [[:alice :person/name "Alice"]])"#) + .map(|_| ()), + ), + ("begin_write", db.begin_write().map(|_| ())), + ("checkpoint", db.checkpoint()), + ("rebuild_indexes", db.rebuild_indexes()), + ]; + for (op, result) in cases { + let err = result.expect_err(op); + assert_eq!(err.code(), "API-014", "{op}"); + assert!(err.to_string().contains(op), "{op} named in the message"); + } + assert_eq!(db.current_tx_count(), tx, "no tx count allocated"); + assert_eq!(names(&db), vec!["Alice".to_string(), "Carol".to_string()]); + } + assert_unchanged(&path, &before); +} + +#[test] +fn rules_and_functions_work() { + const RULES: [&str; 2] = [ + "(rule [(reach ?x ?y) [?x :link ?y]])", + "(rule [(reach ?x ?y) [?x :link ?z] (reach ?z ?y)])", + ]; + const REACH: &str = "(query [:find ?x ?y :where (reach ?x ?y)])"; + let reach = |db: &Minigraf| -> usize { + for r in RULES { + db.execute(r).unwrap(); + } + match db.execute(REACH).unwrap() { + QueryResult::QueryResults { results, .. } => results.len(), + _ => panic!("expected query results"), + } + }; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("rules.graph"); + let expected = { + let db = Minigraf::open(&path).unwrap(); + db.execute("(transact [[:a :link :b] [:b :link :c]])") + .unwrap(); + db.checkpoint().unwrap(); + reach(&db) + }; + assert!(expected > 0); + let before = snapshot(&path); + { + let db = open_ro(&path); + assert_eq!(reach(&db), expected); + db.register_predicate("always?", |_| true).unwrap(); + } + assert_unchanged(&path, &before); +} + +#[test] +fn missing_file_is_stg_042_and_not_created() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("missing.graph"); + let err = open_err(Minigraf::open_with_options(&path, ro()), "missing file"); + assert_eq!(err.code(), "STG-042"); + assert!(!path.exists(), "no file created"); + assert!(!wal_path(&path).exists(), "no WAL created"); +} + +#[test] +fn empty_file_is_an_empty_database() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("empty.graph"); + std::fs::write(&path, b"").unwrap(); + { + let db = open_ro(&path); + assert!(names(&db).is_empty()); + assert_eq!(db.current_tx_count(), 0); + assert_eq!(db.fact_log(&FactFilter::new()).unwrap().count(), 0); + } + assert_eq!( + std::fs::metadata(&path).unwrap().len(), + 0, + "not initialised" + ); +} + +#[test] +fn read_only_handles_share_the_file() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("shared.graph"); + with_pending_wal(&path); + let a = open_ro(&path); + let b = open_ro(&path); + assert_eq!(names(&a), names(&b)); + drop(a); + let c = open_ro(&path); + assert_eq!(names(&b), names(&c)); +} + +#[test] +fn readers_and_a_writer_exclude_each_other() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("exclusive.graph"); + with_pending_wal(&path); + + let r1 = open_ro(&path); + let r2 = open_ro(&path); + let err = open_err(Minigraf::open(&path), "writer while readers hold the file"); + assert_eq!(err.code(), "STG-025"); + drop(r1); + let err = open_err( + Minigraf::open(&path), + "writer while one reader holds the file", + ); + assert_eq!(err.code(), "STG-025"); + drop(r2); + + let w = Minigraf::open(&path).unwrap(); + let err = open_err( + Minigraf::open_with_options(&path, ro()), + "reader while a writer holds it", + ); + assert_eq!(err.code(), "STG-025"); + drop(w); + open_ro(&path); +} + +#[test] +fn page_cache_size_applies_read_only() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cache.graph"); + checkpointed(&path); + let pages = std::fs::metadata(&path).unwrap().len() / 4096; + for size in [1, pages as usize] { + let db = Minigraf::open_with_options(&path, ro().page_cache_size(size)).unwrap(); + assert_eq!(names(&db), vec!["Alice".to_string(), "Carol".to_string()]); + } +} + +#[cfg(unix)] +#[test] +fn opens_a_file_without_write_permission() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("perm.graph"); + checkpointed(&path); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o444)).unwrap(); + // Root ignores file modes, so the read-write open below would succeed. + let as_root = std::fs::OpenOptions::new().write(true).open(&path).is_ok(); + if !as_root { + assert!( + Minigraf::open(&path).is_err(), + "read-write open needs write access" + ); + } + let db = open_ro(&path); + assert_eq!(names(&db), vec!["Alice".to_string(), "Carol".to_string()]); +} + +#[test] +fn in_memory_read_only_refuses_writes() { + let db = OpenOptions::new().read_only(true).open_memory().unwrap(); + let err = db + .execute(r#"(transact [[:a :person/name "A"]])"#) + .expect_err("transact"); + assert_eq!(err.code(), "API-014"); + assert_eq!( + db.begin_write() + .map(|_| ()) + .expect_err("begin_write") + .code(), + "API-014" + ); + assert_eq!(db.checkpoint().expect_err("checkpoint").code(), "API-014"); + assert!(names(&db).is_empty()); +} + +#[test] +fn builder_path_open_is_read_only() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("builder.graph"); + checkpointed(&path); + let db = ro().path(&path).open().unwrap(); + assert_eq!( + db.execute(r#"(transact [[:x :person/name "X"]])"#) + .expect_err("transact") + .code(), + "API-014" + ); +}