diff --git a/CHANGELOG.md b/CHANGELOG.md index 641d71f908..624d861bf0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened; while one is open, checkpoints are deferred (writes stay in the WAL, automatic checkpoints wait, and `checkpoint()` or `rebuild_indexes()` with work to do fail with the new `API-013`). Close, drop or finish the log to release it. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow. +- **Raw log writer: `LogWriter`** (#431). Builds a new database from explicit `FactRecord`s, the load step of an offline migration after `fact_log()` on a read-only source. Each record keeps its `tx_count`, `tx_id`, valid-time window and assertion flag. Records come in transaction order, gaps allowed: leaving a transaction out leaves a hole, and `:as-of` on it shows the state before. `advance_tx_count` keeps the source's counter after trailing purged transactions. The checks are structural only: order (new `API-015`), one `tx_id` per transaction (new `API-016`), one valid-time window per fact per transaction (`API-011`) and size (`WAL-003`); a rejected record changes nothing. There is no WAL: records are committed in batches with the normal copy-on-write checkpoint, never splitting a transaction. The file is built at `.partial` and renamed into place by `finish`, so a crash or an abandoned build never leaves part of a log at `path`; an existing target or WAL is the new `STG-043`. The page-range copy the issue proposed is not provided, since v8 pages share one dictionary and are not self-contained. The bindings follow. - **Read-only open: `OpenOptions::read_only`** (#429). Opens a database so that nothing done through the handle writes to the `.graph` file or its WAL. The file is locked in shared mode: any number of read-only handles, in this process or others, can hold it at once, and they exclude a read-write open (and it excludes them) with the existing STG-025/STG-026. The handle shows the same data as a read-write open: a WAL left by an earlier session is applied in memory but not checkpointed or deleted, and a format v7 file is read into memory instead of migrated. A missing file is the new `STG-042` (nothing is created). `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` fail with the new `API-014`; queries, cursors, `fact_log`, `verify` and rule registration work, and dropping the handle does not checkpoint. `page_cache_size` applies as usual, so a tool can keep the whole file resident. Rust API only for now; the bindings follow. - **Cursor API: `Minigraf::query()` and `PreparedQuery::query()`** (#432). Both return a `Cursor` that hands a query's rows back in batches: `vars()`, `next_batch(max_rows) -> Result, _>`, `close()`, and an iterator over rows. The answer is fixed when the cursor opens, so writes that commit while it is open do not change it. A cursor borrows nothing from the database handle: it can outlive it and move to another thread. For now the answer is still computed when the cursor opens, under the same limits as `execute()`; the API is shaped so the engine can later produce rows as they are pulled without breaking callers. `query()` rejects `transact`, `retract` and `rule` with the new `API-012`, and a query with bind slots with `API-010`. `execute()` is unchanged. Rust API only for now; the bindings follow. - **`Minigraf::verify()` and `Minigraf::rebuild_indexes()`** (#373). Page checksums turn a torn or rotted page into an error, but they cannot see a page that is valid and says the wrong thing. `verify()` walks every committed page and returns an `IntegrityReport` with one structured error per finding: an index that holds different facts than EAVT or a different count than the meta page (new `STG-038`), keys out of order or a page reached twice (new `STG-039`), dictionary maps that disagree (new `STG-040`), ids with no dictionary entry (`STG-036`), and leaked, doubly used or wrongly free pages (`STG-035`). It keeps no per-fact state: indexes are compared by an order-independent digest. `rebuild_indexes()` rebuilds EAVT, AEVT, AVET and VAET from an intact index that agrees with another one (or the only intact one), derives the free list again, and commits like a checkpoint, including uncheckpointed writes; a crash at any point leaves the previous checkpoint. If no index can serve as the source, or the dictionary is damaged, it fails with new `STG-041` and writes nothing. Both are Rust API only for now; the bindings follow. diff --git a/CLAUDE.md b/CLAUDE.md index 98780a9887..a3cd5f26cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -141,6 +141,7 @@ cargo run < demos/demo_negation.txt 6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`, `OpenOptions::read_only` (#429: shared lock, nothing written, writes are `API-014`) - `src/cursor.rs`: `Cursor` / `Batch` — owned, `Send` cursor returned by `Minigraf::query` and `PreparedQuery::query` (#432); answer computed at open today, shaped for streaming operators - `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation, deferring checkpoints (`API-013`) + - `src/log_writer.rs`: `LogWriter` — builds a new file from `FactRecord`s with their tx and valid-time kept (#431): tx-ordered appends with holes (`API-015`/`API-016`), batched copy-on-write checkpoints at tx boundaries, no WAL, built at `.partial` and renamed by `finish` (`STG-043` if the target exists) 7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery @@ -196,7 +197,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are ## Test Coverage -**1329 tests** (1320 passing, 9 ignored; unit + integration + doc). +**1345 tests** (1336 passing, 9 ignored; unit + integration + doc). See `docs/TEST_COVERAGE.md` for the full per-file breakdown. **Testing conventions** — see the Testing Conventions section below before writing any tests. diff --git a/README.md b/README.md index bbff5e3da7..769caf354d 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ v2.x gets data-integrity and security fixes for 12 months after v3.0.0 ships. Se ## Quick Start ```rust -use minigraf::{FactFilter, Minigraf, OpenOptions}; +use minigraf::{FactFilter, LogWriter, Minigraf, OpenOptions}; // Open or create a file-backed database let db = OpenOptions::new().path("myapp.graph").open()?; @@ -100,6 +100,16 @@ while let Some(batch) = log.next_batch(1000)? { // a read-write handle (like `db` on myapp.graph) excludes them. let archive = Minigraf::open_with_options("archive.graph", OpenOptions::new().read_only(true))?; +// Log writer — copy records into a new file with their transactions and valid-time +// windows kept; leave records out to purge them +let mut out = LogWriter::create("archive-clean.graph", OpenOptions::new())?; +for rec in archive.fact_log(&FactFilter::new())? { + let rec = rec?; + if rec.attribute != ":person/ssn" { out.append(&rec)?; } +} +out.advance_tx_count(archive.current_tx_count())?; +out.finish()?; + // Explicit transaction — all-or-nothing let mut tx = db.begin_write()?; tx.execute(r#"(transact [[:alice :person/age 31]])"#)?; @@ -121,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit ```bash cargo run # interactive Datalog REPL -cargo test # run 1329 tests +cargo test # run 1345 tests cargo run < demos/demo_recursive.txt # recursive rules demo ``` diff --git a/ROADMAP.md b/ROADMAP.md index b93cd58cab..73557819ac 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -70,6 +70,7 @@ Scope — query API: - Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`, with the current evaluator computing the answer at open). The binding cursor objects follow. Streaming operators replace the evaluator behind the cursor in 3.x without further API changes; how operators bound their memory (spill, scratch pages in the `.graph` file, or a clear error) is still open. - Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462. - Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows. +- Raw log writer: `LogWriter` builds a new database from `FactRecord`s with their `tx_count`, `tx_id`, valid-time windows and assertion flags kept, gaps allowed (a purged transaction stays a hole for `:as-of`), at `.partial` renamed into place by `finish` (#431; Rust API done on `v3`). Completes extract (#429, #430), transform, load. The binding objects follow. Scope — release-gate testing and documentation: diff --git a/docs/ERROR_REFERENCE.md b/docs/ERROR_REFERENCE.md index 95d2bc29f1..9c74c0d38d 100644 --- a/docs/ERROR_REFERENCE.md +++ b/docs/ERROR_REFERENCE.md @@ -151,6 +151,7 @@ with no `CodedError` anywhere in its chain. | STG-040 | Dictionary inconsistency | Storage | | STG-041 | Cannot rebuild indexes | Storage | | STG-042 | Database file not found (read-only open) | Storage | +| STG-043 | Log writer target already exists | Storage | | WAL-001 | Invalid WAL magic number | WAL | | WAL-002 | Unsupported WAL version | WAL | | WAL-003 | Value size exceeds maximum | WAL | @@ -171,6 +172,8 @@ with no `CodedError` anywhere in its chain. | API-012 | Non-query command passed to query() | Database API | | API-013 | Checkpoint while a fact log is open | Database API | | API-014 | Write on a read-only handle | Database API | +| API-015 | Log writer record out of transaction order | Database API | +| API-016 | Log writer record with a second tx_id for one transaction | Database API | | INT-000 | Unclassified internal error | Internal | | INT-001 | WriteTransaction already in progress on this thread | Internal | | INT-002 | Invalid entity (API layer) | Internal | @@ -2199,6 +2202,25 @@ A read-only handle (`OpenOptions::read_only`) in another process also refuses a **Scenario**: A migration tool is pointed at `memory.graph` from the wrong working directory. + +### STG-043 Log writer target already exists + +**Error text**: `{} already exists; a log writer only creates a new database` + +**Cause**: `LogWriter::create` was given a path where a file, or that path's WAL (`.wal`), already exists; or `LogWriter::finish` found that a file appeared at the path while the build ran. A log writer never overwrites or extends a database, and a stale WAL beside the new file would be replayed onto it. The `{}` is the path that exists. At `finish`, the existing file is left untouched and the build (`.partial`) is deleted. + +**Resolution**: +- Write to a new path, then replace the old database yourself once the new one is checked (`verify()`, your own audit). +- If the old file and WAL are no longer needed, delete them first. +- Do not create the target while a build runs. + +```rust +let mut out = LogWriter::create("memory.v2.graph", OpenOptions::new())?; +// ... append, then out.finish()?; then swap files +``` + +**Scenario**: A migration is re-run after a successful first run, with the same output path. + --- ## WAL — Write-Ahead Log Errors @@ -2235,7 +2257,7 @@ The WAL is replayed on open and deleted on checkpoint. **Error text**: `Value of {} bytes exceeds the maximum of {} bytes. Store large payloads externally and reference them with a Value::String URL/path or Value::Ref entity ID.` -**Cause**: A string value is longer than `MAX_VALUE_BYTES` (4,068 bytes, one value page), or an attribute name or keyword value is longer than 1,024 bytes. This typically means a `Value::String` contains very large content such as raw document text, a base64-encoded image, or binary data. In-memory databases have no WAL and no limit. +**Cause**: A string value is longer than `MAX_VALUE_BYTES` (4,068 bytes, one value page), or an attribute name or keyword value is longer than 1,024 bytes. This typically means a `Value::String` contains very large content such as raw document text, a base64-encoded image, or binary data. `LogWriter::append` checks the same limits. In-memory databases have no WAL and no limit. **Resolution**: - Store large payloads in an external file or object store. @@ -2456,7 +2478,7 @@ let result = pq.execute(&[ **Error text**: `one transaction asserts the same value of {} with two valid-time windows; assert it once with its final window` -**Cause**: One transaction asserts the same entity, attribute and value twice with different `:valid-from`/`:valid-to` bounds. At any transaction time a fact has exactly one current valid-time window, and the window of the latest transaction applies. Two windows in the same transaction leave no latest one, so the transaction is rejected and nothing is written. The `{}` is the attribute. This can come from one `(transact ...)` with per-fact bounds, or from several `execute()` calls inside one `WriteTransaction`. +**Cause**: One transaction asserts the same entity, attribute and value twice with different `:valid-from`/`:valid-to` bounds. At any transaction time a fact has exactly one current valid-time window, and the window of the latest transaction applies. Two windows in the same transaction leave no latest one, so the transaction is rejected and nothing is written. The `{}` is the attribute. This can come from one `(transact ...)` with per-fact bounds, from several `execute()` calls inside one `WriteTransaction`, or from `LogWriter::append` records of one transaction (that record alone is rejected). **Resolution**: - Assert the fact once per transaction, with the window it should have. @@ -2531,6 +2553,31 @@ let mut log = src.fact_log(&FactFilter::new())?; **Scenario**: An offline migration tool opens its source read-only and, by mistake, runs a cleanup `retract` against it instead of against the destination. + +### API-015 Log writer record out of transaction order + +**Error text**: `tx_count {} is out of order: the log writer is at {}, and a record must open a later transaction or join the open one` + +**Cause**: `LogWriter::append` received a record whose `tx_count` is 0, lower than the writer's current `tx_count`, or equal to it after that transaction was closed (by `advance_tx_count`, or by a record of a later transaction). The same code is returned by `advance_tx_count` with a value below the writer's. Records must come in non-decreasing `tx_count` order, each transaction's records together. The record is not written and the writer stays usable. + +**Resolution**: +- Read the source with `fact_log()` in the default `FactOrder::Tx`, which returns records in non-decreasing `tx_count` with each transaction's records together. `FactOrder::Storage` does not, so sort its records by `tx_count` first. +- Gaps are allowed: skip a transaction's records to purge it. +- Call `advance_tx_count` only after the last record, or with a value above every record still to come. + +**Scenario**: A transform step merges two sources and appends their records one source after the other. + +### API-016 Log writer record with a second tx_id for one transaction + +**Error text**: `transaction {} has tx_id {}; a record with tx_id {} cannot join it` + +**Cause**: `LogWriter::append` received a record with the open transaction's `tx_count` but a different `tx_id`. A transaction has one wall-clock time, which `:as-of` with a timestamp compares against. The record is not written and the writer stays usable. + +**Resolution**: +- Keep each record's `tx_id` as the source gave it. If a transform rewrites timestamps, give every record of one transaction the same new `tx_id`. + +**Scenario**: A transform sets `tx_id` from a per-record field instead of from the transaction. + --- ## INT — Internal Errors diff --git a/docs/TEST_COVERAGE.md b/docs/TEST_COVERAGE.md index 9e3e88cfd1..398537a5a5 100644 --- a/docs/TEST_COVERAGE.md +++ b/docs/TEST_COVERAGE.md @@ -4,9 +4,9 @@ **Verified**: 2026-10-07 with `cargo test --quiet` -**Result**: 1,320 passing tests and 9 ignored tests (1,329 total) +**Result**: 1,336 passing tests and 9 ignored tests (1,345 total) -This covers the unreleased file-format-v8 work (#287, #371): the index-key value-collision fix, the v7→v8 migration, and the drop of formats v1–v6. It otherwise still covers the public database API, the Datalog engine, storage and recovery paths, v2.0.0's locking and structured-error-code changes, and v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Merged from the v2.x line: `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a damaged slot directory is rejected rather than copied (#315). B+tree cursor tests check that every scan walks the tree, not the leaf `next_leaf` chain: full scans and start positions match the input, `seek` from the current position lands where a fresh descent from the root would (randomized, plus within a leaf, to a sibling, across subtrees, past the end, into the gap after a leaf, and backwards as a no-op), a seek to the sibling leaf reads exactly one page, reads are unchanged with every `next_leaf` zeroed, a child pointer into a non-B+tree page is STG-013, and a cycle in child pointers is an error, not a hang (v8 storage format PR 1, #374). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389). Disjunction tests check that `or-join` returns an empty result, not INT-031, when no rows reach it: on an empty database, after a pattern that matches nothing, after a `not` that removes every row, and in a rule body (#405). A stratification unit test builds the same two-rule negative cycle 64 times and checks that the INT-054 message names the same predicate pair every time (#410). Magic-sets tests check that a recursive rule with a literal start that recurses through another rule returns the same results as without the literal (#297). API tests check that `execute()` and `WriteTransaction::execute()` reject `$slot` bind tokens in every position with API-010 (#407). The REPL `run()` test now runs in a child process with null stdin, so it no longer hangs when the test runner's stdin is an open non-TTY stream (#415). v8 page-format tests (PR 2, #374, #388) check the 24-byte page header (a flipped bit, a wrong page id and a future generation each give their own STG code, legacy page types fail before the CRC, and a failed page is never cached), meta-page encoding, the free-list chain, WAL v2's base generation, and every row of the meta-selection table: both slots valid, a torn newest slot with the WAL's base generation (opens at the older one), a WAL newer than the valid meta, pages of the next generation on the free list or past `page_count` with no WAL (all STG-033, with the file unmodified), an unknown feature bit (STG-034), and a pre-release v8 single header (STG-032). Fault-injection tests crash `save()` at every page write (cleanly and with torn pages, including the meta write) and every sync, and crash a v7 migration at every write; each reopen finds exactly the old or the new checkpoint, and every reachable page plus the free list covers the file exactly. Twenty-five consecutive saves never write a page the previous meta references. Covering-key tests (PR 3, #433) check that byte order equals logical order for every key component and value type on random tuples (negative, zero and large integers, NaN and ±0.0, strings with 0x00, 64- and 65-byte strings, FOREVER, newest-transaction-first) and that every component round-trips; that prefix-compressed leaves and shortest-separator internal nodes round-trip, agree with a linear scan, keep `max(left) < s ≤ min(right)`, and reject an overflow; that the byte-key B+tree's builds, incremental rebuilds, cursor and `seek` match their expected sets; that dictionary ids are assigned once, never reused, shared between a keyword attribute and value, and deterministic; that a re-asserted or retracted long value writes no new value page and a forced hash collision still finds the right value; that one transaction with two timestamps is STG-037 and a missing dictionary entry STG-036; that committed reads by entity, entity and attribute, attribute, and in full equal an in-memory model across eight checkpoints and a reopen; that short-valued reads touch no value page; that a v7 file with every value type, long strings, refs and retractions migrates to exactly the same facts; that a pre-covering-key v8 file is STG-032; and that string values over 4,068 bytes and attributes or keywords over 1,024 bytes are WAL-003. A WAL test checks that reopening next to a fully checkpointed WAL does not reuse transaction numbers (#447). Copy-on-write checkpoint tests (PR 4, #434) check that lazy free-list pops and pushes conserve every page across page boundaries, partial head pages and chain exhaustion, never hand out a page freed in the same generation, and share unread chain pages unchanged; that a copy-on-write insert equals a sorted merge for random batches (including keys below the first and above the last, many into one leaf, root growth to depth 3, and 1 KB separators that split internal nodes), leaves the old tree readable and unchanged, frees exactly the replaced nodes, writes only new pages, writes exactly one leaf and two internal nodes for one entry in a depth-3 tree, and does not fragment leaves under single inserts; that a checkpoint after 1 or 100 right-edge facts writes at most the tree-depth difference more pages at 100,000 facts than at 10,000, and at most one path per new entry per tree for random keys; and that four reader threads querying throughout 40 checkpoints never fail and never see the fact count go backwards. Hardening tests (PR 5) damage an index leaf, an internal node, a dictionary leaf, a value page and a free-list page in turn. Every committed read or checkpoint that reaches the damaged page fails with STG-029, never with wrong or fewer rows. A query through `Minigraf` on a file with damaged leaves fails with STG-029, and the file still opens at its last checkpoint. A size test measures at most 200 bytes per fact (142 measured) on 144k facts in #433's shape; its 1.15M-fact variant is ignored locally and runs in the nightly smoke workflow. A shared dictionary-cache test checks that cached ident and timestamp lookups return the same facts and that the cache stays bounded. Merged from the v2.x line: storage and `edge_cases_test.rs` tests check that more than 65,535 uncheckpointed facts each resolve to their own fact through entity lookups and attribute scans, in memory and file-backed (#445). Merged from the v2.x line: `tests/golden_corpus_test.rs` opens a copy of each committed golden file in `tests/golden/` (v7 files written by minigraf 2.0.3: one checkpoint, several checkpoints, indexes rebuilt on open, a crash-left WAL, an already-checkpointed WAL, same-transaction multi-values) and checks its manifest's CRC32, `tx_count` and query results after open, after a checkpoint and reopen, and after one more write; a second test checks that every golden file has a manifest (#391). On `v3` the same test checks that each v7 file has a v8 meta page after its first open and gives the same results after reopening the migrated file, and adds seven v8 golden files (one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values at the size limits); the #371 queries, skipped on v2.x, run and pass. Integrity tests (#373) check that intact files verify with no problems and an exact fact and page count, and that `verify` reports each kind of damage with its code: an EAVT, AEVT, AVET or VAET leaf missing an entry (STG-038), keys out of order and a page reached from two trees (STG-039), a rotted index leaf (STG-029), a free page still in use and a leaked page (STG-035), a missing transaction entry (STG-036), and an entity id whose UUID maps to another id (STG-040). After each index or free-list damage, `rebuild_indexes` leaves a file that verifies clean, reads back every fact and accounts for every page. The source is an index that agrees with another one, or the only intact one; three intact indexes that all disagree, or a damaged dictionary, give STG-041 with the file unchanged. A rebuild commits pending facts, including new ids and long values. A crash at every write (clean and torn) and every sync of a rebuild reopens either at the old generation with only its old damage or at the new one with none, never losing a fact. Through `Minigraf` (`tests/integrity_test.rs`), a rebuild commits WAL entries and deletes the WAL, and flipping a byte in each leaf of a file in turn is always reported, then either repaired (index leaves) or refused with STG-041 and the file unchanged (dictionary leaves). Live-read tests (#379) check that `triple_len` ends exactly where `tx↓` begins for every value type and index; that on random histories (several valid-time windows, retractions, re-assertions, several values per attribute, long values, 12 transactions per checkpoint) the committed live read for every scan kind and `:as-of` point is exactly the net-asserted set, that it stays equivalent with pending facts on top, and after reopen; and that a point read on a triple asserted and retracted 2,000 times reads under a third of the pages a history read does, with or without `:as-of`. `tests/live_reads_test.rs` runs the same history in a checkpointed file database and in memory and checks that point, attribute, full-scan, join, valid-time, `:as-of` and rule queries return the same rows, before and after reopen. One-window tests (#435) check that `net_asserted_facts` keeps each triple's newest transaction only (a later window replaces an earlier one, closing an open window, a retraction in the newest transaction hides it, records with two windows in one transaction keep both, once each) against a per-triple model on random histories; that the on-disk key walk agrees at every `:as-of` before and after reopen; and that `API-011` rejects two windows of one fact in one transaction without taking a `tx_count`. `tests/valid_time_window_test.rs` checks replacing, closing, extending, reopening and retract-then-reassert through entity-bound and attribute queries in memory, on a checkpointed file and after reopen, and `API-011` from `transact` and from `WriteTransaction::commit`. Cursor tests (#432, `tests/cursor_test.rs`) check that `query()` batches partition the `execute()` answer for several batch sizes (never empty, never over `max_rows`, `max_rows == 0` still progresses, `None` stays `None`), that the iterator and aggregates, rules, `:as-of` and `:any-valid-time` match `execute()`, that a cursor opened before a `transact`, a `WriteTransaction` commit, a retract or a checkpoint returns the pre-write answer to the end (in memory and file-backed), that `transact`/`retract`/`rule` are `API-012` and bind slots `API-010`, that a `WriteTransaction` thread gets `INT-001`, that rule limits give `INT-020` as from `execute()`, that `PreparedQuery::query` matches `PreparedQuery::execute`, and that a cursor is `Send` and outlives its database handle. Fact-log tests (#430, `tests/fact_log_test.rs`) check that `fact_log()` returns exactly the records written (assertions and retractions, with `tx_count` and `tx_id`) in memory, with everything pending, half checkpointed, fully checkpointed and after reopen, in both orders, with windows of 1 to 1,000 records and batch sizes of 1 to 1,000, in non-decreasing `tx_count` in Tx order; that every layout gives identical records, including valid-time windows and long string values; that a transaction larger than the window is split with no loss or duplicate; that attribute, attribute-prefix, entity and tx-range filters and their combinations match a model, and unknown names match nothing; that writes after open are not in the log; that `checkpoint()` and `rebuild_indexes()` are `API-013` while a log is open (a checkpoint with nothing to write is allowed), that automatic checkpoints from `execute()` and `WriteTransaction::commit` wait for the log, and that closing, dropping or finishing the log releases the database; that a `WriteTransaction` thread gets `INT-001`; and that a log is `Send`, outlives its database handle, and the deferred writes replay on the next open. Read-only open tests (#429, `tests/read_only_open_test.rs`) check that a read-only handle on a file with checkpointed facts and a pending WAL returns the same query results, `tx_count` and fact log as a read-write open; that queries, a cursor, a fact log, `verify`, `prepare`, rules and a UDF leave the `.graph` and WAL bytes and the `.graph` mtime unchanged, and no WAL is created; that `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` are `API-014` without taking a `tx_count`, also on a read-only in-memory database and through the builder path; that a missing file is `STG-042` and nothing is created; that an empty file is an empty database and stays 0 bytes; that read-only handles share a file while a read-write one is refused with `STG-025`, both ways; that a file without write permission opens read-only; and that small and whole-file page caches work. File-backend unit tests check that a shared-mode backend refuses page writes and syncs with `INT-056`, that the in-process registry counts readers and admits readers or one writer (also without kernel locks), and that a reader in another process shares the file with a reader here and refuses a writer here with `STG-026`. Storage unit tests check that a read-only open of a v7 file at every point of an interrupted migration (clean and torn writes) sees every fact and writes nothing, and that an empty backend stays empty. `tests/golden_corpus_test.rs` opens a copy of every golden file read-only first and checks its `tx_count` and queries, and that neither the file nor its WAL changes. +This covers the unreleased file-format-v8 work (#287, #371): the index-key value-collision fix, the v7→v8 migration, and the drop of formats v1–v6. It otherwise still covers the public database API, the Datalog engine, storage and recovery paths, v2.0.0's locking and structured-error-code changes, and v2.0.1's regression test for rebuilding indexes on open after several checkpoints (#370). Merged from the v2.x line: `tests/point_query_history_test.rs` and new storage unit tests check that attribute-narrowed bound-entity lookups return exactly what a full scan returns, and that `net_asserted_facts` matches its previous implementation on randomized histories (#323). Attribute-scan tests check that the AEVT range covers exactly the queried attribute, including non-ASCII names such as `:丿` and prefix siblings such as `:ab` for `:a` (#381). Incremental B+tree rebuild tests check that checkpoints which copy untouched index leaves produce exactly the entries and file checksum a full rebuild would, including over repeated checkpoints, that single-fact checkpoints do not fragment the index, and that a damaged slot directory is rejected rather than copied (#315). B+tree cursor tests check that every scan walks the tree, not the leaf `next_leaf` chain: full scans and start positions match the input, `seek` from the current position lands where a fresh descent from the root would (randomized, plus within a leaf, to a sibling, across subtrees, past the end, into the gap after a leaf, and backwards as a no-op), a seek to the sibling leaf reads exactly one page, reads are unchanged with every `next_leaf` zeroed, a child pointer into a non-B+tree page is STG-013, and a cycle in child pointers is an error, not a hang (v8 storage format PR 1, #374). Directory-sync unit tests check that creating the `.graph` file or the WAL, and deleting the WAL, fsync the parent directory after the file operation, and that reopening an existing file does not (#389). Disjunction tests check that `or-join` returns an empty result, not INT-031, when no rows reach it: on an empty database, after a pattern that matches nothing, after a `not` that removes every row, and in a rule body (#405). A stratification unit test builds the same two-rule negative cycle 64 times and checks that the INT-054 message names the same predicate pair every time (#410). Magic-sets tests check that a recursive rule with a literal start that recurses through another rule returns the same results as without the literal (#297). API tests check that `execute()` and `WriteTransaction::execute()` reject `$slot` bind tokens in every position with API-010 (#407). The REPL `run()` test now runs in a child process with null stdin, so it no longer hangs when the test runner's stdin is an open non-TTY stream (#415). v8 page-format tests (PR 2, #374, #388) check the 24-byte page header (a flipped bit, a wrong page id and a future generation each give their own STG code, legacy page types fail before the CRC, and a failed page is never cached), meta-page encoding, the free-list chain, WAL v2's base generation, and every row of the meta-selection table: both slots valid, a torn newest slot with the WAL's base generation (opens at the older one), a WAL newer than the valid meta, pages of the next generation on the free list or past `page_count` with no WAL (all STG-033, with the file unmodified), an unknown feature bit (STG-034), and a pre-release v8 single header (STG-032). Fault-injection tests crash `save()` at every page write (cleanly and with torn pages, including the meta write) and every sync, and crash a v7 migration at every write; each reopen finds exactly the old or the new checkpoint, and every reachable page plus the free list covers the file exactly. Twenty-five consecutive saves never write a page the previous meta references. Covering-key tests (PR 3, #433) check that byte order equals logical order for every key component and value type on random tuples (negative, zero and large integers, NaN and ±0.0, strings with 0x00, 64- and 65-byte strings, FOREVER, newest-transaction-first) and that every component round-trips; that prefix-compressed leaves and shortest-separator internal nodes round-trip, agree with a linear scan, keep `max(left) < s ≤ min(right)`, and reject an overflow; that the byte-key B+tree's builds, incremental rebuilds, cursor and `seek` match their expected sets; that dictionary ids are assigned once, never reused, shared between a keyword attribute and value, and deterministic; that a re-asserted or retracted long value writes no new value page and a forced hash collision still finds the right value; that one transaction with two timestamps is STG-037 and a missing dictionary entry STG-036; that committed reads by entity, entity and attribute, attribute, and in full equal an in-memory model across eight checkpoints and a reopen; that short-valued reads touch no value page; that a v7 file with every value type, long strings, refs and retractions migrates to exactly the same facts; that a pre-covering-key v8 file is STG-032; and that string values over 4,068 bytes and attributes or keywords over 1,024 bytes are WAL-003. A WAL test checks that reopening next to a fully checkpointed WAL does not reuse transaction numbers (#447). Copy-on-write checkpoint tests (PR 4, #434) check that lazy free-list pops and pushes conserve every page across page boundaries, partial head pages and chain exhaustion, never hand out a page freed in the same generation, and share unread chain pages unchanged; that a copy-on-write insert equals a sorted merge for random batches (including keys below the first and above the last, many into one leaf, root growth to depth 3, and 1 KB separators that split internal nodes), leaves the old tree readable and unchanged, frees exactly the replaced nodes, writes only new pages, writes exactly one leaf and two internal nodes for one entry in a depth-3 tree, and does not fragment leaves under single inserts; that a checkpoint after 1 or 100 right-edge facts writes at most the tree-depth difference more pages at 100,000 facts than at 10,000, and at most one path per new entry per tree for random keys; and that four reader threads querying throughout 40 checkpoints never fail and never see the fact count go backwards. Hardening tests (PR 5) damage an index leaf, an internal node, a dictionary leaf, a value page and a free-list page in turn. Every committed read or checkpoint that reaches the damaged page fails with STG-029, never with wrong or fewer rows. A query through `Minigraf` on a file with damaged leaves fails with STG-029, and the file still opens at its last checkpoint. A size test measures at most 200 bytes per fact (142 measured) on 144k facts in #433's shape; its 1.15M-fact variant is ignored locally and runs in the nightly smoke workflow. A shared dictionary-cache test checks that cached ident and timestamp lookups return the same facts and that the cache stays bounded. Merged from the v2.x line: storage and `edge_cases_test.rs` tests check that more than 65,535 uncheckpointed facts each resolve to their own fact through entity lookups and attribute scans, in memory and file-backed (#445). Merged from the v2.x line: `tests/golden_corpus_test.rs` opens a copy of each committed golden file in `tests/golden/` (v7 files written by minigraf 2.0.3: one checkpoint, several checkpoints, indexes rebuilt on open, a crash-left WAL, an already-checkpointed WAL, same-transaction multi-values) and checks its manifest's CRC32, `tx_count` and query results after open, after a checkpoint and reopen, and after one more write; a second test checks that every golden file has a manifest (#391). On `v3` the same test checks that each v7 file has a v8 meta page after its first open and gives the same results after reopening the migrated file, and adds seven v8 golden files (one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values at the size limits); the #371 queries, skipped on v2.x, run and pass. Integrity tests (#373) check that intact files verify with no problems and an exact fact and page count, and that `verify` reports each kind of damage with its code: an EAVT, AEVT, AVET or VAET leaf missing an entry (STG-038), keys out of order and a page reached from two trees (STG-039), a rotted index leaf (STG-029), a free page still in use and a leaked page (STG-035), a missing transaction entry (STG-036), and an entity id whose UUID maps to another id (STG-040). After each index or free-list damage, `rebuild_indexes` leaves a file that verifies clean, reads back every fact and accounts for every page. The source is an index that agrees with another one, or the only intact one; three intact indexes that all disagree, or a damaged dictionary, give STG-041 with the file unchanged. A rebuild commits pending facts, including new ids and long values. A crash at every write (clean and torn) and every sync of a rebuild reopens either at the old generation with only its old damage or at the new one with none, never losing a fact. Through `Minigraf` (`tests/integrity_test.rs`), a rebuild commits WAL entries and deletes the WAL, and flipping a byte in each leaf of a file in turn is always reported, then either repaired (index leaves) or refused with STG-041 and the file unchanged (dictionary leaves). Live-read tests (#379) check that `triple_len` ends exactly where `tx↓` begins for every value type and index; that on random histories (several valid-time windows, retractions, re-assertions, several values per attribute, long values, 12 transactions per checkpoint) the committed live read for every scan kind and `:as-of` point is exactly the net-asserted set, that it stays equivalent with pending facts on top, and after reopen; and that a point read on a triple asserted and retracted 2,000 times reads under a third of the pages a history read does, with or without `:as-of`. `tests/live_reads_test.rs` runs the same history in a checkpointed file database and in memory and checks that point, attribute, full-scan, join, valid-time, `:as-of` and rule queries return the same rows, before and after reopen. One-window tests (#435) check that `net_asserted_facts` keeps each triple's newest transaction only (a later window replaces an earlier one, closing an open window, a retraction in the newest transaction hides it, records with two windows in one transaction keep both, once each) against a per-triple model on random histories; that the on-disk key walk agrees at every `:as-of` before and after reopen; and that `API-011` rejects two windows of one fact in one transaction without taking a `tx_count`. `tests/valid_time_window_test.rs` checks replacing, closing, extending, reopening and retract-then-reassert through entity-bound and attribute queries in memory, on a checkpointed file and after reopen, and `API-011` from `transact` and from `WriteTransaction::commit`. Cursor tests (#432, `tests/cursor_test.rs`) check that `query()` batches partition the `execute()` answer for several batch sizes (never empty, never over `max_rows`, `max_rows == 0` still progresses, `None` stays `None`), that the iterator and aggregates, rules, `:as-of` and `:any-valid-time` match `execute()`, that a cursor opened before a `transact`, a `WriteTransaction` commit, a retract or a checkpoint returns the pre-write answer to the end (in memory and file-backed), that `transact`/`retract`/`rule` are `API-012` and bind slots `API-010`, that a `WriteTransaction` thread gets `INT-001`, that rule limits give `INT-020` as from `execute()`, that `PreparedQuery::query` matches `PreparedQuery::execute`, and that a cursor is `Send` and outlives its database handle. Fact-log tests (#430, `tests/fact_log_test.rs`) check that `fact_log()` returns exactly the records written (assertions and retractions, with `tx_count` and `tx_id`) in memory, with everything pending, half checkpointed, fully checkpointed and after reopen, in both orders, with windows of 1 to 1,000 records and batch sizes of 1 to 1,000, in non-decreasing `tx_count` in Tx order; that every layout gives identical records, including valid-time windows and long string values; that a transaction larger than the window is split with no loss or duplicate; that attribute, attribute-prefix, entity and tx-range filters and their combinations match a model, and unknown names match nothing; that writes after open are not in the log; that `checkpoint()` and `rebuild_indexes()` are `API-013` while a log is open (a checkpoint with nothing to write is allowed), that automatic checkpoints from `execute()` and `WriteTransaction::commit` wait for the log, and that closing, dropping or finishing the log releases the database; that a `WriteTransaction` thread gets `INT-001`; and that a log is `Send`, outlives its database handle, and the deferred writes replay on the next open. Read-only open tests (#429, `tests/read_only_open_test.rs`) check that a read-only handle on a file with checkpointed facts and a pending WAL returns the same query results, `tx_count` and fact log as a read-write open; that queries, a cursor, a fact log, `verify`, `prepare`, rules and a UDF leave the `.graph` and WAL bytes and the `.graph` mtime unchanged, and no WAL is created; that `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` are `API-014` without taking a `tx_count`, also on a read-only in-memory database and through the builder path; that a missing file is `STG-042` and nothing is created; that an empty file is an empty database and stays 0 bytes; that read-only handles share a file while a read-write one is refused with `STG-025`, both ways; that a file without write permission opens read-only; and that small and whole-file page caches work. File-backend unit tests check that a shared-mode backend refuses page writes and syncs with `INT-056`, that the in-process registry counts readers and admits readers or one writer (also without kernel locks), and that a reader in another process shares the file with a reader here and refuses a writer here with `STG-026`. Storage unit tests check that a read-only open of a v7 file at every point of an interrupted migration (clean and torn writes) sees every fact and writes nothing, and that an empty backend stays empty. `tests/golden_corpus_test.rs` opens a copy of every golden file read-only first and checks its `tx_count` and queries, and that neither the file nor its WAL changes. Log-writer tests (#431, `tests/log_writer_test.rs`) copy a source's fact log (assertions, retractions, an explicit valid-time window, long strings, refs, keywords, checkpointed and pending facts) through `LogWriter` and check that the new file has no `.partial` or WAL, verifies clean, and gives the source's `tx_count`, fact log and query results (current, `:valid-at`, `:any-valid-time`, every `:as-of`, and entity-, attribute- and value-bound lookups) read-write, read-only and after reopen (#370); that it takes new writes at the source's next `tx_count`; that 90,000 facts over two batch commits verify clean and count correctly at `:as-of`; that a purged middle transaction is a hole whose `:as-of` equals the one before, and trailing holes keep the counter through `advance_tx_count`; that an empty log is an empty database with the counter; that `tx_count` 0, a lower or closed transaction and a backwards advance are `API-015`, a second `tx_id` `API-016`, a second window `API-011` and oversize values or idents `WAL-003`, each leaving the writer usable and the output exact, with repeats written once; that an existing target or WAL is `STG-043`, a second writer `STG-025` and `read_only` `API-014`; that a stale `.partial` is reused; that dropping an unfinished writer leaves nothing; that `finish` refuses a target created during the build and leaves it untouched; and that `LogWriter` is `Send`. Unit tests check that a batch commits only when a transaction opens past the batch size, never splitting one, and that `FileBackend::truncate` empties a file. ### Covered Areas diff --git a/docs/superpowers/specs/2026-10-07-log-writer-design.md b/docs/superpowers/specs/2026-10-07-log-writer-design.md new file mode 100644 index 0000000000..6c4be258c5 --- /dev/null +++ b/docs/superpowers/specs/2026-10-07-log-writer-design.md @@ -0,0 +1,138 @@ +# Raw log writer (#431, v3.0.0) — design + +**Issue:** #431 (build a fresh graph from explicit `(tx, asserted, valid-time)` records). +**Branch:** `v3`. **Related:** #430 (fact log, merged as `FactLog`/`FactRecord`), #429 +(read-only open), temporal_reasoning#378 (the migration tool). + +## Goal + +The load step of an offline migration: write a stream of `FactRecord`s (from +`Minigraf::fact_log` on a read-only source, after a transform) into a new graph file that +keeps each record's `tx_count`, `tx_id`, `asserted` flag and valid-time window. The +result must be an ordinary file, as if the transactions had been written normally and +checkpointed. Purging data is then "do not append it". + +`transact`/`retract` cannot do this: they take the next `tx_count`, stamp `tx_id` and +`valid_from` with the current time, and write a WAL entry per transaction. + +## API + +```rust +pub struct LogWriter { /* private */ } // not on wasm32 + +impl LogWriter { + pub fn create(path: impl AsRef, opts: OpenOptions) -> Result; + pub fn append(&mut self, rec: &FactRecord) -> Result<(), MinigrafError>; + pub fn advance_tx_count(&mut self, tx_count: u64) -> Result<(), MinigrafError>; + pub fn tx_count(&self) -> u64; + pub fn finish(self) -> Result<(), MinigrafError>; +} +``` + +```rust +let src = Minigraf::open_with_options("old.graph", OpenOptions::new().read_only(true))?; +let mut out = LogWriter::create("new.graph", OpenOptions::new())?; +for rec in src.fact_log(&FactFilter::new())? { + let rec = rec?; + if !rec.attribute.starts_with(":secret/") { + out.append(&rec)?; + } +} +out.advance_tx_count(src.current_tx_count())?; +out.finish()?; +``` + +## Decisions + +1. **Built beside the target, renamed into place.** `create` refuses with the new + **`STG-043`** if `path` or `.wal` exists: a migration never overwrites a + database, and a stale WAL would be replayed onto the new file. The file is built at + `.partial`, under the usual exclusive lock. A `.partial` left by a crashed or + abandoned build is truncated and reused (a live builder holds its lock, so a second + `create` for the same path fails with STG-025/STG-026). `finish` commits the last + batch, closes the file, checks again that `path` does not exist (`STG-043`), renames + `.partial` to `path` and syncs the directory. Dropping a writer without `finish` + deletes the `.partial`. A crash at any point therefore never leaves a file at `path` + that holds only part of the log; the single-file rule holds for every database that + can be opened. The check before the rename does not lock the directory: the caller + must not create `path` while the build runs. +2. **No WAL; batched checkpoints.** Records are loaded into the in-memory pending list + and committed with the normal copy-on-write checkpoint (`PersistentFactStorage::save`) + whenever at least 65,536 facts are pending and a new transaction starts, and once + more by `finish`. Memory is O(batch + largest transaction), as for normal writes, which + also hold a whole transaction in memory. A transaction is never split across + checkpoints. The page cache uses `opts.page_cache_size`. +3. **Transaction order.** The writer keeps a counter (`tx_count()`, starting at 0) and the + open transaction. A record must have `tx_count > counter`, which opens a new + transaction, or equal the open transaction's `tx_count`, with the same `tx_id`. + Otherwise the new **`API-015`** (out of order, including `tx_count` 0 and a record for + a transaction that is already closed) or **`API-016`** (one transaction, two `tx_id`s) + is returned, and nothing changes. Gaps are allowed and are simply absent: `:as-of N` + on a hole compares `tx_count <= N` and so shows the preceding state. + `tx_id` may go backwards between transactions; the source's clock is kept as it was. +4. **Trailing holes.** `advance_tx_count(n)` closes the open transaction and raises the + counter to `n` without facts (an empty or purged transaction). `n` below the counter + is `API-015`. `finish` writes the counter as `last_checkpointed_tx_count`, so the new + file's first write after a purge of the last transactions still takes the source's + next number, and `current_tx_count()` matches the source. +5. **Checks: structure, not schema.** Per record: + - order (`API-015`, `API-016`, above); + - size: attribute and keyword idents and string values within the format limits + (`WAL-003`, as for a normal write); + - one valid-time window per `(entity, attribute, value)` assertion in one transaction + (`API-011`, #435). Reads assume it, and a normal write refuses it the same way. + An exact repeat of a record already in the open transaction is dropped, as a normal + write drops a repeated fact. Nothing else is checked: `valid_from > valid_to`, + retractions without an assertion, and attribute semantics are the caller's business; + a migration checks its result with `verify()` and its own audit. +6. **`finish` output.** After the final checkpoint the file has every record in + EAVT/AEVT/AVET/VAET and the dictionary, a meta at the last generation, and no WAL. It + opens read-write or read-only, `verify()` is clean, and queries, `:as-of` and + `fact_log()` give the same answers as a database built by `transact` with the same + records. The tests check exactly this against #370's failure mode (indexes missing + facts after reopen). +7. **No `copy_pages`.** The issue offered a page-range fast path "if the format permits". + Format v8 shares one dictionary across all trees and uses copy-on-write pages, so a + range of pages from another file is not self-contained (ids, value references and + free-list state all differ). The fact-log design (#430, decision 5) reached the same + conclusion. Every record goes through `append`. +8. **Errors leave the writer usable.** A rejected record changes nothing. An I/O error + from a checkpoint inside `append` is returned and the writer should be dropped (which + deletes the `.partial`). `opts.read_only` is `API-014`. +9. **Bindings** (`LogWriter` in the FFI, so the tool can live in the temporal_reasoning + Python package) follow in the binding repositories, as with #429. + +## Implementation + +- `src/log_writer.rs`: `LogWriter` holds a `PersistentFactStorage`, the + target and partial paths, the counter, the open transaction `(tx_count, tx_id)`, and + the open transaction's windows `HashMap<(EntityId, String, Vec), (i64, i64)>`. + `append` converts the record to a `Fact`, runs the checks, `load_fact`s it and marks + the storage dirty. Before each `save`, the storage's tx counter is set to the + writer's counter. Re-exported from `lib.rs` (not on wasm32). +- `FileBackend::truncate()` resets a reused `.partial` to zero pages. +- `PersistentFactStorage::discard()` clears the dirty flag so an abandoned writer's drop + does not commit. +- `wal::check_fact_size` becomes `pub(crate)`. +- `STG-043`, `API-015`, `API-016` in `src/error.rs` and `docs/ERROR_REFERENCE.md`. + +## Tests (`tests/log_writer_test.rs`) + +- Round trip: records from a source built with transacts, retracts, explicit valid-time + windows, long strings, refs and keywords, written through `LogWriter`. The new file's + `fact_log` equals the source's; queries (current, `:as-of N` for every N, + `:valid-at`) match; `current_tx_count` matches; `verify()` is clean; a read-only open + works; and after a reopen, indexed lookups by entity, attribute and value return every + fact (#370). +- Many batches (more than one checkpoint) give the same result. +- Holes: purge one middle transaction; `:as-of` the hole equals `:as-of` the one before. + Trailing holes with `advance_tx_count`; the next `transact` takes the following number. +- An empty log gives an empty database with the advanced counter. +- Rejections: `tx_count` 0, decreasing, a closed transaction (`API-015`), two `tx_id`s + (`API-016`), two windows (`API-011`), oversize (`WAL-003`), `advance_tx_count` below the + counter. Each leaves the writer usable and the output unchanged. +- Duplicate records in one transaction are written once. +- `create` refuses an existing target or WAL (`STG-043`), a second writer for the same + path (`STG-025`), and `read_only` (`API-014`); a stale `.partial` is reused. +- Dropping without `finish` leaves no `.partial` and no target. `finish` refuses a target + created during the build and leaves it untouched. diff --git a/src/error.rs b/src/error.rs index 93d569f752..b09874ce3a 100644 --- a/src/error.rs +++ b/src/error.rs @@ -183,6 +183,7 @@ pub(crate) enum ErrorCode { Stg040, Stg041, Stg042, + Stg043, Wal001, Wal002, Wal003, @@ -203,6 +204,8 @@ pub(crate) enum ErrorCode { Api012, Api013, Api014, + Api015, + Api016, Int001, Int002, Int003, @@ -1052,6 +1055,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[ "database file not found ({}); a read-only open does not create one", ErrorCategory::Storage, ), + ( + ErrorCode::Stg043, + "STG-043", + "{} already exists; a log writer only creates a new database", + ErrorCategory::Storage, + ), ( ErrorCode::Wal001, "WAL-001", @@ -1172,6 +1181,18 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[ "database is open read-only; {} is not allowed", ErrorCategory::Api, ), + ( + ErrorCode::Api015, + "API-015", + "tx_count {} is out of order: the log writer is at {}, and a record must open a later transaction or join the open one", + ErrorCategory::Api, + ), + ( + ErrorCode::Api016, + "API-016", + "transaction {} has tx_id {}; a record with tx_id {} cannot join it", + ErrorCategory::Api, + ), ( ErrorCode::Int001, "INT-001", @@ -1911,6 +1932,7 @@ mod tests { | ErrorCode::Stg040 | ErrorCode::Stg041 | ErrorCode::Stg042 + | ErrorCode::Stg043 | ErrorCode::Wal001 | ErrorCode::Wal002 | ErrorCode::Wal003 @@ -1931,6 +1953,8 @@ mod tests { | ErrorCode::Api012 | ErrorCode::Api013 | ErrorCode::Api014 + | ErrorCode::Api015 + | ErrorCode::Api016 | ErrorCode::Int001 | ErrorCode::Int002 | ErrorCode::Int003 @@ -2122,6 +2146,7 @@ mod tests { ErrorCode::Stg040, ErrorCode::Stg041, ErrorCode::Stg042, + ErrorCode::Stg043, ErrorCode::Wal001, ErrorCode::Wal002, ErrorCode::Wal003, @@ -2142,6 +2167,8 @@ mod tests { ErrorCode::Api012, ErrorCode::Api013, ErrorCode::Api014, + ErrorCode::Api015, + ErrorCode::Api016, ErrorCode::Int001, ErrorCode::Int002, ErrorCode::Int003, diff --git a/src/lib.rs b/src/lib.rs index 39f16d0723..3bf378e7c6 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -81,6 +81,9 @@ pub mod error; /// Every fact version, streamed: [`FactLog`], [`FactFilter`], [`FactRecord`]. pub mod fact_log; pub(crate) mod graph; +/// Build a new database from explicit fact records: [`LogWriter`]. +#[cfg(not(target_arch = "wasm32"))] +pub mod log_writer; pub(crate) mod query; /// Interactive REPL for exploring a [`Minigraf`] database from the command line. pub mod repl; @@ -104,6 +107,8 @@ pub use graph::types::{EntityId, Value}; // Query result types pub use cursor::{Batch, Cursor}; pub use fact_log::{FactFilter, FactLog, FactOrder, FactRecord}; +#[cfg(not(target_arch = "wasm32"))] +pub use log_writer::LogWriter; pub use query::datalog::executor::QueryResult; // Bi-temporal query types diff --git a/src/log_writer.rs b/src/log_writer.rs new file mode 100644 index 0000000000..c00bcad3b2 --- /dev/null +++ b/src/log_writer.rs @@ -0,0 +1,349 @@ +//! Build a new database from explicit fact records (#431). +//! +//! [`LogWriter`] is the load step of an offline migration: it writes +//! [`FactRecord`]s, such as those read by [`crate::Minigraf::fact_log`] from a +//! read-only source, into a new `.graph` file, keeping each record's +//! `tx_count`, `tx_id`, `asserted` flag and valid-time window. The finished +//! file is an ordinary database, as if the same transactions had been written +//! normally and checkpointed. +//! +//! The file is built beside the target, at `.partial`, and renamed into +//! place by [`LogWriter::finish`]. A crash or an abandoned build never leaves a +//! database at `path` that holds only part of the log. + +use crate::db::OpenOptions; +use crate::error::{ErrorCode, MinigrafError, bail_coded, err_coded}; +use crate::fact_log::FactRecord; +use crate::graph::types::{EntityId, Fact}; +use crate::storage::backend::file::{FileBackend, LockMode}; +use crate::storage::dir_sync::sync_parent_dir; +use crate::storage::index::encode_value; +use crate::storage::persistent_facts::PersistentFactStorage; +use anyhow::Result; +use std::collections::HashMap; +use std::ffi::OsString; +use std::path::{Path, PathBuf}; + +/// Pending facts after which the next transaction boundary commits a batch. +const BATCH_FACTS: usize = 1 << 16; + +/// Writes explicit fact records into a new database file. +/// +/// Records must come in transaction order: each one either opens a new +/// transaction, with a `tx_count` above every earlier one, or joins the open +/// transaction, with its `tx_count` and `tx_id`. Gaps between transactions are +/// allowed and stay gaps, so `:as-of N` on a missing transaction shows the +/// state before it. Nothing beyond the file's structure is checked: the caller +/// owns the meaning of the records. +/// +/// ```no_run +/// use minigraf::{FactFilter, LogWriter, Minigraf, OpenOptions}; +/// +/// # fn main() -> Result<(), minigraf::MinigrafError> { +/// let src = Minigraf::open_with_options("old.graph", OpenOptions::new().read_only(true))?; +/// let mut out = LogWriter::create("new.graph", OpenOptions::new())?; +/// for rec in src.fact_log(&FactFilter::new())? { +/// let rec = rec?; +/// if !rec.attribute.starts_with(":secret/") { +/// out.append(&rec)?; +/// } +/// } +/// out.advance_tx_count(src.current_tx_count())?; +/// out.finish()?; +/// # Ok(()) +/// # } +/// ``` +pub struct LogWriter { + /// `None` only after `finish` or `drop` took it. + pfs: Option>, + path: PathBuf, + partial: PathBuf, + /// The highest `tx_count` appended or advanced to. + tx_count: u64, + /// The open transaction's `(tx_count, tx_id)`, if a record can join it. + open_tx: Option<(u64, u64)>, + /// The open transaction's valid-time window per asserted `(e, a, v)`. + windows: HashMap<(EntityId, String, Vec), (i64, i64)>, + /// Facts loaded since the last batch commit. + pending: usize, +} + +impl LogWriter { + /// Start building a new database at `path`. + /// + /// `opts.page_cache_size` and `opts.allow_unlocked` apply; the WAL options + /// do not, since the writer has no WAL. + /// + /// # Errors + /// + /// - `STG-043` if `path` or its WAL (`.wal`) already exists. + /// - `STG-025`/`STG-026` if another writer is building the same `path`. + /// - `API-014` if `opts.read_only` is set. + /// - An I/O error creating `.partial`. + pub fn create(path: impl AsRef, opts: OpenOptions) -> Result { + Self::create_inner(path.as_ref(), opts).map_err(MinigrafError::from) + } + + fn create_inner(path: &Path, opts: OpenOptions) -> Result { + if opts.read_only { + bail_coded!(ErrorCode::Api014, "LogWriter::create"); + } + let wal = with_suffix(path, ".wal"); + for p in [path, wal.as_path()] { + if p.exists() { + bail_coded!(ErrorCode::Stg043, p.display()); + } + } + let partial = with_suffix(path, ".partial"); + let mut backend = + FileBackend::open_with(&partial, opts.allow_unlocked, LockMode::Exclusive)?; + // A `.partial` from an earlier build that never finished: start over. + backend.truncate()?; + let pfs = PersistentFactStorage::open(backend, opts.page_cache_size, None)?; + Ok(LogWriter { + pfs: Some(pfs), + path: path.to_path_buf(), + partial, + tx_count: 0, + open_tx: None, + windows: HashMap::new(), + pending: 0, + }) + } + + /// The highest `tx_count` appended or passed to + /// [`advance_tx_count`](Self::advance_tx_count); 0 before either. + pub fn tx_count(&self) -> u64 { + self.tx_count + } + + /// Append one record. + /// + /// A record with `tx_count` above [`tx_count`](Self::tx_count) opens a + /// new transaction. A record with the open transaction's `tx_count` joins + /// it and must carry its `tx_id`. An exact repeat of a record already in + /// the open transaction is dropped, as a normal write drops a repeated + /// fact. When enough facts are pending, opening a transaction first + /// commits the earlier ones to the file. + /// + /// # Errors + /// + /// A rejected record changes nothing, and the writer stays usable: + /// - `API-015` if `tx_count` is 0, below the writer's, or names a + /// transaction that is already closed. + /// - `API-016` if it joins the open transaction with a different `tx_id`. + /// - `API-011` if it asserts an `(entity, attribute, value)` the open + /// transaction already asserts with another valid-time window. + /// - `WAL-003` if the attribute, a keyword or a string value is longer + /// than the format stores. + /// + /// An I/O error from committing a batch is also returned; drop the writer + /// after one. + pub fn append(&mut self, rec: &FactRecord) -> Result<(), MinigrafError> { + self.append_inner(rec).map_err(MinigrafError::from) + } + + fn append_inner(&mut self, rec: &FactRecord) -> Result<()> { + let joins = match self.open_tx { + Some((tx, tx_id)) if tx == rec.tx_count => { + if tx_id != rec.tx_id { + bail_coded!(ErrorCode::Api016, tx, tx_id, rec.tx_id); + } + true + } + _ if rec.tx_count > self.tx_count => false, + _ => bail_coded!(ErrorCode::Api015, rec.tx_count, self.tx_count), + }; + let fact = Fact { + entity: rec.entity, + attribute: rec.attribute.clone(), + value: rec.value.clone(), + tx_id: rec.tx_id, + tx_count: rec.tx_count, + valid_from: rec.valid_from, + valid_to: rec.valid_to, + asserted: rec.asserted, + }; + crate::wal::check_fact_size(&fact)?; + let key = ( + fact.entity, + fact.attribute.clone(), + encode_value(&fact.value), + ); + let window = (fact.valid_from, fact.valid_to); + if fact.asserted && joins && self.windows.get(&key).is_some_and(|w| *w != window) { + bail_coded!(ErrorCode::Api011, fact.attribute); + } + + if !joins { + self.close_tx(); + if self.pending >= BATCH_FACTS { + self.commit_batch()?; + } + self.tx_count = rec.tx_count; + self.open_tx = Some((rec.tx_count, rec.tx_id)); + } + if fact.asserted { + self.windows.insert(key, window); + } + let pfs = self.pfs_mut()?; + if pfs.storage().load_fact(fact)? { + pfs.mark_dirty(); + self.pending = self.pending.saturating_add(1); + } + Ok(()) + } + + /// Close the open transaction and raise the writer's `tx_count` to + /// `tx_count`, with no facts: the transactions in between, or a trailing + /// run of purged ones, stay empty. The finished file's + /// [`current_tx_count`](crate::Minigraf::current_tx_count) is the writer's, + /// so after a purge of the last transactions the next write still takes the + /// source's next number. + /// + /// # Errors + /// + /// `API-015` if `tx_count` is below the writer's. Nothing changes. + pub fn advance_tx_count(&mut self, tx_count: u64) -> Result<(), MinigrafError> { + if tx_count < self.tx_count { + return Err(MinigrafError::from(err_coded!( + ErrorCode::Api015, + tx_count, + self.tx_count + ))); + } + self.close_tx(); + self.tx_count = tx_count; + Ok(()) + } + + /// Commit every pending record, close the file and rename it to `path`. + /// + /// The result is an ordinary database with no WAL: its indexes hold every + /// record and its transaction counter is [`tx_count`](Self::tx_count). + /// + /// # Errors + /// + /// - `STG-043` if a file appeared at `path` during the build. It is left + /// untouched, and the build is discarded. + /// - An I/O error from the commit, the rename or the directory sync. + pub fn finish(mut self) -> Result<(), MinigrafError> { + self.finish_inner().map_err(MinigrafError::from) + } + + fn finish_inner(&mut self) -> Result<()> { + self.close_tx(); + let mut pfs = self + .pfs + .take() + .ok_or_else(|| err_coded!(ErrorCode::Int049, "log writer already finished"))?; + pfs.storage().restore_tx_counter_from(self.tx_count); + // Also with no records: the meta must carry the counter. + pfs.force_dirty(); + let saved = pfs.save(); + // Release the file (and its lock) before the rename. + pfs.discard(); + drop(pfs); + saved?; + if self.path.exists() { + bail_coded!(ErrorCode::Stg043, self.path.display()); + } + std::fs::rename(&self.partial, &self.path)?; + sync_parent_dir(&self.path)?; + Ok(()) + } + + fn close_tx(&mut self) { + self.open_tx = None; + self.windows.clear(); + } + + /// Commit the pending records as the next generation of the file. + fn commit_batch(&mut self) -> Result<()> { + let tx_count = self.tx_count; + let pfs = self.pfs_mut()?; + pfs.storage().restore_tx_counter_from(tx_count); + pfs.save()?; + self.pending = 0; + Ok(()) + } + + fn pfs_mut(&mut self) -> Result<&mut PersistentFactStorage> { + self.pfs + .as_mut() + .ok_or_else(|| err_coded!(ErrorCode::Int049, "log writer already finished")) + } +} + +impl Drop for LogWriter { + /// An unfinished build (or one whose `finish` failed) is deleted. + fn drop(&mut self) { + if let Some(mut pfs) = self.pfs.take() { + pfs.discard(); + drop(pfs); + } + if self.partial.exists() { + let _ = std::fs::remove_file(&self.partial); + } + } +} + +/// `path` with `suffix` appended to its file name. +fn with_suffix(path: &Path, suffix: &str) -> PathBuf { + let mut name: OsString = path.file_name().map(OsString::from).unwrap_or_default(); + name.push(suffix); + path.with_file_name(name) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::graph::types::Value; + + fn rec(tx_count: u64, i: u64) -> FactRecord { + FactRecord { + entity: uuid::Uuid::from_u128(u128::from(i) + 1), + attribute: ":n/v".to_string(), + value: Value::Integer(i64::try_from(i).unwrap()), + tx_count, + tx_id: 1_000 + tx_count, + valid_from: 0, + valid_to: i64::MAX, + asserted: true, + } + } + + fn generation(w: &LogWriter) -> u64 { + w.pfs.as_ref().unwrap().generation() + } + + #[test] + fn a_batch_commits_only_when_a_transaction_opens_past_the_limit() { + let dir = tempfile::tempdir().unwrap(); + let mut w = LogWriter::create(dir.path().join("b.graph"), OpenOptions::new()).unwrap(); + let n = u64::try_from(BATCH_FACTS).unwrap(); + for i in 0..n + 10 { + w.append(&rec(1, i)).unwrap(); + } + // One transaction is never split, however large. + assert_eq!(generation(&w), 1); + assert_eq!(w.pending, BATCH_FACTS + 10); + w.append(&rec(2, 0)).unwrap(); + assert_eq!(generation(&w), 2); + assert_eq!(w.pending, 1); + w.finish().unwrap(); + } + + #[test] + fn truncate_empties_a_file_backend() { + use crate::storage::{PAGE_SIZE, StorageBackend}; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("t.graph"); + let mut b = FileBackend::open_with(&path, false, LockMode::Exclusive).unwrap(); + b.write_page(2, &vec![1u8; PAGE_SIZE]).unwrap(); + assert_eq!(b.page_count().unwrap(), 3); + b.truncate().unwrap(); + assert_eq!(b.page_count().unwrap(), 0); + assert_eq!(std::fs::metadata(&path).unwrap().len(), 0); + } +} diff --git a/src/storage/backend/file.rs b/src/storage/backend/file.rs index 7b68b012b2..e921e89134 100644 --- a/src/storage/backend/file.rs +++ b/src/storage/backend/file.rs @@ -360,6 +360,18 @@ impl FileBackend { pub fn path(&self) -> &Path { &self.path } + + /// Cut the file to zero pages and sync, keeping the lock. Used to reuse + /// a log writer's `.partial` file left by an earlier build (#431). + pub(crate) fn truncate(&mut self) -> Result<()> { + if self.read_only { + bail_coded!(ErrorCode::Int056, "truncate"); + } + self.file.set_len(0)?; + self.file.sync_all()?; + self.page_count = 0; + Ok(()) + } } impl StorageBackend for FileBackend { diff --git a/src/storage/persistent_facts.rs b/src/storage/persistent_facts.rs index c818fc104f..9e9f3fe47d 100644 --- a/src/storage/persistent_facts.rs +++ b/src/storage/persistent_facts.rs @@ -662,6 +662,13 @@ impl PersistentFactStorage { self.dirty } + /// Forget unsaved changes, so dropping this storage commits nothing. Used + /// by an abandoned log writer, whose file is deleted next (#431). + #[cfg(not(target_arch = "wasm32"))] + pub(crate) fn discard(&mut self) { + self.dirty = false; + } + /// Capacity (in pages) of the internal LRU page cache passed to `new()`. /// /// Exposed only for the browser WASM layer's tests, which verify that diff --git a/src/wal.rs b/src/wal.rs index 94b0dbbc31..57f97ae96b 100644 --- a/src/wal.rs +++ b/src/wal.rs @@ -108,7 +108,7 @@ fn check_wal_header_length(file: &mut File) -> Result { /// Reject a fact a checkpoint could not store (WAL-003): a string value longer /// than one value page's payload, or an attribute or keyword longer than an /// ident may be (spec §6.3). -fn check_fact_size(fact: &Fact) -> Result<()> { +pub(crate) fn check_fact_size(fact: &Fact) -> Result<()> { if fact.attribute.len() > MAX_IDENT_BYTES { bail_coded!(ErrorCode::Wal003, fact.attribute.len(), MAX_IDENT_BYTES); } diff --git a/tests/log_writer_test.rs b/tests/log_writer_test.rs new file mode 100644 index 0000000000..8af2fc2780 --- /dev/null +++ b/tests/log_writer_test.rs @@ -0,0 +1,504 @@ +//! `LogWriter` builds a new database from explicit fact records (#431). These +//! tests copy a source's fact log through it and check that the result answers +//! every query, `:as-of` and fact-log read as the source does, survives a +//! reopen with intact indexes (#370), and that holes, rejections, and the +//! `.partial` build file behave as documented. + +use minigraf::{FactFilter, FactRecord, LogWriter, Minigraf, OpenOptions, QueryResult, Value}; +use std::path::{Path, PathBuf}; +use uuid::Uuid; + +const NAMES: [&str; 4] = ["e1", "e2", "e3", "e4"]; +const LONG: &str = "a long string value that is stored in a value page, not in the key: \ + 0123456789abcdefghijklmnopqrstuvwxyz"; + +fn uuid(i: usize) -> Uuid { + Uuid::from_u128(0x2000 + i as u128) +} + +fn e(i: usize) -> String { + format!("#uuid \"{}\"", uuid(i)) +} + +fn name_of(u: &Uuid) -> &'static str { + (0..NAMES.len()) + .find(|&i| uuid(i) == *u) + .map_or("?", |i| NAMES[i]) +} + +fn show(v: &Value) -> String { + match v { + Value::Ref(u) => format!("ref {}", name_of(u)), + other => format!("{other:?}"), + } +} + +/// A record without UUIDs, for comparisons. +fn row(r: &FactRecord) -> String { + format!( + "{} {} {} {} {} {} {} {}", + r.tx_count, + r.tx_id, + name_of(&r.entity), + r.attribute, + show(&r.value), + r.valid_from, + r.valid_to, + r.asserted + ) +} + +fn log_rows(db: &Minigraf) -> Vec { + let mut rows: Vec = db + .fact_log(&FactFilter::new()) + .unwrap() + .map(|r| row(&r.unwrap())) + .collect(); + rows.sort(); + rows +} + +fn records(db: &Minigraf) -> Vec { + db.fact_log(&FactFilter::new()) + .unwrap() + .collect::>() + .unwrap() +} + +/// Query rows without UUIDs, sorted. +fn rows(db: &Minigraf, query: &str) -> Vec { + match db.execute(query).unwrap() { + QueryResult::QueryResults { results, .. } => { + let mut out: Vec = results + .iter() + .map(|r| r.iter().map(show).collect::>().join(" ")) + .collect(); + out.sort(); + out + } + _ => panic!("expected query results"), + } +} + +fn all_at(as_of: Option, valid: &str) -> String { + let as_of = as_of.map(|n| format!(":as-of {n}")).unwrap_or_default(); + format!("(query [:find ?e ?a ?v {as_of} :valid-at {valid} :where [?e ?a ?v]])") +} + +/// A source with assertions, retractions, explicit windows, long strings, +/// refs, keywords and a multi-fact transaction. +fn source(path: &Path) -> Minigraf { + let db = Minigraf::open(path).unwrap(); + let writes = [ + format!( + r#"(transact [[{} :person/name "Alice"] [{} :person/age 30] + [{} :person/name "Bob"] [{} :person/friend {}]])"#, + e(0), + e(0), + e(1), + e(1), + e(0) + ), + format!( + "(transact [[{} :person/age 31] [{} :person/role :role/admin]])", + e(0), + e(1) + ), + format!("(retract [[{} :person/age 30]])", e(0)), + format!("(transact [[{} :doc/body \"{LONG}\"]])", e(2)), + format!( + r#"(transact {{:valid-from "2020-01-01" :valid-to "2022-01-01"}} [[{} :person/salary 100]])"#, + e(1) + ), + format!("(retract [[{} :person/role :role/admin]])", e(1)), + format!( + "(transact [[{} :person/score 2.5] [{} :flag/on true]])", + e(3), + e(3) + ), + ]; + for (i, w) in writes.iter().enumerate() { + db.execute(w).unwrap(); + if i == 3 { + db.checkpoint().unwrap(); + } + } + db +} + +fn copy(src: &Minigraf, out: &Path, keep: impl Fn(&FactRecord) -> bool) { + let mut w = LogWriter::create(out, OpenOptions::new()).unwrap(); + for rec in records(src) { + if keep(&rec) { + w.append(&rec).unwrap(); + } + } + w.advance_tx_count(src.current_tx_count()).unwrap(); + w.finish().unwrap(); +} + +fn same_answers(a: &Minigraf, b: &Minigraf) { + assert_eq!(a.current_tx_count(), b.current_tx_count(), "tx count"); + assert_eq!(log_rows(a), log_rows(b), "fact log"); + let mut queries = vec![ + all_at(None, ":any-valid-time"), + all_at(None, "\"2021-06-01\""), + "(query [:find ?e ?a ?v :where [?e ?a ?v]])".to_string(), + format!("(query [:find ?a ?v :where [{} ?a ?v]])", e(0)), + "(query [:find ?e :where [?e :person/name \"Bob\"]])".to_string(), + "(query [:find ?e ?v :where [?e :person/friend ?v]])".to_string(), + format!("(query [:find ?e :where [?e :person/friend {}]])", e(0)), + ]; + for n in 0..=a.current_tx_count() { + queries.push(all_at(Some(n), ":any-valid-time")); + } + for q in &queries { + assert_eq!(rows(a, q), rows(b, q), "{q}"); + } +} + +struct Dir { + _tmp: tempfile::TempDir, + root: PathBuf, +} + +impl Dir { + fn new() -> Self { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path().to_path_buf(); + Dir { _tmp: tmp, root } + } + fn path(&self, name: &str) -> PathBuf { + self.root.join(name) + } +} + +fn partial(p: &Path) -> PathBuf { + p.with_file_name(format!( + "{}.partial", + p.file_name().unwrap().to_str().unwrap() + )) +} + +fn wal(p: &Path) -> PathBuf { + p.with_file_name(format!("{}.wal", p.file_name().unwrap().to_str().unwrap())) +} + +#[test] +fn copy_answers_like_the_source_and_survives_reopen() { + let dir = Dir::new(); + let src = source(&dir.path("src.graph")); + let out = dir.path("out.graph"); + copy(&src, &out, |_| true); + + assert!(out.exists()); + assert!(!partial(&out).exists(), "no .partial left"); + assert!(!wal(&out).exists(), "no WAL"); + + let db = Minigraf::open(&out).unwrap(); + assert!(db.verify().unwrap().is_ok(), "verify is clean"); + same_answers(&src, &db); + drop(db); + + // Read-only, then read-write again: indexed lookups after reopen (#370). + let ro = Minigraf::open_with_options(&out, OpenOptions::new().read_only(true)).unwrap(); + same_answers(&src, &ro); + drop(ro); + let db = Minigraf::open(&out).unwrap(); + same_answers(&src, &db); +} + +#[test] +fn copy_equals_a_database_built_by_transact() { + // Records with the same timestamps in a source and its copy give equal + // fact logs; the copy also takes new writes after the source's counter. + let dir = Dir::new(); + let src = source(&dir.path("src.graph")); + let out = dir.path("out.graph"); + copy(&src, &out, |_| true); + let db = Minigraf::open(&out).unwrap(); + src.execute(&format!("(transact [[{} :person/name \"Cy\"]])", e(2))) + .unwrap(); + db.execute(&format!("(transact [[{} :person/name \"Cy\"]])", e(2))) + .unwrap(); + assert_eq!(db.current_tx_count(), src.current_tx_count()); + let q = "(query [:find ?e ?v :where [?e :person/name ?v]])"; + assert_eq!(rows(&db, q), rows(&src, q)); + db.checkpoint().unwrap(); + assert!( + db.verify().unwrap().is_ok(), + "verify is clean after new writes" + ); +} + +#[test] +fn many_batches_give_the_same_file_contents() { + // Two batch commits: the first when transaction 8 opens with 70,000 facts + // pending (over the 65,536 batch), the second in `finish`. + let dir = Dir::new(); + let out = dir.path("big.graph"); + let mut w = LogWriter::create(&out, OpenOptions::new()).unwrap(); + let per_tx = 10_000u64; + for tx in 1..=9u64 { + for i in 0..per_tx { + let rec = FactRecord { + entity: Uuid::from_u128(u128::from(i % 1000) + 1), + attribute: format!(":n/a{}", i % 7), + value: Value::Integer((tx * per_tx + i) as i64), + tx_count: tx * 2, + tx_id: 1_700_000_000_000 + tx, + valid_from: 1_700_000_000_000 + tx as i64, + valid_to: i64::MAX, + asserted: true, + }; + w.append(&rec).unwrap(); + } + } + assert_eq!(w.tx_count(), 18); + w.finish().unwrap(); + + let db = Minigraf::open(&out).unwrap(); + assert!(db.verify().unwrap().is_ok(), "verify is clean"); + assert_eq!(db.current_tx_count(), 18); + let recs = records(&db); + assert_eq!(recs.len(), 90_000); + assert!(recs.windows(2).all(|p| p[0].tx_count <= p[1].tx_count)); + let count = |q: &str| match db.execute(q).unwrap() { + QueryResult::QueryResults { results, .. } => match results[0][0] { + Value::Integer(n) => n, + _ => panic!("count"), + }, + _ => panic!("results"), + }; + let a3_per_tx = (0..per_tx).filter(|i| i % 7 == 3).count() as i64; + assert_eq!( + count("(query [:find (count ?v) :where [?e :n/a3 ?v]])"), + 9 * a3_per_tx + ); + assert_eq!( + count("(query [:find (count ?v) :as-of 7 :where [?e :n/a3 ?v]])"), + // Transactions 2, 4 and 6. + 3 * a3_per_tx + ); +} + +#[test] +fn a_purged_transaction_is_a_hole_that_shows_the_state_before_it() { + let dir = Dir::new(); + let src = source(&dir.path("src.graph")); + let out = dir.path("out.graph"); + // Transaction 2 asserted age 31 and the admin role. + copy(&src, &out, |r| r.tx_count != 2); + let db = Minigraf::open(&out).unwrap(); + assert_eq!(db.current_tx_count(), src.current_tx_count()); + assert_eq!( + rows(&db, &all_at(Some(2), ":any-valid-time")), + rows(&src, &all_at(Some(1), ":any-valid-time")) + ); + assert!( + log_rows(&db).iter().all(|r| !r.starts_with("2 ")), + "nothing from transaction 2" + ); + // Later transactions are unchanged. + assert_eq!( + rows( + &db, + &format!("(query [:find ?v :as-of 4 :where [{} :doc/body ?v]])", e(2)) + ), + vec![format!("{:?}", Value::String(LONG.to_string()))] + ); +} + +#[test] +fn trailing_holes_keep_the_counter() { + let dir = Dir::new(); + let src = source(&dir.path("src.graph")); + let last = src.current_tx_count(); + let out = dir.path("out.graph"); + copy(&src, &out, |r| r.tx_count < last - 1); + let db = Minigraf::open(&out).unwrap(); + assert_eq!(db.current_tx_count(), last); + db.execute(&format!("(transact [[{} :person/name \"Dee\"]])", e(3))) + .unwrap(); + assert_eq!(db.current_tx_count(), last + 1); +} + +#[test] +fn an_empty_log_is_an_empty_database_with_the_counter() { + let dir = Dir::new(); + let out = dir.path("empty.graph"); + let mut w = LogWriter::create(&out, OpenOptions::new()).unwrap(); + w.advance_tx_count(5).unwrap(); + w.finish().unwrap(); + let db = Minigraf::open(&out).unwrap(); + assert_eq!(db.current_tx_count(), 5); + assert!(records(&db).is_empty()); + assert!(db.verify().unwrap().is_ok(), "verify is clean"); + + let out = dir.path("nothing.graph"); + LogWriter::create(&out, OpenOptions::new()) + .unwrap() + .finish() + .unwrap(); + assert_eq!(Minigraf::open(&out).unwrap().current_tx_count(), 0); +} + +fn rec(tx_count: u64, tx_id: u64, attr: &str, value: i64) -> FactRecord { + FactRecord { + entity: uuid(0), + attribute: attr.to_string(), + value: Value::Integer(value), + tx_count, + tx_id, + valid_from: tx_id as i64, + valid_to: i64::MAX, + asserted: true, + } +} + +#[test] +fn rejected_records_change_nothing() { + let dir = Dir::new(); + let out = dir.path("out.graph"); + let mut w = LogWriter::create(&out, OpenOptions::new()).unwrap(); + let code = |r: Result<(), minigraf::MinigrafError>| r.unwrap_err().code().to_string(); + + assert_eq!(code(w.append(&rec(0, 100, ":a/x", 0))), "API-015", "tx 0"); + w.append(&rec(3, 300, ":a/x", 1)).unwrap(); + assert_eq!( + code(w.append(&rec(2, 200, ":a/x", 2))), + "API-015", + "lower tx" + ); + assert_eq!( + code(w.append(&rec(3, 301, ":a/x", 3))), + "API-016", + "two tx_ids" + ); + + let mut other_window = rec(3, 300, ":a/x", 1); + other_window.valid_to = 400; + assert_eq!(code(w.append(&other_window)), "API-011", "two windows"); + // A retraction with another window is a different record, not a conflict. + let mut retraction = other_window.clone(); + retraction.asserted = false; + w.append(&retraction).unwrap(); + + let mut big = rec(3, 300, ":a/x", 0); + big.value = Value::String("x".repeat(1 << 20)); + assert_eq!(code(w.append(&big)), "WAL-003", "oversize value"); + let mut long_attr = rec(3, 300, ":a/x", 0); + long_attr.attribute = format!(":a/{}", "x".repeat(1 << 16)); + assert_eq!(code(w.append(&long_attr)), "WAL-003", "oversize ident"); + + // Exact repeats are written once. + w.append(&rec(3, 300, ":a/x", 1)).unwrap(); + w.append(&rec(3, 300, ":a/y", 9)).unwrap(); + + // Closing a transaction by advancing makes it unjoinable. + assert_eq!(code(w.advance_tx_count(2)), "API-015", "advance backwards"); + w.advance_tx_count(3).unwrap(); + assert_eq!( + code(w.append(&rec(3, 300, ":a/x", 5))), + "API-015", + "closed tx" + ); + w.append(&rec(4, 250, ":a/x", 6)).unwrap(); // tx_id may go backwards + assert_eq!(w.tx_count(), 4); + w.finish().unwrap(); + + let db = Minigraf::open(&out).unwrap(); + let mut got: Vec = records(&db).iter().map(row).collect(); + got.sort(); + let want = vec![ + "3 300 e1 :a/x Integer(1) 300 400 false".to_string(), + "3 300 e1 :a/x Integer(1) 300 9223372036854775807 true".to_string(), + "3 300 e1 :a/y Integer(9) 300 9223372036854775807 true".to_string(), + "4 250 e1 :a/x Integer(6) 250 9223372036854775807 true".to_string(), + ]; + assert_eq!(got, want); + assert!(db.verify().unwrap().is_ok(), "verify is clean"); +} + +#[test] +fn create_refuses_existing_targets_and_read_only() { + let dir = Dir::new(); + let existing = dir.path("db.graph"); + std::fs::write(&existing, b"").unwrap(); + let err = LogWriter::create(&existing, OpenOptions::new()) + .err() + .unwrap(); + assert_eq!(err.code(), "STG-043"); + assert_eq!(std::fs::metadata(&existing).unwrap().len(), 0); + + let with_wal = dir.path("w.graph"); + std::fs::write(wal(&with_wal), b"stale").unwrap(); + let err = LogWriter::create(&with_wal, OpenOptions::new()) + .err() + .unwrap(); + assert_eq!(err.code(), "STG-043"); + assert!(!with_wal.exists()); + + let err = LogWriter::create(dir.path("r.graph"), OpenOptions::new().read_only(true)) + .err() + .unwrap(); + assert_eq!(err.code(), "API-014"); + assert!(!partial(&dir.path("r.graph")).exists()); +} + +#[test] +fn a_second_writer_for_the_same_path_is_refused() { + let dir = Dir::new(); + let out = dir.path("out.graph"); + let mut first = LogWriter::create(&out, OpenOptions::new()).unwrap(); + let err = LogWriter::create(&out, OpenOptions::new()).err().unwrap(); + assert_eq!(err.code(), "STG-025"); + first.append(&rec(1, 100, ":a/x", 1)).unwrap(); + first.finish().unwrap(); + assert_eq!(records(&Minigraf::open(&out).unwrap()).len(), 1); +} + +#[test] +fn a_stale_partial_is_reused() { + let dir = Dir::new(); + let out = dir.path("out.graph"); + std::fs::write(partial(&out), vec![0xAB; 3 * 4096]).unwrap(); + let mut w = LogWriter::create(&out, OpenOptions::new()).unwrap(); + w.append(&rec(1, 100, ":a/x", 1)).unwrap(); + w.finish().unwrap(); + let db = Minigraf::open(&out).unwrap(); + assert_eq!(records(&db).len(), 1); + assert!(db.verify().unwrap().is_ok(), "verify is clean"); +} + +#[test] +fn dropping_an_unfinished_writer_leaves_nothing() { + let dir = Dir::new(); + let out = dir.path("out.graph"); + let mut w = LogWriter::create(&out, OpenOptions::new()).unwrap(); + w.append(&rec(1, 100, ":a/x", 1)).unwrap(); + assert!(partial(&out).exists()); + drop(w); + assert!(!partial(&out).exists()); + assert!(!out.exists()); + assert!(!wal(&out).exists()); +} + +#[test] +fn finish_refuses_a_target_created_during_the_build() { + let dir = Dir::new(); + let out = dir.path("out.graph"); + let mut w = LogWriter::create(&out, OpenOptions::new()).unwrap(); + w.append(&rec(1, 100, ":a/x", 1)).unwrap(); + std::fs::write(&out, b"someone else's").unwrap(); + let err = w.finish().err().unwrap(); + assert_eq!(err.code(), "STG-043"); + assert_eq!(std::fs::read(&out).unwrap(), b"someone else's"); + assert!(!partial(&out).exists()); +} + +#[test] +fn log_writer_is_send() { + fn is_send() {} + is_send::(); +}