Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- **Binding support for cursors, open options, the fact log and the log writer** (#462, #465, #467). Two new codes let language bindings report what Rust's types rule out: `API-017` (an argument the binding cannot convert, such as an entity id that is not a UUID string), from `MinigrafError::invalid_argument`, and `API-018` (a log writer used after `finish` or `close`), from `MinigrafError::closed`. `BrowserDb.query(datalog)` returns a `BrowserCursor` with `vars()`, `nextBatch(maxRows)` (rows as JSON, as in `execute()`, or `undefined` at the end) and `close()`. The shared binding shape is in `docs/superpowers/specs/2026-10-07-binding-etl-surface-design.md`. The wasm32 test build compiles again (a duplicate `page_cache_capacity` and `tempfile` tests on wasm32).
- **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened; while one is open, checkpoints are deferred (writes stay in the WAL, automatic checkpoints wait, and `checkpoint()` or `rebuild_indexes()` with work to do fail with the new `API-013`). Close, drop or finish the log to release it. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow.
- **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened. Writes, checkpoints and `rebuild_indexes()` go on while it is open (#543): the log pins the committed generation it opened on, and no checkpoint reuses a page that generation references, so the file grows by the pages checkpoints replace until the log is closed, dropped or read to the end; later checkpoints reuse them. `API-013` (checkpoint while a fact log is open), added with the fact log in earlier v3.0.0 development builds, is deprecated: no longer emitted, still registered, never reused. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow.
- **Raw log writer: `LogWriter`** (#431). Builds a new database from explicit `FactRecord`s, the load step of an offline migration after `fact_log()` on a read-only source. Each record keeps its `tx_count`, `tx_id`, valid-time window and assertion flag. Records come in transaction order, gaps allowed: leaving a transaction out leaves a hole, and `:as-of` on it shows the state before. `advance_tx_count` keeps the source's counter after trailing purged transactions. The checks are structural only: order (new `API-015`), one `tx_id` per transaction (new `API-016`) and size (`WAL-003`); a rejected record changes nothing. Every file it builds keeps the rules of a normal write (#436, #477): a valid-time window that ends at or before it starts is `API-019`, a second window of one fact in one transaction `API-011`, and an assertion and a retraction of one fact in one transaction the new `API-020`. An unfiltered copy of a source that keeps these rules is identical to it. A source written before them (v2.x, or earlier v3 builds) can break them, and copying it fails at the first such record, so a middleware step must repair those records first; `docs/ERROR_REFERENCE.md` describes each repair. There is no WAL: records are committed in batches with the normal copy-on-write checkpoint, never splitting a transaction. The file is built at `<path>.partial` and renamed into place by `finish`, so a crash or an abandoned build never leaves part of a log at `path`; an existing target or WAL is the new `STG-043`. The page-range copy the issue proposed is not provided, since v8 pages share one dictionary and are not self-contained. The bindings follow.
- **Read-only open: `OpenOptions::read_only`** (#429). Opens a database so that nothing done through the handle writes to the `.graph` file or its WAL. The file is locked in shared mode: any number of read-only handles, in this process or others, can hold it at once, and they exclude a read-write open (and it excludes them) with the existing STG-025/STG-026. The handle shows the same data as a read-write open: a WAL left by an earlier session is applied in memory but not checkpointed or deleted, and a format v7 file is read into memory instead of migrated. A missing file is the new `STG-042` (nothing is created). `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` fail with the new `API-014`; queries, cursors, `fact_log`, `verify` and rule registration work, and dropping the handle does not checkpoint. `page_cache_size` applies as usual, so a tool can keep the whole file resident. Rust API only for now; the bindings follow.
- **Cursor API: `Minigraf::query()` and `PreparedQuery::query()`** (#432). Both return a `Cursor` that hands a query's rows back in batches: `vars()`, `next_batch(max_rows) -> Result<Option<Batch>, _>`, `close()`, and an iterator over rows. The answer is fixed when the cursor opens, so writes that commit while it is open do not change it. A cursor borrows nothing from the database handle: it can outlive it and move to another thread. For now the answer is still computed when the cursor opens, under the same limits as `execute()`; the API is shaped so the engine can later produce rows as they are pulled without breaking callers. `query()` rejects `transact`, `retract` and `rule` with the new `API-012`, and a query with bind slots with `API-010`. `execute()` is unchanged. Rust API only for now; the bindings follow.
Expand Down
7 changes: 4 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@ cargo run < demos/demo_negation.txt
- `value_pages.rs`: append-only value pages for strings over 64 bytes
- `reader.rs`: `OnDiskReader` — covering reads of committed facts (`CommittedReader`); `live_facts` decides net-assert on index keys (#379)
- `verify.rs`: integrity walk for `Minigraf::verify` (tree order, index digests, DICT, page accounting) and the source choice for `rebuild_indexes` (#373)
- `meta.rs` / `page.rs` / `freelist.rs`: meta pages A/B, the common page header and allocator, free-list chain
- `meta.rs` / `page.rs` / `freelist.rs`: meta pages A/B, the common page header and allocator (skips quarantined ids), free-list chain
- `pins.rs`: `Pins` / `GenerationPin` — generations held by open snapshots and the quarantine of pages checkpoints freed after them (#543)
- `cache.rs`: LRU page cache (`PageCache`, default 256 pages)
- `dir_sync.rs`: `sync_parent_dir` — fsyncs the parent directory after creating the `.graph`/WAL or deleting the WAL (no-op off Unix)
- `packed_pages.rs`: v7 fact pages, read only for migration
Expand All @@ -144,7 +145,7 @@ cargo run < demos/demo_negation.txt

6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`, `OpenOptions::read_only` (#429: shared lock, nothing written, writes are `API-014`)
- `src/cursor.rs`: `Cursor` / `Batch` — owned, `Send` cursor returned by `Minigraf::query` and `PreparedQuery::query` (#432); answer computed at open today, shaped for streaming operators
- `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation, deferring checkpoints (`API-013`)
- `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation (`storage::pins`), so checkpoints run but do not reuse its pages (`API-013` deprecated, #543)
- `src/log_writer.rs`: `LogWriter` — builds a new file from `FactRecord`s with their tx and valid-time kept (#431): tx-ordered appends with holes (`API-015`/`API-016`), batched copy-on-write checkpoints at tx boundaries, no WAL, built at `<path>.partial` and renamed by `finish` (`STG-043` if the target exists)

7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery. Like `storage::verify` and `storage::dir_sync`, not compiled for wasm32 (no file backend there)
Expand Down Expand Up @@ -201,7 +202,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1446 tests** (1436 passing, 10 ignored; unit + integration + doc).
**1458 tests** (1448 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Scope — format and storage integrity:
Scope — query API:

- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`, with the current evaluator computing the answer at open). The binding cursor objects follow. Streaming operators replace the evaluator behind the cursor in 3.x without further API changes; how operators bound their memory (spill, scratch pages in the `.graph` file, or a clear error) is still open.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log pins its generation: checkpoints run but do not reuse its pages (#543; `API-013` deprecated). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462.
- Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows.
- Raw log writer: `LogWriter` builds a new database from `FactRecord`s with their `tx_count`, `tx_id`, valid-time windows and assertion flags kept, gaps allowed (a purged transaction stays a hole for `:as-of`), at `<path>.partial` renamed into place by `finish` (#431; Rust API done on `v3`). Completes extract (#429, #430), transform, load. The binding objects follow.

Expand Down
4 changes: 2 additions & 2 deletions docs/DURABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ out.advance_tx_count(tx_count)?;
out.finish()?;
```

- The log reads the database as it was when it opened, uncheckpointed transactions included. Writes go on meanwhile; they stay in the WAL, and checkpoints wait until the log is dropped (`checkpoint()` with work to do fails with `API-013`).
- The log reads the database as it was when it opened, uncheckpointed transactions included. Writes and checkpoints go on meanwhile. The log pins the committed generation it opened on, so checkpoints do not reuse the pages it reads: on a database taking many writes, the file grows by the pages checkpoints replace until the log is dropped, and later checkpoints reuse them.
- Read the transaction count after opening the log, so it covers every logged transaction. The copy keeps every fact version with its `tx_count`, `tx_id` and valid-time window, so `:as-of` and `:valid-at` answer the same; internal ids and page layout are new.
- The copy is built at `backup/app.graph.partial` and renamed into place by `finish()`, so an interrupted backup never leaves a partial file under the final name.

Expand Down Expand Up @@ -185,7 +185,7 @@ Every error has a code (`MinigrafError::code()`); the [error reference](https://

| What to do | Codes |
|---|---|
| **Retry later.** Nothing is wrong with the file | `STG-026` (another process holds the database: wait for it to exit), `API-013` (a fact log is open: drop it, then checkpoint) |
| **Retry later.** Nothing is wrong with the file | `STG-026` (another process holds the database: wait for it to exit) |
| **Fix the setup.** The file is fine | `STG-025` (this process already has the file open: reuse that handle), `STG-027` (the filesystem cannot lock), `STG-042` (no file at the path, read-only open), `STG-043` (the log writer's target exists), `WAL-006` (the WAL could not be deleted after a checkpoint, usually directory permissions; nothing is lost) |
| **Reopen.** The handle refuses writes; every committed transaction is in the WAL | `STG-046`, `WAL-007`, an I/O error from a write or checkpoint (fix the cause first), `API-001` (a thread panicked while holding the write lock) |
| **Use another library version.** The file is not modified | `STG-006` (format newer than this library), `STG-034` (feature bits from a newer v3.x), `STG-028` (format older than v7: open with v2.x first), `WAL-002` (WAL from another version: open with that version and `checkpoint()`) |
Expand Down
6 changes: 4 additions & 2 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ with no `CodedError` anywhere in its chain.
| API-010 | Query with bind slots passed to execute() | Database API |
| API-011 | Two valid-time windows for one fact in one transact | Database API |
| API-012 | Non-query command passed to query() | Database API |
| API-013 | Checkpoint while a fact log is open | Database API |
| API-013 | Checkpoint while a fact log is open (deprecated) | Database API |
| API-014 | Write on a read-only handle | Database API |
| API-015 | Log writer record out of transaction order | Database API |
| API-016 | Log writer record with a second tx_id for one transaction | Database API |
Expand Down Expand Up @@ -244,7 +244,7 @@ with no `CodedError` anywhere in its chain.
| INT-055 | Rule predicate disappeared during rollback | Internal |
| INT-056 | Write to a read-only storage backend | Internal |

Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. They stay registered and documented, and error codes are never removed or reused.
Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. API-013 (checkpoint while a fact log is open) is deprecated as of v3.0.0 and is no longer emitted, because checkpoints run while fact logs are open. They stay registered and documented, and error codes are never removed or reused.

---

Expand Down Expand Up @@ -2651,6 +2651,8 @@ while let Some(batch) = cursor.next_batch(1000)? {

### API-013 Checkpoint while a fact log is open

> **Deprecated in v3.0.0.** No longer emitted: a fact log pins the committed generation it opened on, and checkpoints run while it is open without reusing the pages it reads. Kept for reference; this code will not be reused.

**Error text**: `checkpoint deferred while {} fact log(s) are open; close them and checkpoint again`

**Cause**: `db.fact_log()` returns a `FactLog` that reads the committed pages as they were when it opened. A checkpoint frees pages that a later checkpoint reuses, so while any fact log is open, checkpoints are deferred. Writes still succeed and stay in the WAL, and automatic checkpoints wait. `checkpoint()` or `rebuild_indexes()` with uncheckpointed writes, or `rebuild_indexes()` on a file database, returns this error. The `{}` is the number of open logs. Nothing is written.
Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,9 @@ the rest of the docs.
the pin holds pages (no reuse) until the cursor is dropped.

### 6. Fact log
- PR 2 adds `Snapshot` with only what the log needs (`committed`, the pin, the
pending store and its length, `pending_range`) and `FactStorage::snapshot()`;
step 5 adds `now`, the read methods and `Clone` (pin in an `Arc`).
- `FactLog` holds a `Snapshot` instead of `LogPin` + `FactStorage` + `pending_end`;
pending reads go to `snapshot.pending_range`. `Minigraf::fact_log` no longer takes
`write_lock`. Remove `LogPin`, `pin_log`, `log_pins`, `log_snapshot`.
Expand Down
Loading
Loading