Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened; while one is open, checkpoints are deferred (writes stay in the WAL, automatic checkpoints wait, and `checkpoint()` or `rebuild_indexes()` with work to do fail with the new `API-013`). Close, drop or finish the log to release it. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow.
- **Read-only open: `OpenOptions::read_only`** (#429). Opens a database so that nothing done through the handle writes to the `.graph` file or its WAL. The file is locked in shared mode: any number of read-only handles, in this process or others, can hold it at once, and they exclude a read-write open (and it excludes them) with the existing STG-025/STG-026. The handle shows the same data as a read-write open: a WAL left by an earlier session is applied in memory but not checkpointed or deleted, and a format v7 file is read into memory instead of migrated. A missing file is the new `STG-042` (nothing is created). `transact`, `retract`, `begin_write`, `checkpoint` and `rebuild_indexes` fail with the new `API-014`; queries, cursors, `fact_log`, `verify` and rule registration work, and dropping the handle does not checkpoint. `page_cache_size` applies as usual, so a tool can keep the whole file resident. Rust API only for now; the bindings follow.
- **Cursor API: `Minigraf::query()` and `PreparedQuery::query()`** (#432). Both return a `Cursor` that hands a query's rows back in batches: `vars()`, `next_batch(max_rows) -> Result<Option<Batch>, _>`, `close()`, and an iterator over rows. The answer is fixed when the cursor opens, so writes that commit while it is open do not change it. A cursor borrows nothing from the database handle: it can outlive it and move to another thread. For now the answer is still computed when the cursor opens, under the same limits as `execute()`; the API is shaped so the engine can later produce rows as they are pulled without breaking callers. `query()` rejects `transact`, `retract` and `rule` with the new `API-012`, and a query with bind slots with `API-010`. `execute()` is unchanged. Rust API only for now; the bindings follow.
- **`Minigraf::verify()` and `Minigraf::rebuild_indexes()`** (#373). Page checksums turn a torn or rotted page into an error, but they cannot see a page that is valid and says the wrong thing. `verify()` walks every committed page and returns an `IntegrityReport` with one structured error per finding: an index that holds different facts than EAVT or a different count than the meta page (new `STG-038`), keys out of order or a page reached twice (new `STG-039`), dictionary maps that disagree (new `STG-040`), ids with no dictionary entry (`STG-036`), and leaked, doubly used or wrongly free pages (`STG-035`). It keeps no per-fact state: indexes are compared by an order-independent digest. `rebuild_indexes()` rebuilds EAVT, AEVT, AVET and VAET from an intact index that agrees with another one (or the only intact one), derives the free list again, and commits like a checkpoint, including uncheckpointed writes; a crash at any point leaves the previous checkpoint. If no index can serve as the source, or the dictionary is damaged, it fails with new `STG-041` and writes nothing. Both are Rust API only for now; the bindings follow.

Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ cargo run < demos/demo_negation.txt

5. **`src/repl.rs`** — Interactive REPL; TTY-aware (suppresses prompts/banner for piped input)

6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`
6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`, `OpenOptions::read_only` (#429: shared lock, nothing written, writes are `API-014`)
- `src/cursor.rs`: `Cursor` / `Batch` — owned, `Send` cursor returned by `Minigraf::query` and `PreparedQuery::query` (#432); answer computed at open today, shaped for streaming operators
- `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation, deferring checkpoints (`API-013`)

Expand Down Expand Up @@ -196,7 +196,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1309 tests** (1300 passing, 9 ignored; unit + integration + doc).
**1329 tests** (1320 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,10 @@ while let Some(batch) = log.next_batch(1000)? {
for rec in batch { /* rec.tx_count, rec.entity, rec.attribute, rec.value, rec.asserted */ }
}

// Read-only open — for backups and offline tools. Read-only handles share a file;
// a read-write handle (like `db` on myapp.graph) excludes them.
let archive = Minigraf::open_with_options("archive.graph", OpenOptions::new().read_only(true))?;

// Explicit transaction — all-or-nothing
let mut tx = db.begin_write()?;
tx.execute(r#"(transact [[:alice :person/age 31]])"#)?;
Expand All @@ -117,7 +121,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1309 tests
cargo test # run 1329 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
3 changes: 2 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ Scope — format and storage integrity:
Scope — query API:

- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`, with the current evaluator computing the answer at open). The binding cursor objects follow. Streaming operators replace the evaluator behind the cursor in 3.x without further API changes; how operators bound their memory (spill, scratch pages in the `.graph` file, or a clear error) is still open.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). Read-only open (#429) and a raw log writer (#431) complete the extract-transform-load path; the binding objects follow with #462.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462.
- Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows.

Scope — release-gate testing and documentation:

Expand Down
48 changes: 48 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ with no `CodedError` anywhere in its chain.
| STG-039 | Malformed tree | Storage |
| STG-040 | Dictionary inconsistency | Storage |
| STG-041 | Cannot rebuild indexes | Storage |
| STG-042 | Database file not found (read-only open) | Storage |
| WAL-001 | Invalid WAL magic number | WAL |
| WAL-002 | Unsupported WAL version | WAL |
| WAL-003 | Value size exceeds maximum | WAL |
Expand All @@ -169,6 +170,7 @@ with no `CodedError` anywhere in its chain.
| API-011 | Two valid-time windows for one fact in one transaction | Database API |
| API-012 | Non-query command passed to query() | Database API |
| API-013 | Checkpoint while a fact log is open | Database API |
| API-014 | Write on a read-only handle | Database API |
| INT-000 | Unclassified internal error | Internal |
| INT-001 | WriteTransaction already in progress on this thread | Internal |
| INT-002 | Invalid entity (API layer) | Internal |
Expand Down Expand Up @@ -225,6 +227,7 @@ with no `CodedError` anywhere in its chain.
| INT-053 | Header checksum mismatch: possible file corruption | Internal |
| INT-054 | Unstratifiable negative recursion cycle | Internal |
| INT-055 | Rule predicate disappeared during rollback | Internal |
| INT-056 | Write to a read-only storage backend | Internal |

Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. They stay registered and documented, and error codes are never removed or reused.

Expand Down Expand Up @@ -1974,6 +1977,8 @@ See the [file format section in README](../README.md#file-format) for version hi
- Reuse the handle you already have. `Minigraf` is cheap to clone and all clones share one database.
- If you cannot find the other handle, it is usually held by a longer-lived object than you expect — a cache, a registry, or a background task.

Read-only handles (`OpenOptions::read_only`) share a file with each other, but not with a read-write handle: either kind refuses the other with this error.

**Scenario**: A request handler calls `Minigraf::open` per request while a connection pool already holds the same path open.

---
Expand All @@ -1989,6 +1994,8 @@ See the [file format section in README](../README.md#file-format) for version hi
- There is no lock file to delete. If you find a stale `.graph.lock`, it is a leftover from a version before 2.0 and has no effect — it is not read or deleted, and should not be removed by hand in case an old process still depends on it.
- If two services genuinely need the same file, put one in front of the other. Minigraf is embedded, not a server.

A read-only handle (`OpenOptions::read_only`) in another process also refuses a read-write open, and a read-write handle refuses a read-only one. Read-only handles do not refuse each other.

**Scenario**: A rolling deployment starts the new pod before the old one has exited, and both mount the same volume.

---
Expand Down Expand Up @@ -2180,6 +2187,20 @@ See the [file format section in README](../README.md#file-format) for version hi

---

### STG-042 Database file not found (read-only open)

**Error text**: `database file not found ({}); a read-only open does not create one`

**Cause**: The database was opened with `OpenOptions::read_only` and no file exists at the path. A read-write open creates a new, empty database; a read-only open never creates or writes anything.

**Resolution**:
- Check the path. The `{}` is the path as given.
- To create the database, open it once without `read_only`.

**Scenario**: A migration tool is pointed at `memory.graph` from the wrong working directory.

---

## WAL — Write-Ahead Log Errors

WAL errors relate to the sidecar `.wal` file written alongside the `.graph` file.
Expand Down Expand Up @@ -2492,6 +2513,24 @@ db.checkpoint()?;

**Scenario**: A backup job streams the fact log on a thread while the application calls `checkpoint()` on a timer.

### API-014 Write on a read-only handle

**Error text**: `database is open read-only; {} is not allowed`

**Cause**: The database was opened with `OpenOptions::read_only`, and the call would write to the file or its WAL. The `{}` names the call: `transact`, `retract`, `begin_write`, `checkpoint` or `rebuild_indexes`. Nothing is written and no transaction number is used.

**Resolution**:
- Open the database without `read_only` to write. A read-write open waits for no one: it fails with STG-025 or STG-026 while any read-only handle holds the file, so close those first.
- Queries, cursors, `fact_log()`, `verify()` and rule registration work on a read-only handle.

```rust
let src = Minigraf::open_with_options("old.graph", OpenOptions::new().read_only(true))?;
let mut log = src.fact_log(&FactFilter::new())?;
// ... copy the records elsewhere; src.execute("(transact ...)") would be API-014
```

**Scenario**: An offline migration tool opens its source read-only and, by mistake, runs a cleanup `retract` against it instead of against the destination.

---

## INT — Internal Errors
Expand Down Expand Up @@ -3189,3 +3228,12 @@ this transaction, but the rule registry no longer contained it.
**Resolution**:
- File a bug report — this indicates a lifecycle bug in
`WriteTransaction`'s rule-staging/rollback logic.

### INT-056 Write to a read-only storage backend

**Error text**: `write to a read-only storage backend: {}`

**Cause**: A page write or sync reached the file backend of a handle opened with `OpenOptions::read_only`. The public API refuses every write on such a handle first (API-014), so this is a missed check, not a usage error. The `{}` is the page or `sync`. Nothing was written.

**Resolution**:
- File a bug report with the call that returned it.
Loading
Loading