Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **Fact log: `Minigraf::fact_log()`** (#430). Streams every fact version in the database as `FactRecord`s (entity, attribute, value, `tx_count`, `tx_id`, valid-time bounds, and whether it is an assertion or a retraction), without Datalog. `FactFilter` selects attributes (exact or by prefix), entities and a `tx_count` range, checked on index keys so excluded records are never decoded. `FactOrder::Tx` (the default) returns records in ascending `tx_count`; format v8 has no index in transaction order, so it holds at most `FactFilter::window` records (default 262,144) and scans the selected keys once per window. `FactOrder::Storage` reads in one pass. The log reads the database as it was when it opened; while one is open, checkpoints are deferred (writes stay in the WAL, automatic checkpoints wait, and `checkpoint()` or `rebuild_indexes()` with work to do fail with the new `API-013`). Close, drop or finish the log to release it. Like `Cursor`, a `FactLog` is owned and `Send`. Rust API only for now; the bindings follow.
- **Cursor API: `Minigraf::query()` and `PreparedQuery::query()`** (#432). Both return a `Cursor` that hands a query's rows back in batches: `vars()`, `next_batch(max_rows) -> Result<Option<Batch>, _>`, `close()`, and an iterator over rows. The answer is fixed when the cursor opens, so writes that commit while it is open do not change it. A cursor borrows nothing from the database handle: it can outlive it and move to another thread. For now the answer is still computed when the cursor opens, under the same limits as `execute()`; the API is shaped so the engine can later produce rows as they are pulled without breaking callers. `query()` rejects `transact`, `retract` and `rule` with the new `API-012`, and a query with bind slots with `API-010`. `execute()` is unchanged. Rust API only for now; the bindings follow.
- **`Minigraf::verify()` and `Minigraf::rebuild_indexes()`** (#373). Page checksums turn a torn or rotted page into an error, but they cannot see a page that is valid and says the wrong thing. `verify()` walks every committed page and returns an `IntegrityReport` with one structured error per finding: an index that holds different facts than EAVT or a different count than the meta page (new `STG-038`), keys out of order or a page reached twice (new `STG-039`), dictionary maps that disagree (new `STG-040`), ids with no dictionary entry (`STG-036`), and leaked, doubly used or wrongly free pages (`STG-035`). It keeps no per-fact state: indexes are compared by an order-independent digest. `rebuild_indexes()` rebuilds EAVT, AEVT, AVET and VAET from an intact index that agrees with another one (or the only intact one), derives the free list again, and commits like a checkpoint, including uncheckpointed writes; a crash at any point leaves the previous checkpoint. If no index can serve as the source, or the dictionary is damaged, it fails with new `STG-041` and writes nothing. Both are Rust API only for now; the bindings follow.

Expand Down
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,9 @@ cargo run < demos/demo_negation.txt

5. **`src/repl.rs`** — Interactive REPL; TTY-aware (suppresses prompts/banner for piped input)

6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`
6. **`src/db.rs`** — Public API: `Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes`, `WriteTransaction`, `IntegrityReport`, `OpenOptions::page_cache_size`
- `src/cursor.rs`: `Cursor` / `Batch` — owned, `Send` cursor returned by `Minigraf::query` and `PreparedQuery::query` (#432); answer computed at open today, shaped for streaming operators
- `src/fact_log.rs`: `FactLog` / `FactFilter` / `FactRecord` / `FactOrder` — every fact version streamed by `Minigraf::fact_log` (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation, deferring checkpoints (`API-013`)

7. **`src/wal.rs`** — Fact-level sidecar WAL, CRC32-protected entries, crash recovery

Expand Down Expand Up @@ -195,7 +196,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1285 tests** (1276 passing, 9 ignored; unit + integration + doc).
**1309 tests** (1300 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ v2.x gets data-integrity and security fixes for 12 months after v3.0.0 ships. Se
## Quick Start

```rust
use minigraf::{Minigraf, OpenOptions};
use minigraf::{FactFilter, Minigraf, OpenOptions};

// Open or create a file-backed database
let db = OpenOptions::new().path("myapp.graph").open()?;
Expand All @@ -90,6 +90,12 @@ while let Some(batch) = cursor.next_batch(1000)? {
for row in batch.rows() { /* row[0] is ?name */ }
}

// Fact log — every assertion and retraction, in transaction order, without Datalog
let mut log = db.fact_log(&FactFilter::new().attribute_prefix(":person/"))?;
while let Some(batch) = log.next_batch(1000)? {
for rec in batch { /* rec.tx_count, rec.entity, rec.attribute, rec.value, rec.asserted */ }
}

// Explicit transaction — all-or-nothing
let mut tx = db.begin_write()?;
tx.execute(r#"(transact [[:alice :person/age 31]])"#)?;
Expand All @@ -111,7 +117,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1285 tests
cargo test # run 1309 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
1 change: 1 addition & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ Scope — format and storage integrity:
Scope — query API:

- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`, with the current evaluator computing the answer at open). The binding cursor objects follow. Streaming operators replace the evaluator behind the cursor in 3.x without further API changes; how operators bound their memory (spill, scratch pages in the `.graph` file, or a clear error) is still open.
- Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log defers checkpoints (`API-013` for an explicit one). Read-only open (#429) and a raw log writer (#431) complete the extract-transform-load path; the binding objects follow with #462.

Scope — release-gate testing and documentation:

Expand Down
20 changes: 20 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ with no `CodedError` anywhere in its chain.
| API-010 | Query with bind slots passed to execute() | Database API |
| API-011 | Two valid-time windows for one fact in one transaction | Database API |
| API-012 | Non-query command passed to query() | Database API |
| API-013 | Checkpoint while a fact log is open | Database API |
| INT-000 | Unclassified internal error | Internal |
| INT-001 | WriteTransaction already in progress on this thread | Internal |
| INT-002 | Invalid entity (API layer) | Internal |
Expand Down Expand Up @@ -2472,6 +2473,25 @@ while let Some(batch) = cursor.next_batch(1000)? {

**Scenario**: Code that dispatches every command string through one function switches from `execute()` to `query()` for all of them.

### API-013 Checkpoint while a fact log is open

**Error text**: `checkpoint deferred while {} fact log(s) are open; close them and checkpoint again`

**Cause**: `db.fact_log()` returns a `FactLog` that reads the committed pages as they were when it opened. A checkpoint frees pages that a later checkpoint reuses, so while any fact log is open, checkpoints are deferred. Writes still succeed and stay in the WAL, and automatic checkpoints wait. `checkpoint()` or `rebuild_indexes()` with uncheckpointed writes, or `rebuild_indexes()` on a file database, returns this error. The `{}` is the number of open logs. Nothing is written.

**Resolution**:
- Finish reading, or call `close()` on the log (dropping it is the same), then checkpoint again.
- Keep fact logs short-lived on a database that takes writes, because the WAL grows until they close.

```rust
let mut log = db.fact_log(&FactFilter::new())?;
while let Some(batch) = log.next_batch(1000)? { /* ... */ }
log.close();
db.checkpoint()?;
```

**Scenario**: A backup job streams the fact log on a thread while the application calls `checkpoint()` on a timer.

---

## INT — Internal Errors
Expand Down
Loading
Loading