Skip to content

feat: fact-log iterator — Minigraf::fact_log (#430) - #463

Merged
adityamukho merged 2 commits into
v3from
feat/430-fact-log
Oct 7, 2026
Merged

adityamukho merged 2 commits into
v3from
feat/430-fact-log

Conversation

@adityamukho

Copy link
Copy Markdown
Collaborator

Part of #430 (Rust API; the binding objects follow with #462, read-only open is #429, the raw writer is #431).

What

Minigraf::fact_log(&FactFilter) -> FactLog streams every fact version (assertions and retractions) as FactRecord { entity, attribute, value, tx_count, tx_id, valid_from, valid_to, asserted }, without Datalog. Like Cursor, FactLog is owned and Send and provides next_batch(max), close() and an iterator.

  • Filters (ANDed): exact attributes, attribute prefixes (the two are ORed), entities, and a tx_count range. They are checked on index key bytes, so excluded records are never decoded. An entity set walks EAVT entity prefixes, an exact-attribute set walks AEVT attribute prefixes, and anything else walks all of EAVT with a memoised attribute check.

  • Order, chosen by the caller:

    • FactOrder::Tx (default) returns records in ascending tx_count. v8 has no tx-ordered index and the format is frozen, so each pass keeps the window smallest (tx, key) entries after the last one emitted, in a bounded heap. Memory is O(window) (default 262,144) and the cost is ceil(N/window) key scans. Order within one transaction is unspecified but deterministic.
    • FactOrder::Storage reads in one pass.
  • Snapshot and pin. A reader must not span two checkpoints, because freed pages are reused by the next one. An open log therefore pins the generation it opened on:

    • automatic checkpoints, and the checkpoint on drop, are deferred;
    • checkpoint() and rebuild_indexes() fail with the new API-013 when they have work to do;
    • writes still succeed and stay in the WAL.

    Opening takes the write lock briefly, so a log never sees half a transaction. Closing, dropping or finishing the log releases the pin.

The issue also asked for two filters this PR does not provide:

  • Entity ident prefix: keyword entities are stored as UUIDv5 hashes, so the ident can't be recovered.
  • Entity type: this needs a join. Instead, run a query for the UUIDs and pass them to entities.

The issue's "untouched page ranges" doesn't apply either: v8's copy-on-write trees share a dictionary, so a range of pages can't be copied on its own. All three are written up in the spec.

Design: docs/superpowers/specs/2026-10-07-fact-log-iterator-design.md

Tests

tests/fact_log_test.rs has 22 tests and there are 2 new doctests, for 1309 total (1300 passing, 9 ignored). The tests cover:

  • Model checks: the records are checked against a model of the writes in memory, with everything pending, half checkpointed, fully checkpointed and after reopen. This runs in both orders, with windows of 1 to 1000 and batch sizes of 1 to 1000.
  • Edge cases: a transaction larger than the window, long values, and valid-time windows.
  • Filters: each filter and their combinations are compared against the model.
  • Snapshot and pin: writes after open are excluded; checkpoints are refused or deferred with API-013 while a log is open; auto-checkpoint from execute and WriteTransaction::commit waits for the log; close, drop or finish releases it.
  • Threading and errors: INT-001 on a write-transaction thread; a log outlives its handle and the deferred writes replay on reopen.

A local release run on 200k facts reads about 200k records/s, with decoding dominating. One Tx-order key pass over 200k takes about 0.15 s.

Docs updated: CHANGELOG, ROADMAP, README, CLAUDE.md, TEST_COVERAGE, ERROR_REFERENCE (API-013). The wiki is left for the v3.0.0 doc sync, as with the earlier v3 PRs.

🤖 Generated with Claude Code

https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn

adityamukho and others added 2 commits October 7, 2026 09:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
Streams every fact version (assertions and retractions, with tx_count,
tx_id and valid-time bounds) as FactRecords, without Datalog. FactFilter
selects exact attributes, attribute prefixes, entities and a tx range,
checked on index keys before decoding.

FactOrder::Tx (default) returns ascending tx_count. v8 has no tx-ordered
index, so each pass keeps the next `window` smallest (tx, key) entries in
a bounded heap: memory O(window), ceil(N/window) key scans.
FactOrder::Storage reads in one pass.

An open log pins the committed generation it opened on (pages freed by a
checkpoint are reused by the next one): auto-checkpoints and the
checkpoint on drop are deferred, and checkpoint()/rebuild_indexes() with
work to do fail with the new API-013. Opening takes the write lock
briefly, so the snapshot never holds half a transaction.

Design: docs/superpowers/specs/2026-10-07-fact-log-iterator-design.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015GgLEs8YozJjYhwPEYgTUn
@adityamukho
adityamukho merged commit 5cc35ac into v3 Oct 7, 2026
19 checks passed
@codecov

codecov Bot commented Oct 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.17808% with 17 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/fact_log.rs 94.49% 12 Missing ⚠️
src/storage/reader.rs 90.32% 3 Missing ⚠️
src/storage/mod.rs 0.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant