Skip to content

enhancement: streaming fact-log iterator with full history (tx, asserted, valid bounds) and coarse filters #430

Closed

Description

@adityamukho

Motivation

An offline migration (project-minigraf/temporal_reasoning#378) needs to read every fact version in a graph, retracted ones included, in transaction order, without going through Datalog. Today history is reachable only per transaction via :as-of N, and a full-graph Datalog scan on the server's thread has starved an active ingestion for over 20 minutes; there is no API that yields the log as a stream.

Proposal

A read-side API on Minigraf (usable in the read-only mode requested separately):

pub struct FactRecord { entity, attribute, value, tx: u64, asserted: bool, valid_from, valid_to }
pub fn iter_facts(&self, filter: &FactFilter) -> impl Iterator<Item = Result<FactRecord>>
  • Order: transaction order, then insertion order within a transaction, so a consumer can rebuild the log and keep :as-of N meaning the same thing for facts it copies unchanged.
  • Filters (all optional, ANDed): entity ident prefix (:lineage-marker/, :ingestion/), attribute set, tx range, entity-type. A migration that only rewrites two namespaces must not pay to decode the 90% it will copy unchanged; when a filter is present the iterator should also expose the complementary "untouched" ranges so a caller can copy them page-wise (see the writer issue).
  • Streaming: bounded memory regardless of graph size; no materialised result set, so max_results does not apply.
  • Exposed through the FFI as a cursor (next_batch(n)), since uniffi has no iterator type.

This also gives the temporal_reasoning FTS sidecar a rebuild path that does not depend on the entity index (cf. minigraf #370).

Activity

  1. added this to the v3.0.0 milestone on Oct 1, 2026
  2. added
    rustPull requests that update rust code
    performanceBenchmark performance regressions
    on Oct 1, 2026
  3. added a commit that references this issue on Oct 7, 2026
  4. adityamukho commented on Oct 7, 2026

    @adityamukho
    CollaboratorAuthor

    Done. Core: Minigraf::fact_log(&FactFilter) in #463: tx order or storage order, bounded memory, attribute/attribute-prefix/entity/tx-range filters, and a checkpoint pin while a log is open. Bindings: a next_batch(n) cursor of lossless FactRecords in every binding via #467. Differences from this proposal, decided in the design (docs/superpowers/specs/2026-10-07-fact-log-iterator-design.md): no entity-type filter and no 'untouched ranges' output, because #431's page-range copy was dropped (v8 pages share one dictionary and are not self-contained). Ships in v3.0.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceBenchmark performance regressionsrustPull requests that update rust code

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions