Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **Per-transaction WAL sync mode and chunk size: `TxOptions` and `Minigraf::begin_write_with`** (#871, part of #862). `TxOptions` (`#[non_exhaustive]`; fields `synchronous: Option<SyncMode>` and `chunk_facts: Option<usize>`, builders `new`, `synchronous`, `chunk_facts`) sets either option for one `WriteTransaction`; `None` means the database's value, and `begin_write()` is `begin_write_with(TxOptions::default())`. `synchronous` decides whether that transaction's WAL entry is synced, whatever `OpenOptions::synchronous` is, so a bulk load can commit with `Normal` on a `Full` database; a `Full` commit syncs the whole WAL, so earlier `Normal` entries become durable with it. It has no effect on a transaction that becomes large (a checkpoint, durable in both modes; its WAL guard header is always synced, a deviation from the #862 design recorded there) or on an in-memory database. `chunk_facts` replaces `OpenOptions::transaction_chunk_facts` for that transaction, with the same clamp (1..=1,000,000). The options are not stored in the file, not gated by `allow_overrides`, and apply to that transaction only. `begin_write_with` fails as `begin_write` does (`INT-001`, `API-014`). The WAL writer no longer keeps a sync mode: each append is given its transaction's. See [docs/DURABILITY.md](docs/DURABILITY.md) for what survives a crash when modes are mixed.

- **Copyable prepared queries and browser limit setters** (#872, part of #862). `PreparedQuery` is `Clone`: a copy is an independent statement over the same database, so a limit setter applied to one leaves the other unchanged; language bindings use it to return a new prepared object from each setter. `BrowserPreparedQuery` gains `withMaxDerivedFacts(n)`, `withQueryMemoryLimit(bytes)` and `withMaxIterations(n)`, each returning a new prepared query that keeps the database open; a number at or above the platform's `usize::MAX` means no limit, and a negative, fractional or non-finite number throws `[API-017]`. `BrowserDb` still takes no open options. `docs/CONFIGURATION.md` lists which settings every binding exposes.

- **Configurable recursion round limit** (#861, part of #862). `OpenOptions::max_iterations` sets the most rounds one stratum of a rule query may run (default 1,000,000, see Breaking changes); `:max-iterations N` or `$slot` replaces it for one query, up or down (INT-012/013/014/015 as for `:max-derived-facts`), gated by the new `Overridable::MaxIterations`, which **is** in `Overridable::DEFAULTS`; `PreparedQuery::with_max_iterations` replaces it for one statement, ungated, and an allowed keyword still overrides it. Rounds are still counted per stratum. See [docs/CONFIGURATION.md](docs/CONFIGURATION.md).

### Changed
Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ cargo run < demos/demo_negation.txt
- `rules.rs`: `RuleRegistry` — thread-safe rule management
- `types.rs`: `EdnValue`, `Pattern`, `DatalogQuery`, `AsOf`, `ValidAt`, `Count` (`:limit`/`:offset` (#306) and `:max-derived-facts`/`:max-results`/`:query-memory-limit` (#870)/`:max-iterations` (#861) literal or slot), `DatalogQuery::order_by` (`:find` elements with `Order`, #306), `DatalogQuery::distinct` (default-false bare flag, #493), `WhereClause` (incl. `Not`, `NotJoin`); `PseudoAttr` enum, `AttributeSpec` wrapper
- `optimizer.rs`: Selectivity-based join reordering on every build (the `wasm` feature is a deprecated no-op). `reorder()` gates every reordering site; tests turn it off per thread (`set_reorder`, `minigraf::__reorder` under `test-hooks`, #596); `plan_seeded` orders a body with its imported variables counted as bound, `plan_after` orders a rule body's clauses after its delta, each joining on a variable bound before it where one can (#694), and `subplans()` lets tests force the engine's negation bodies and `or` branches hashed or correlated (`set_subplans`, `minigraf::__subplans`, #546)
- `prepared.rs`: `BindValue`, `PreparedQuery` — parse-once/execute-many with named `$slot` bind slots; `prepare_query`, `substitute`; `Entity` and `Val(Ref)` fit each other's slots, bind mistakes are `API-022`/`023`/`024` (#181); a `:limit $n` or `:offset $o` slot takes `Val(Integer(n >= 0))` (`resolve_count_slot`, #306); limit-keyword slots follow their literal range (`resolve_limit_slot`, #870); `with_max_derived_facts`/`with_query_memory_limit`/`with_max_iterations` replace the database limits for one statement, ungated (#870, #861)
- `prepared.rs`: `BindValue`, `PreparedQuery` — parse-once/execute-many with named `$slot` bind slots; `prepare_query`, `substitute`; `Entity` and `Val(Ref)` fit each other's slots, bind mistakes are `API-022`/`023`/`024` (#181); a `:limit $n` or `:offset $o` slot takes `Val(Integer(n >= 0))` (`resolve_count_slot`, #306); limit-keyword slots follow their literal range (`resolve_limit_slot`, #870); `with_max_derived_facts`/`with_query_memory_limit`/`with_max_iterations` replace the database limits for one statement, ungated (#870, #861); `PreparedQuery: Clone`, so bindings derive a new prepared query per setter (#872)
- `magic_sets.rs`: Magic-sets rewriting for demand-driven recursive rule evaluation (not applied to rules with `not`/`not-join`)

3a. **`src/query/engine/`** — Streaming query engine (#473), runs every query (#551): `plan.rs` (the operator tree of every query, or the executor's error code for it: patterns, expression clauses, `not`/`not-join` and `or`/`or-join`, and rule invocations, read as patterns of their relations (`plan_rules`, #548; a magic-set head constant is `HeadArg::Const`), in `optimizer::plan` order, then the `or`s in text order, then the deferred `not`s; an `or` branch or a `not` body is planned as a `:where` is; a clause list with no pattern first starts from `Seed`; a constant no cell equals is `Term::Never`, a scan that reads nothing; `:as-of` resolved when the query opens (`Snapshot::tx_bound`: a counter bound when `tx_id` rises with `tx_count`, else `TxBound::Timestamp`: committed `tx_count` ranges and pending `tx_id <= t`); a per-fact pseudo-attribute pattern reads the metadata column its source's scan fills (the hidden variable the parser named, #594), and `:db/valid-at` joins as `[e _ _]`; plain `:find` variables, aggregates with `:with`, and window functions, over an aggregate's groups too), `scan.rs` (one pattern over committed key ranges via `CommittedReader::key_source`/`live_keys`, AVET ranges for cross-kind constants, then the pending range by whole runs; one entity per probe for a join), `join.rs` (`IndexNestedLoopJoin` when the pattern's entity is bound, or, under a constant attribute and entity not constant, its value (#721: per outer row the AVET ranges of the values that join-equal it and the attribute's pending facts with its key, read once per query in `IdSpace::probes`; an entity cell needs the keyword naming it, known only where the query made the entity from a keyword (rule heads, magic seeds), else the join becomes the planner's `HashJoin`); `HashJoin` otherwise, and on a relation; `JoinKey`: floats by IEEE `==`, a keyword equals a ref to the entity it names), `filter.rs` (`Filter`/`Bind` and pseudo-attribute steps over batches; a `Bind` on a bound slot keeps the rows where the result is join-equal, #596), `collect.rs` (`CollectSet`, #316: `[(collect-set ?e :a) ?s]` per row probes `[e :a ?v]` as an `IndexNestedLoopJoin` does and binds the values as one set, `#{}` when none, metered; `Expr::CollectSet`, parsed only as a whole binding clause), `anti.rs` (`AntiJoin`, #546: a body with the same rows for every outer row runs once into a hash set of its imported-variable keys; any other body, or one whose first pattern's entity is imported, runs per outer row from `seed.rs`'s one-row `Seed`, via `Operator::rewind`; never deduplicates), `aggregate.rs` (`Aggregate`, #547: a hash table of groups keyed on cells as `Value`'s `Ord` compares values, one entry when ungrouped; `count`/`count-distinct` on cells, other aggregates on values via `datalog::aggregate::Agg`; groups sorted by value and finished in that order; a float group shows its least value by `value_order`; #641/#781 certify a complete grouping tuple as `Grouping::Ordered`: count/count-distinct/sum/avg, fixed-scalar extrema and Integer/nil sum-distinct; one varying ordered slot plus permitted scalar constants/repetitions, including :with (which retains Sort), no window/callback/CollectSet; one active group, bounded output batches, shared finish_group, internal DICT-cache eviction; unproved arguments and registered finalizers keep full-drain validation), `order.rs` and `order/arguments.rs` (#641/#781: snapshot-bound grouping/order and argument-domain proofs of the complete result order; safe committed integer/float/Boolean ranges can set the private `ScanSpec.index` hint to AVET and omit Sort, with `plan.find` restored from Sort.cols; entity-and-attribute scans retain EAVT; pending/staged records in that range, callbacks and CollectSet keep the original strategies; DICT comparators prove only window partition adjacency, never final Sort or aggregate group order; mixed integer/float keys can prove grouping order but keep Sort, and unique group keys discharge tie suffixes; `minigraf::__input_order` forces fallback under `test-hooks`), `sort.rs` (`Sort`, #306: the `:find` columns as values, in the full order of `:order-by` — its keys, then every other `:find` column ascending, by `datalog::aggregate::row_order`/`value_order`; under a `:limit` a heap of the best `offset + limit` rows; reads its whole input, dropping the DICT lookups after each batch, metered; its rows turned back into cells only as returned), `limit.rs` (`Limit`, #306: skips `offset` rows, passes on at most `limit`, then closes its child so nothing below reads more; planned after the last step, after `Builder::finish`, and after `Sort`), `window.rs` (`Window`, #547: holds each input row's passed-on cells and its named variables' values, then computes every window with `datalog::aggregate::windows`, the executor's code, and returns the rows in input order; #641 resolves compatible built-in windows to `WindowPartitioned`, retaining one partition and charged lookahead, coalescing completed partitions into bounded output batches, evicting DICT caches at internal input/result boundaries; full-input mode keeps query-lifetime caches), `union.rs`/`distinct.rs` (`Union` and `Distinct`, #546: each `or` branch planned as a `not` body is, hashed into a table or run per outer row; then one row per distinct binding, as the executor's `Value`s compare; `:distinct` adds `DistinctFind` after aggregates/windows and before sorting/pagination, keyed on final `:find` columns by aggregate-group equality, with canonical float zero/NaN output including set elements, #493), `cell.rs`/`batch.rs` (id cells in columnar batches of 1,024), `ids.rs` (DICT and provisional ids, `keyword_eid`), `project.rs` (ids → values for returned rows only; bounded stack UUID hints share DICT paths, with cache insertion and charges in projection order, #530), `relation.rs`/`recursive.rs` (#548: rule queries; a relation per evaluated predicate holds its derived rows, a set by `encode_value` identity, appended per round so `Old`/`Delta`/`Full` are ranges; `Recursive` runs each stratum's semi-naive rounds, one run per same-stratum top-level invocation, a rule reading its stratum otherwise run in full every round; scans read derived rows after committed and pending ones, with `nil` metadata), `meter.rs` (#549: one `Meter` per query, shared by every holder of state through a `Held` — hash join build sides, `not`/`or` tables, distinct sets, aggregate groups, window rows, rule relations and their temporaries, `IdSpace`, the set a `CollectSet` builds; each entry charged by a per-entry estimate including retained set elements and strings before it is inserted, containers grown with `try_reserve`, released on `rewind`/`close`/drop; over `OpenOptions::query_memory_limit`, or a refused allocation, is `QRY-012`; `Stream::close` drops the operator tree, keeping its counters). `mod.rs`: `Operator` (`next_batch`/`rewind`/`close`/`counters`), `Stream` (the operator tree over an owned `Snapshot`, pulled one batch at a time). `DatalogExecutor::open` / `execute_query` run every query on it: a cursor gets the `Stream` (`Answer::Stream`), `execute` drains it.
Expand All @@ -197,7 +197,7 @@ cargo run < demos/demo_negation.txt

8. **`src/error.rs`** — Structured error codes: `MinigrafError`, `ErrorCategory`, `ErrorCode` registry (PRS/QRY/STG/WAL/API/INT codes); must match `docs/ERROR_REFERENCE.md`. `MinigrafError::invalid_argument` (API-017) and `MinigrafError::closed` (API-018) exist for language bindings

9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object, read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`
9. **`src/browser/`** — Browser WASM backend (`browser` feature): `buffer.rs` (`BrowserBufferBackend`, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, `pins::split_released`), `indexeddb.rs` (IndexedDB persistence); `BrowserDb.query` returns a `BrowserCursor` (JSON row batches); `BrowserDb.prepare` returns a `BrowserPreparedQuery` (`execute`/`query` with a bindings object; `withMaxDerivedFacts`/`withQueryMemoryLimit`/`withMaxIterations` return a new one, #872; read by `binds.rs`: plain JS values or `{type, value}` forms, #181; a set as `{type: 'set', value: [..]}` or a JS `Set`, #316); `udfs.rs`: `registerPredicate`/`registerAggregate` (#180; values as `{type, value}` objects, a throw is `QRY-014`; JS values held in a thread-local table, the registry keeps ids). Tests here must be `#[wasm_bindgen_test]` (the module only builds for wasm32); `.github/workflows/wasm.yml` runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need `#![cfg(not(target_arch = "wasm32"))]`

### Data Model

Expand Down Expand Up @@ -250,7 +250,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**2132 tests** (2111 active, 21 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
**2133 tests** (2112 active, 21 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings.
Expand Down
35 changes: 35 additions & 0 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,3 +141,38 @@ let pq = db
.with_query_memory_limit(16 << 20);
let rows = pq.execute(&[])?;
```

`PreparedQuery` is `Clone` (#872): `pq.clone().with_max_iterations(10)` gives
a second statement with its own limits and leaves `pq` unchanged.

## Language bindings

From v3.0.0 every binding exposes the override list, its default, the
`transaction_chunk_facts` and `max_iterations` open options and the three
setters (#872). Each setter returns a new prepared query and leaves the one
it was called on unchanged. The default list is read from
`Overridable::DEFAULTS`, never written out again in a binding. `TxOptions`
reaches the bindings with binding write transactions (#824).

| Binding | Override list | Default list | Open options | Setters |
|---|---|---|---|---|
| Python | `minigraf.open(..., allow_overrides=[Overridable.MAX_DERIVED_FACTS, ...])` | `minigraf.OVERRIDABLE_DEFAULTS` | `transaction_chunk_facts=`, `max_iterations=` | `with_max_derived_facts`, `with_query_memory_limit`, `with_max_iterations` |
| Java | builder `Minigraf.options().allowOverrides(...)` | `Minigraf.OVERRIDABLE_DEFAULTS` | `transactionChunkFacts`, `maxIterations` | `withMaxDerivedFacts`, `withQueryMemoryLimit`, `withMaxIterations` |
| Android | `OpenOptions(allowOverrides = ...)` | top-level `OVERRIDABLE_DEFAULTS` | `transactionChunkFacts`, `maxIterations` | `withMaxDerivedFacts`, `withQueryMemoryLimit`, `withMaxIterations` |
| Swift | `OpenOptions(allowOverrides: ...)` | `Overridable.defaults` | `transactionChunkFacts`, `maxIterations` | `withMaxDerivedFacts(n:)`, `withQueryMemoryLimit(bytes:)`, `withMaxIterations(n:)` |
| Node | `allowOverrides: ['maxDerivedFacts', 'maxResults', 'queryMemoryLimit', 'maxIterations']` | `OVERRIDABLE_DEFAULTS` | `transactionChunkFacts`, `maxIterations` | `withMaxDerivedFacts`, `withQueryMemoryLimit`, `withMaxIterations` |
| C | `"allow_overrides": ["max_derived_facts", "max_results", "query_memory_limit", "max_iterations"]` in the options JSON | `minigraf_overridable_defaults()` (a JSON array) | `"transaction_chunk_facts"`, `"max_iterations"` | `minigraf_prepared_with_max_derived_facts`, `minigraf_prepared_with_query_memory_limit`, `minigraf_prepared_with_max_iterations` |
| Browser (`BrowserDb`) | none | none | none | `withMaxDerivedFacts`, `withQueryMemoryLimit`, `withMaxIterations` on `BrowserPreparedQuery` |

As in Rust, a missing override list means the default list and an empty one
refuses every override keyword; a list replaces the defaults rather than
adding to them. Counts are signed in the bindings, and a negative one is
`API-017`. The no-limit value of a setter is the one each binding already
uses for `query_memory_limit`: `i64::MAX` in Python, Java, Android and Swift,
`Number.MAX_SAFE_INTEGER` or more in Node, `SIZE_MAX` in C.

`BrowserDb` takes no open options, so it always runs with the defaults,
`Overridable::DEFAULTS` among them, and refuses `:query-memory-limit` in
query text. Its prepared queries' setters take a non-negative integer
number; at or above the platform's largest `usize` (2^32 − 1) it means no
limit, and a negative, fractional or non-finite number is `API-017`.
Loading
Loading