Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A per-fact pseudo-attribute reads the nearest pattern before it with the same entity** (#594). `:db/valid-from`, `:db/valid-to`, `:db/tx-count` and `:db/tx-id` read the metadata of one fact: the one matched by their *source*, chosen when the query, rule or prepared query is parsed, as the nearest pattern before them in the `:where` text with the same entity term and a stored attribute (a keyword, or a variable as in `[?e ?a _]`). The search starts in the pseudo-attribute's own clause list and moves outwards, before each enclosing `not` / `or`; it crosses a `not-join` / `or-join` only for an entity variable in its join variables, and stops at a rule body, which never reads its caller's patterns. Terms are the same when they are one variable, one `$slot`, or constants naming one entity (`:alice` and its UUID). Before, they read the last pattern *the planner ran* that matched the entity, so the answer depended on the plan and differed under the `wasm` feature: in `[?e :person/name ?n] [?e :person/age ?a] [?e :db/tx-count ?t]`, `?t` could be the tx-count of either fact, and is now the `:person/age` fact's. Three further changes follow: (1) with no source, the query fails at parse time with the new `PRS-081` instead of matching every visible fact of the entity (`[?e :db/tx-count ?t]` before `[?e :person/name ?n]`, a `_` entity, a pseudo-attribute in a rule body whose only candidate is in the caller); to read every fact of an entity, write `[?e ?a _]` before it; (2) inside `not`, `not-join`, `or` and `or-join`, a pseudo-attribute reads the outer fact, so `[?e :person/name ?n] (not [?e :db/tx-count 1])` excludes the entities whose *name* fact is from tx 1, where before the body matched any fact of the entity; (3) rows that differ only in a source fact's metadata are no longer merged by `or`. `:db/valid-at` is unchanged. The pseudo-attribute pattern now binds or checks a value; it never scans facts, so the planner places it after its source rather than by selectivity.
- **A pseudo-attribute pattern never gives a variable its value** (#606). A variable bound by several patterns returns the value of the first real pattern in the `:where` text that binds it (#545); a `:db/valid-at` pattern that the planner ran first used to bind its entity variable itself, as a ref, so `[:find ?b ?t :where [?b :db/valid-at ?t] [?a :friend ?b] [?b :name "Bob"]]` returned the ref of `:bob` with source order and the keyword `:bob` stored by `:friend` natively. It returns the keyword now, whatever runs first. Only the keyword-or-ref kind changes; rows and counts are the same.
- **Every build reorders query clauses; the `wasm` feature does nothing** (#594, #596). The `wasm` Cargo feature kept patterns, `not` clauses and `or` branches in source order and turned off the `or` short-circuits (#250), for small browser datasets and a predictable order. With every answer independent of the plan (#594, #596, #606) it only made queries slower, and no first-party build enabled it (the browser package uses `browser`). The feature is kept, so builds that name it still resolve, but is deprecated and has no effect.
- **Aggregate and window answers no longer depend on the order of their input** (#547). Before, aggregates and window functions read their rows in the order the plan produced them, and several answers followed that order. Six cases changed. (1) `sum` over floats is exact: the values are added without rounding and the total is rounded once, so `1e16 + 1.0 + -1e16 + 1.0 + 0.5` gives `2.5` in every order (before, `1.5` or `2.5`). Window `sum` and `avg` likewise. (2) `sum` over integers saturates once, at the end: `i64::MAX + 1 + -1` gives `i64::MAX` (before, `i64::MAX - 1` or `i64::MAX` by order). (3) Window `sum` over integers adds them exactly; before they were added as floats, so a running total above 2^53 lost its low digits. (4) `min` / `max` of floats follow one order: `-0.0` before `0.0`, NaN after every other number (before, the first of two such values seen won). Window `min` / `max` use the same order over any values, so the minimum of keywords is the first in text order. (5) Window functions sort each partition by `:order-by` under one total order of values (numbers by numeric value, an integer before an equal float, then strings, booleans, refs, keywords, `nil`), so keywords, refs and booleans now sort (before, any two of them compared equal and kept their input order). Rows with equal `:order-by` values are peers: `rank` gives them the same rank, and `sum`, `count`, `min`, `max`, `avg` and registered window functions give every peer the value at the last peer, as SQL does; `row-number` numbers peers in the order of the other variables `:find` names. For example, with `?s` of 1.5, 5, 10, 10, `(sum ?s :over (:order-by ?s))` gave 1.5, 6.5, 16.5, 26.5, the 16.5 going to either tied row, and now gives 1.5, 6.5, 26.5, 26.5; ranking `:x :x :y :y` by `:order-by` gave ranks depending on the order the scan met them (1 2 3 4 when interleaved) and now gives 1 1 3 3. (6) In a query with aggregates, a window's argument, `:partition-by` and `:order-by` must be plain `:find` variables, or the query fails at parse time with the new `PRS-082`; before, any other variable read `nil` on every group row. The error a failing aggregate query reports is the first error, in `:find` order, of the first failing group in output order. A registered aggregate still receives its values in input order, which can depend on the plan; one whose result depends on that order may answer equivalent queries differently.

### Added

Expand Down Expand Up @@ -96,6 +97,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Answers that depend on the plan fail the tests (#596).** A test-only switch (`minigraf::__reorder`, `test-oracle` feature) turns off every place the native build reorders: the pattern selectivity sort, the `not`/`not-join` and `or`/`or-join` cost sorts and the `or` short-circuits (#250), and the engine's `HashJoin` build side. `tests/plan_independence_test.rs` runs a matrix of queries (joins, expressions, rules, `or`/`or-join`, `not`/`not-join`, keyword and ref, floats, long strings, aggregates, time travel), in memory and on file, through `execute` and a cursor, with the switch on and off, and requires the same rows. Under `test-oracle` the model-based (#385) and property (#386) tests also run every generated query with the switch off.
- **Negation bodies in both forms (#546).** A second test-only switch (`minigraf::__subplans`, `test-oracle` feature) forces the engine's `not` / `not-join` bodies to run hashed or correlated. `tests/plan_independence_test.rs` runs every query with the body forms chosen, forced hashed and forced correlated, each with reordering on and off; its matrix gains negations over floats near zero, `0.0` and `-0.0`, keywords and refs of one entity, long strings, bodies reading an imported variable they do not bind, variable-free bodies and `:db/valid-at`. `tests/negation_equality_test.rs` checks the changed answers on every executor path, in memory and after a checkpoint. The property test's rerun also forces hashed bodies. Engine unit tests cover the form the planner picks, routing, multiplicity, failing expressions in bodies, and that a hashed body is read once and a correlated one probes once per row.
- **Disjunctions (#546).** `tests/or_equality_test.rs` checks the changed `or` / `or-join` answers through `execute` and a cursor, in memory and after a checkpoint. `tests/plan_independence_test.rs` gains disjunctions (keywords meeting refs, floats near zero, long strings as keys, branches naming different outer variables or no join variable, a branch with `not`, an `or` with nothing before it, a cross product, `:db/valid-at` in a branch), run with branches chosen, forced hashed and forced correlated. The property test generates `or` and `or-join` over bound variables, with a reference evaluator that deduplicates the rows they keep. Engine unit tests run every disjunction shape with each branch form forced against the executor, and cover routing, the form the planner picks, `Distinct`, and that a hashed branch is read once.
- **Aggregates and windows (#547).** `src/query/datalog/aggregate.rs` unit tests check every accumulator over every order of small inputs (exact float sums against hand-computed values, cancellation, subnormals, signed zeros, infinities and NaN, integer saturation, errors decided at the end) and that the value order is total. `tests/aggregate_window_rules_test.rs` checks the changed answers; `tests/plan_independence_test.rs` gains aggregate, window and mixed queries over ties, keywords and refs, and floats.
- `tests/snapshot_atomicity_test.rs` races a writer against a reader: a query must see each transaction whole, whether written with `execute` or a `WriteTransaction` (#554), and must read one state for all its patterns (#555), in memory and on file. It also runs the v2.x checkpoint race from #558, which never failed on v3: copy-on-write checkpoints do not overwrite pages a published reader uses.
- **The `datalog_eval` fuzz guard counts partial bindings** (#597). The guard skipped inputs whose patterns join to more than 50,000 rows, counting only the rows after the last pattern. A `not-join` body still runs on the executor pattern by pattern in text order, so a 120-pattern body built hundreds of thousands of partial bindings before its last pattern matched nothing, and ran past the 60-second timeout. The guard now counts the bindings after every pattern. On the `v3` corpus (4,783 inputs) coverage drops by 7 of 9,719 edges. The input is added as a seed, as is the `btree_page` input of the cursor fix above.

Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ cargo run < demos/demo_negation.txt
- `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`; `resolve_pseudo_sources` names each per-fact pseudo-attribute's source (nearest earlier same-entity pattern, lexical scope) and its hidden metadata variable `?#<n>.<attr>`, else PRS-081 (#594)
- `executor.rs`: Query executor — temporal filter (tx-time → net-assert → valid-time), not/not-join post-filter
- `matcher.rs`: Pattern matching with variable unification; `edn_to_value`, `edn_to_entity_id`
- `aggregate.rs`: `Agg` (one aggregate of one group) and `Running` (a window's running value), shared by the executor and the engine; exact sums (`i128` integers saturated once, fixed-point floats rounded once), `value_order` (one total order of values for window sorts, peers, `rank` and `min`/`max`), `named_vars` (a window's tie-break variables) (#547)
- `evaluator.rs`: `RecursiveEvaluator` (semi-naive), `StratifiedEvaluator` (repeats a stratum whose rules recurse through a `not`/`or` rule until it derives nothing new, #633), `evaluate_not_join`
- `stratification.rs`: `DependencyGraph`, `stratify()` — negative edges + cycle detection; a positive edge to a rule with `not`/`not-join`/`or`/`or-join` also raises the caller, off a cycle (#628)
- `rules.rs`: `RuleRegistry` — thread-safe rule management
Expand Down Expand Up @@ -204,7 +205,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1606 tests** (1596 passing, 10 ignored; unit + integration + doc).
**1616 tests** (1606 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1606 tests
cargo test # run 1616 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
17 changes: 17 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ with no `CodedError` anywhere in its chain.
| PRS-079 | Rule rejects unexpected trailing argument(s) | Parser |
| PRS-080 | :find variable not bound in :where | Parser |
| PRS-081 | Pseudo-attribute has no source pattern | Parser |
| PRS-082 | Window reads a non-grouping variable | Parser |
| QRY-001 | Invalid entity | Query Execution |
| QRY-002 | Attribute must be a keyword | Query Execution |
| QRY-003 | Cannot transact a pseudo-attribute | Query Execution |
Expand Down Expand Up @@ -1596,6 +1597,22 @@ $<4097-char-slot-name>
```
*Move `[?e :person/name "Alice"]` before `[?e :db/tx-count ?t]`*

### PRS-082 Window reads a non-grouping variable

**Error text**: `{} in {} must be a plain :find variable when the query has aggregates`

**Cause**: In a query whose `:find` holds an aggregate, window functions run over the groups the aggregation produces, after it. Each group row holds only the plain `:find` variables and the aggregate results, so a window's argument, `:partition-by` and `:order-by` must each be a plain `:find` variable. `:with` variables do not count: they do not group when `:find` holds a window. Before v3.0.0 such a variable read `nil` on every group row, so the window ordered or summed nothing.

**Resolution**:
- Add the variable to `:find` as a plain variable, so it groups.
- Or run the window in a query without aggregates.

**Example**:
```datalog
(query [:find ?dept (sum ?salary) (rank :over (:order-by ?salary)) :where [?e :dept ?dept] [?e :salary ?salary]])
```
*`?salary` is aggregated, not a group: order by `?dept`, or drop `(sum ?salary)`*

---

## QRY — Query Execution Errors
Expand Down
5 changes: 3 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

Loading
Loading