diff --git a/CHANGELOG.md b/CHANGELOG.md index 32a586bb96..73ae944049 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **Ordinary average** (#777): `(avg ?v)` no longer raises PRS-027. Grouped and ungrouped queries use the same numeric population and arithmetic as window `avg`: skip nil and nonnumeric values, sum exactly, round the total once to Float, then divide by the numeric count. Integer totals are not saturated. Results are Float, or nil for an existing group without numbers; no matching rows yields no rows. The total can overflow before division, and division rounds separately. Existing window behavior is unchanged. Retained exact-sum allocations for ordinary `sum` and `avg` are now charged to the query memory limit. + +### Documentation + +- Retire tracker #383 in favor of the v3.0.0 milestone and its release gates. Other ordinary/window value-handling differences are tracked in #789. + ### Performance - **Resolve expression keyword literals once per operator** (#774). `=` and `!=` reuse a literal keyword's entity UUID across input rows, including nested expressions, rule bodies and prepared slots. The cache contains only literals from that expression and is discarded with the operator; dynamic operands retain the existing comparison semantics. No API, dependency or file-format change. See the [measurements](docs/benchmarks/774-expression-keywords.md). diff --git a/CLAUDE.md b/CLAUDE.md index 6a4dc20228..e842cff671 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -218,13 +218,15 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are ## Test Coverage -**2006 tests** (1989 passing, 17 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551). +**2018 tests** (2001 passing, 17 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in `tests/answers/` (#551). See `docs/TEST_COVERAGE.md` for the full per-file breakdown. The #760 deterministic storage suite adds three native unit tests in a separate `--cfg minigraf_shuttle` build, outside the default count. Run only `graph::storage::deterministic::` with that cfg; ordinary unit tests do not enter its scheduler. `deterministic-schedules.yml` runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See `docs/testing/deterministic-schedules.md` for scope, controls and findings. **Testing conventions** — see the Testing Conventions section below before writing any tests. +On v3, ordinary `avg` skips nil/nonnumeric values and divides the exact rounded total by the numeric count, matching window `avg` (#777). It returns Float, or nil for an existing group with no numbers; no matching bindings yields no rows. Other aggregate consistency work is #789; ordered-avg optimization remains #781. + ## Current Maintainer Context On `v3`, expression `=` / `!=` resolves literal keywords once per operator (#774), including nested expressions and substituted prepared slots. Dynamic values do not populate the cache. This is a performance-only change, with no v2.x backport; see `docs/benchmarks/774-expression-keywords.md`. @@ -234,7 +236,7 @@ On `v3`, expression `=` / `!=` resolves literal keywords once per operator (#774 Current plan (see `ROADMAP.md` and the milestones): - **In-memory ancestors (#723)**: eligible positive three-pattern delta runs retain an entity-first alternative with the same `Old`/`Delta`/`Full` ranges and separately planned head slots. Each round can select it when the committed dictionary is empty, the source relation has at most 128 rows and no stored records, and a capped AEVT count favors entity probes over building an uncached pending value map. Populated committed generations keep AVET probes. No additional write-time index. - **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release. -- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, `lag`/`lead` #182, sliding row frames #183, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`. +- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, optional `:distinct` output #493, `lag`/`lead` #182, sliding row frames #183, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1)) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`. - **Input order (#641)** targets v3.0.0 in three slices: committed scalar Sort omission, one-group count/count-distinct and partition-at-a-time windows are implemented in the dependent PRs for `v3`. The reviewed design, fallback scope and acceptance gates are in `docs/superpowers/specs/2026-10-28-641-input-order-design.md`. - **Scalar key decoder proofs (#693)** target v3.0.0 before the release cut. The pure decoder refactor preserves key bytes and INT-049 behavior; Kani covers round trips and bounded malformed inputs. The approved design and CI limits are in `docs/superpowers/specs/2026-10-29-693-decoder-proofs.md`. - **v3.next** (the release after v3.0.0) holds new features (rule persistence #241; OPFS `BrowserDb` #441). diff --git a/PHILOSOPHY.md b/PHILOSOPHY.md index d384c5afd5..6cf72e6485 100644 --- a/PHILOSOPHY.md +++ b/PHILOSOPHY.md @@ -403,7 +403,7 @@ When evaluating a feature or design choice, ask: These are goals, not a description of today. You'll know Minigraf has succeeded when: 1. **Ubiquity**: Developers say "just use Minigraf" for embedded graph storage -2. **Trust**: Known for never losing data, crash-safe, reliable. Not yet met: v2.x has a known data-integrity bug (#371), and the work to meet this goal is tracked in #383. See the [known issues](https://github.com/project-minigraf/minigraf/issues?q=is%3Aissue+is%3Aopen+label%3Aknown-issue). +2. **Trust**: Known for never losing data, crash-safe, reliable. Not yet met: v2.x has a known data-integrity bug (#371), and the work to meet this goal is tracked in the [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1). See the [known issues](https://github.com/project-minigraf/minigraf/issues?q=is%3Aissue+is%3Aopen+label%3Aknown-issue). 3. **Simplicity**: New users are productive in under 5 minutes 4. **Size**: Core binary within its ~1.2MB budget, minimal dependencies 5. **Portability**: Runs everywhere from Raspberry Pi to browsers diff --git a/README.md b/README.md index a5bb68eb1c..0f6357c16d 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ Minigraf is a **single-file embedded graph database** that lets you: For example, `(lag ?salary :over (:partition-by ?dept :order-by ?hire-date))` reads the preceding employee's salary within each department; the first row gets `nil`. `lead` reads the following row. +The unreleased v3.0.0 branch supports ordinary `(avg ?score)` (#777). It skips nil and nonnumeric values, counts only numbers, and returns a Float average or nil for an existing group without numbers. No matching rows produces no result row. It uses the same exact-total-then-divide arithmetic as window `avg`; a finite total can overflow before division. V2 requires `:over` for `avg`. + The unreleased v3.0.0 branch adds bounded sliding windows such as `(avg ?salary :over (:partition-by ?dept :order-by ?hire-date :rows 2 preceding))` (#183). On v3.0.0, an ascending value-only query such as `[:find ?v :where [?e :score ?v] :order-by [?v] :limit 10]` can read directly in index order and stop early when all committed `:score` history has one value kind (integer, float, or Boolean) and its source range has no pending writes. A query whose full order cannot be proved uses the existing sort (#641). Built-in `count` and `count-distinct` can also release completed groups when one committed integer/Boolean/float key is proved ordered (including mixed integer/float groups), with no `:with`, windows, callbacks or `collect-set`. Memory then follows the active group and batches; distinct members within one group still grow. Other aggregates keep full-input validation. When every built-in window uses the same proved nonconstant partition key, the engine can also finish and release one partition at a time. Its memory follows the largest partition plus bounded batches; one growing partition still grows. Unpartitioned windows, incompatible keys, aggregate queries, callbacks and `collect-set` keep full-input evaluation. A surrounding sort or distinct operation can still require more memory. diff --git a/ROADMAP.md b/ROADMAP.md index 3a2a0e6838..2dd8392dcf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,9 +1,11 @@ # Minigraf Roadmap -> Planned work for Minigraf, a bi-temporal Datalog database. v2.x has known data-integrity issues ([pinned list](https://github.com/project-minigraf/minigraf/issues/421)); v3.0.0 fixes them (tracker #383). +> Planned work for Minigraf, a bi-temporal Datalog database. v2.x has known data-integrity issues ([pinned list](https://github.com/project-minigraf/minigraf/issues/421)); v3.0.0 fixes them ([v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1)). **Philosophy**: Embedded graph memory for agents, mobile, and the browser — built on the SQLite approach: be boring, be reliable, be embeddable. +Ordinary `avg` is implemented for v3.0.0 (#777), matching window numeric filtering. Consistent handling for the other ordinary aggregates remains a v3.0.0 release requirement (#789). + Completed releases and their implementation details live in the [CHANGELOG](CHANGELOG.md). This document tracks planned, exploratory, and other future work. --- @@ -47,7 +49,7 @@ Further v2.x releases are data-integrity and security patches only. ## v2.0.2 — Final Planned v2.x Release (released 2026-09-27) -**The v2.x line (`main`) ends with a patch.** File format stays v7 and the public API does not change. All new feature work has moved to v3.x. Tracker: #383. Details are in the [CHANGELOG](CHANGELOG.md). +**The v2.x line (`main`) ends with a patch.** File format stays v7 and the public API does not change. All new feature work has moved to v3.x. Work list: [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1). Details are in the [CHANGELOG](CHANGELOG.md). Shipped: @@ -222,7 +224,7 @@ When evaluating features, ask: - v2.x maintenance on `main`: data-integrity and security fixes only - v3.0.0 (format v8 and data integrity) on the `v3` branch, then v3.next features -- Production readiness for high-stakes deployments, tracked in #383 +- Production readiness for high-stakes deployments, tracked in the [v3.0.0 milestone](https://github.com/project-minigraf/minigraf/milestone/1) - Ecosystem work tracked in [`minigraf-examples`](https://github.com/project-minigraf/minigraf-examples) - Developer tools tracked in `minigraf-inspector` and `minigraf-visualizer` diff --git a/docs/ERROR_REFERENCE.md b/docs/ERROR_REFERENCE.md index e87caf3d5c..0829449424 100644 --- a/docs/ERROR_REFERENCE.md +++ b/docs/ERROR_REFERENCE.md @@ -736,16 +736,17 @@ See the [Datalog Reference](https://project-minigraf.github.io/minigraf-docs/lat **Error text**: `'{}' is a window function and requires an ':over (...)' clause` -**Cause**: Window functions (`rank`, `row-number`, `dense-rank`, `ntile`, `percent-rank`, `cume-dist`) must be accompanied by an `:over` clause specifying ordering and/or partitioning. +**Cause**: Window functions `rank` and `row-number` must be accompanied by an `:over` clause specifying ordering and/or partitioning. **Resolution**: - Add an `:over` clause: `(rank :over (:order-by ?score))`. +- Since v3.0.0, `avg` also works as an ordinary aggregate without `:over` (#777); v2 requires the window form. - See the [Datalog Reference — Window Functions](https://project-minigraf.github.io/minigraf-docs/latest/datalog-reference/#window-functions). **Example**: ```datalog -(query {:find [?e (rank)] - :where [[?e :score ?score]]}) +(query [:find ?e (rank ?score) + :where [?e :score ?score]]) ``` *Missing `:over`; use `(rank :over (:order-by ?score))`* @@ -4046,7 +4047,7 @@ these two arities. **Error text**: `unknown aggregate function: '{}'` **Cause**: A `:find` or `:with` clause referenced an aggregate function -name that is neither a built-in (`count`, `sum`, `avg`, `min`, `max`, etc.) +name that is neither a built-in (`count`, `count-distinct`, `sum`, `sum-distinct`, `avg`, `min`, `max`) nor a user-defined function registered via `db.register_aggregate()`. **Resolution**: diff --git a/docs/TEST_COVERAGE.md b/docs/TEST_COVERAGE.md index f2ec7ae934..f07f763a54 100644 --- a/docs/TEST_COVERAGE.md +++ b/docs/TEST_COVERAGE.md @@ -4,9 +4,11 @@ | | Default features | With `--features test-hooks` | |---|---|---| -| **Passing** | 1,989 | 2,022 | +| **Passing** | 2,001 | 2,035 | | **Ignored** | 17 | 18 | -| **Total** | 2,006 | 2,040 | +| **Total** | 2,018 | 2,053 | + +#777 adds 12 default-feature tests (three accumulator tests, one engine memory test and eight integration tests) and one additional `test-hooks` plan matrix. The focused 13-test run passes. Strict native/browser/WASI clippy passes, and the rebased release binary is 1,263,144 bytes against the unchanged 1,263,616-byte cap. #774 adds one default-feature unit test (also present with `test-hooks`). On 2026-10-29, after rebasing onto #723, its full default suite passed 1,989 tests @@ -43,6 +45,8 @@ Also outside those counts: three native unit tests in the separate #760 `minigra ## Covered Areas +- **Ordinary average** (#777; `tests/ordinary_avg_test.rs`, `aggregate/tests.rs`, `engine/tests.rs`): grouped and ungrouped averages, every nonnumeric value kind skipped, nil versus no matching rows, Float results, exact unsaturated totals, signed zero/subnormal underflow, infinities/NaNs, input permutations, prepared execution/cursors, batches, recursive rules, transaction reads, checkpoint/reopen and alternative plans. Default and bounded windows preserve skipped rows. Sum/avg exact-sum heap state is charged once before allocation, transferred on finish, and released on errors, rewind and close. Existing neighboring aggregate errors and dropped groups are preserved. + - **Expression keyword literals** (#774): a test-only conversion counter verifies that increasing input from 8 to 2,048 rows does not increase literal resolution work, across `=` / `!=`, both operand orders, nested expressions, bindings, negation, disjunction and rule bodies. Existing join-equality tests also cover changing prepared keyword slots across executions and unchanged keyword output, in memory, after checkpoint and after reopen. The envelope harness checks all eight new expression/control shapes on regular and irregular fixtures. - **In-memory ancestor join ordering** (#723; four tests): a 1,999-edge tree returns three ancestors within a 128 KiB query budget, also after staged edge replacement; cursor close releases its metered state. Reordered and text-order plans return identical bags across keyword/ref edges, cycles, multi-valued edges, retractions, transaction/valid-time filters, empty and unbound queries, in memory and with checkpointed facts. A direct comparison against the original delta plan protects keyword/ref output representation in rule heads. The bounded cost count excludes writes after a captured snapshot, includes staged records, respects the cap, and excludes prefix-neighbor attributes. diff --git a/docs/superpowers/plans/2026-10-29-777-ordinary-avg.md b/docs/superpowers/plans/2026-10-29-777-ordinary-avg.md new file mode 100644 index 0000000000..3a195e342f --- /dev/null +++ b/docs/superpowers/plans/2026-10-29-777-ordinary-avg.md @@ -0,0 +1,108 @@ +# #777 implementation plan + +Status: Claude-approved (second review, 2026-10-29). Maintainer chose window-compatible avg on +2026-10-29 and authorized avg-only implementation; other aggregates are #789. +Spec: [ordinary avg design](../specs/2026-10-29-777-ordinary-avg-design.md). + +## Baseline and independent main docs + +Use .worktrees/777-avg from v3 and separate main/docs-site worktrees. Run +baseline cargo test on the idle bench machine. Commit reviewed spec/plan first. +Retain bench scratch only while reused; remove it at completion. + +Main documentation is independent: correct ERROR_REFERENCE PRS-021, PRS-024, +PRS-027 and INT-029. List the six ordinary built-ins and explain avg requires +:over. Replace unsupported PRS-027 names and use (rank ?score), which actually +raises PRS-027. Audit README/ROADMAP/CLAUDE/TEST_COVERAGE; no v2 runtime changes +or version/tag bump. Review this docs PR independently, but do not merge it +without authorization. + +## Red tests and runtime support + +1. In src/query/datalog/aggregate/tests.rs, replace count_skips_nil's obsolete + avg INT-029 assertion. Add tests for Float results, nil/no-number populations, + every nonnumeric variant, mixed numbers, integers above 2^53, unsaturated + integer totals, NaN/infinities, finite-total overflow, signed zeros and both + signs of subnormal underflow. Permute small cancellation populations against + explicit expected results. Avoid Debug assertion messages. +2. Add parser acceptance and public 10/20 => Float(15) reproduction tests in + tests/ordinary_avg_test.rs. Run red before production edits. +3. Add Agg::Avg(Sum) with new/step/finish arms in aggregate.rs. Remove avg from + parser.rs WINDOW_ONLY; preserve all existing window paths and arithmetic. + Run tests green. +4. Replace tests/grammar/invalid/semantic/window_only_func_without_over.edn + with (rank ?salary). Add grouped/ungrouped valid avg fixtures. + tests/error_codes_prs_test.rs already covers rank; add row-number rejection. + Audit found no avg/PRS-027-specific rule in docs/GRAMMAR.md, + parser/verifier.rs, grammar.pest or fuzz target code. Structural grammar + accepts generic names; keep the SEMANTIC allowlist because genuine window-only + functions still use PRS-027. Run grammar_conformance and grammar_doc tests. + Existing fuzz corpus inputs are not expected-result assertions; retain them. + +## Retained exact-sum allocation + +5. Add failing tests in src/query/engine/tests.rs for the first finite float's + retained Exact charge in sum/avg, no repeated charge, and QRY-012 before + allocation under a tight limit. Cover successful finish, failure, close and + rewind release. Add private Agg heap-growth and retained-byte helpers in + aggregate.rs. In engine/aggregate.rs charge growth before stepping and + transfer retained bytes alongside the existing sum-distinct member charge + into the sorted finishing map. Preserve distinct accounting and all existing + sum value/error semantics. Run memory/aggregate tests green. + +## Execution coverage + +6. Extend tests/ordinary_avg_test.rs for grouped/ungrouped/:with multiplicity, + all-nil/all-nonnumeric groups, empty queries including avg next to count, + neighboring sum errors and min group-drop precedence, prepared binds through + execute/query, multi-batch cursors, recursive rules, write-transaction reads, + pending/checkpointed/reopened data. +7. Add test-hooks cases using __reorder, __subplans (Choose/Hash/Correlated) and + __input_order, through execute and cursors. Include joins/negation so choices + affect real operators. Assert explicit expected answers. Keep avg buffered; + #781 optimizer expansion is out of scope. Existing engine tests compare + frozen answers in tests/answers, not a live reference executor; run them + unchanged. Compare ordinary avg with the final cumulative window over the + same population, and preserve bounded-frame row positions for skipped values. + Commit implementation/tests after focused verification. + +## Docs and verification + +8. Sync ERROR_REFERENCE, README, ROADMAP, CHANGELOG, CLAUDE and TEST_COVERAGE + using measured counts. Add an unreleased Changed entry: (avg ?v) no longer + raises PRS-027. Document skipping/count distinction, nil versus no rows, + Float results and existing mean rounding/overflow. Correct PRS-027; QRY-010's + existing avg claim becomes true. Current direction references use the + v3.0.0 milestone after retiring #383; historical records remain. +9. Docs site: version-scope the compatibility table, aggregate list, semantic + notes and later feature summary in content/datalog-reference.md. Preserve + v2's six built-ins. Audit architecture/comparison/use-cases and edit only + affected claims. site.toml already has unreleased v3.0.0. Build with + cargo run --release -- build --source-repo ; inspect v2/v3 output + and link checks. Commit docs after validation. + +Run on the bench machine (independent jobs may use separate target dirs): + +~~~sh +cargo test +cargo test --features test-hooks --test ordinary_avg_test --test plan_independence_test --test input_order_test +cargo fmt -- --check +cargo clippy --all-features -- -D warnings +cargo check --target wasm32-unknown-unknown --features browser +cargo check --target wasm32-wasip1 +cargo build --release +test "$(stat -c %s target/release/minigraf)" -le 1263616 +~~~ + +Run applicable wasm tests with installed runners; otherwise report that local +limitation and verify corresponding wasm.yml PR CI. Preserve the 1,234 KiB +budget. Inspect all results, fix failures and rerun affected checks. + +## Review and delivery + +10. Obtain Claude final review of v3, main-docs and docs-site diffs. Resolve + verified findings and rerun affected checks before opening separate PRs. + Monitor CI to green. No merge/release without explicit authorization. + Keep #777 open until its branch and docs-site scopes land. The #781 + handoff is infallible avg over decoded values, skipping nonnumbers, with + retained exact-sum memory charged; ordered eligibility needs its own proof. diff --git a/docs/superpowers/specs/2026-10-29-777-ordinary-avg-design.md b/docs/superpowers/specs/2026-10-29-777-ordinary-avg-design.md new file mode 100644 index 0000000000..36b3d4696e --- /dev/null +++ b/docs/superpowers/specs/2026-10-29-777-ordinary-avg-design.md @@ -0,0 +1,133 @@ +# #777: ordinary average and accurate v2 documentation + +Date: 2026-10-29. Target: v3.0.0, matching #777's milestone. +Status: Claude-approved (second review, 2026-10-29); maintainer selected matching window semantics +on 2026-10-29. Implementation is on feat/777-avg. +Issue: https://github.com/project-minigraf/minigraf/issues/777 +Base: v3, rebased onto `c2e862fe8` (#774); v2 documentation is a separate main-based change. + +## Scope and philosophy + +Expose `(avg ?v)` through v3's existing shared aggregate path. Correct v2's +misleading documentation without changing its parser or runtime. This follows +PHILOSOPHY.md's reliability, simplicity, portability and documentation principles: +no dependencies, configuration, API or storage/WAL format changes. + +## Ordinary avg contract (v3.0.0) + +- Accept one variable, using the existing ordinary aggregate syntax and binding + checks. Plain `:find` variables and `:with` retain their existing grouping + behavior. Feed avg precisely the same rows an existing aggregate receives; + do not add distinctness or change row multiplicity. +- Take Integer and Float values; skip nil and every nonnumeric type (including + strings, booleans, keywords, refs and sets). Count only numbers. This follows + the existing window avg. The maintainer explicitly requires consistency + between ordinary and window forms; this supersedes the briefly selected + nonnumeric-rejection option. Other functions' changes belong to #789, not this + implementation. Numeric count differs from count, which includes nonnumbers. +- A nonempty numeric population always yields Float, including integer-only + populations and exact integral results. An existing group with no numeric + values yields nil and is not dropped. With no matching bindings, grouped and + ungrouped queries containing avg yield no rows, preserving the existing + non-count aggregate empty-input rule (`engine/plan.rs`'s `zero_row`). This + includes `(count ?v) (avg ?v)` and queries with `:with`. Distinguish this from + a constructed empty accumulator, whose `mean` is nil. +- Use the existing `Sum::mean` unchanged: accumulate integers and finite floats + exactly, round the total once to nearest f64 (ties to even), then divide by + the numeric count converted to f64. Integer totals are not saturated before + division. This is an order-independent floating-point mean, not a promise of + a correctly rounded exact rational mean. The final division rounds separately. +- In particular, a finite total exceeding f64 range becomes signed infinity + before division, even when the mathematical mean would be finite (e.g. two + f64::MAX values). Preserve that existing window behavior and document it. +- Any numeric NaN or both infinity signs produce NaN. A single infinity sign + produces that infinity. All numeric inputs equal to -0.0 produce -0.0; exact + cancellation or a zero total with another numeric value produces +0.0. + A nonzero negative total can underflow to -0.0 on division (e.g. the smallest + negative subnormal plus two +0.0 values, divided by three). Test both signs + of underflow, zero signs via float bits, and NaNs with is_nan. +- Execution order and legal plan choice cannot change avg's result. This covers + mixed integer/float cancellation and integer values above 2^53. Storage's + existing normalization of values is unchanged. +- Other aggregate failures and group dropping keep existing find-order/group + precedence. Adding avg must not suppress a neighboring aggregate's error. + +## Implementation boundary + +Add `Agg::Avg(Sum)`; construct it for Builtin::Avg, step it with `Sum::add` +(ignoring nonnumeric values), and finish it with `Sum::mean`. The engine's +`Acc::Values` already delegates to Agg. Keep Sum's arithmetic, Running, +SlidingSum, and window evaluation unchanged. Do not generalize the ordered +Aggregate optimizer. + +Account for the retained Box (36 u64 limbs) used by Sum-backed ordinary +accumulators. Add private Agg helpers for the additional heap bytes a step +would allocate and the heap bytes already retained. Before the first finite +float is stepped into Sum or Avg, charge that allocation once through Held; +integers, NaN, infinities and subsequent floats add no such charge. Transfer +this charge with the accumulator when building the sorted finishing map, +alongside existing sum-distinct member charges, and release it on finish, +failure, rewind and close. No arithmetic or input-type behavior of other +aggregates changes. This closes the same pre-existing Sum accounting gap rather +than adding an uncharged allocation for Avg. Temporary arithmetic scratch is +outside the existing retained-state accounting model. Test exact charge deltas +and a QRY-012 boundary on the first float, including cleanup. + +Remove avg from parser WINDOW_ONLY, preserving rank/row-number restrictions +and current lag/lead restrictions. Replace the invalid avg fixture with a +genuinely window-only expression, and add valid ordinary avg fixtures. The +structural grammar already accepts generic aggregate names; do not specialize it. + +Prepared queries, cursors, rule queries and write-transaction queries must use +the same semantics. Aggregate/window combinations retain PRS-082 restrictions. + +## Required evidence + +Before production edits, add failing tests for parser acceptance, shared Agg, +and the public API reproduction (10 and 20 => Float(15)). Test the contract's +empty/nil/nonnumeric, mixed, large integer, special float and zero cases. Run +all permutations of small adversarial cancellation populations against explicit +expected values; include a rounding-sensitive integer case and finite-sum +overflow. Preserve and run existing window tests, including sliding frames. + +Integration coverage must include grouped/ungrouped/:with multiplicity, +coexisting aggregates and their errors, prepared bind execution, multi-batch +cursor reads, rules, and legal plan alternatives on pending versus checkpointed +data. Assert semantic outcomes, not just agreement between two implementations. +Run parser fixture/grammar checks and existing frozen-answer engine tests +(tests/answers/; the old reference executor was removed in #551). New avg tests +use explicit expectations and plan comparisons, not generated snapshots. +Run the full default test suite, wasm feature checks, fmt and the repository's +clippy gate on the bench machine; inspect binary-size impact without changing +the budget. Capture baseline and final results on exact revisions. + +## Documentation and delivery + +1. A main-based documentation PR removes ordinary avg claims from PRS-021, + PRS-024, INT-029 and any other current documentation. Name the six ordinary + built-ins: count, count-distinct, sum, sum-distinct, min, max. State that v2 + avg requires :over. Correct PRS-027's unsupported function list and use a + real PRS-027 example (rank ?score); bare (rank) gives PRS-024. + No runtime changes and no new release/version tag. +2. A v3 PR implements ordinary avg, tests it, and synchronizes + docs/ERROR_REFERENCE.md, README.md, ROADMAP.md, CHANGELOG.md, CLAUDE.md and + docs/TEST_COVERAGE.md. Record the feature in unreleased v3.0.0 only. Audit + related current references; do not rewrite historical release claims. + Fix v3 PRS-027 to name rank/row-number only; QRY-010's existing avg list + becomes accurate. Explicitly record that ordinary avg no longer raises + PRS-027. Use the v3.0.0 milestone in current release-direction references + after retiring #383; preserve historical records. +3. A docs-site PR scopes compatibility tables and aggregation reference with + @until v3.0.0 / @since v3.0.0; preserve v2's six built-ins. Document ordinary + avg type, skipping, empty and rounding behavior. Audit architecture, + comparison and use-cases for affected claims; update only where relevant. + Check site.toml's existing v3 entry; no invented release or version bump. +4. #781 receives the reviewed contract: avg is an infallible, fixed-state + aggregate over decoded values, skipping nonnumbers. Its ordered eligibility + still requires #781's pipeline/order/memory analysis. No optimizer expansion + in #777. Keep the issue open until both branch scopes and site docs land. + +Claude spec and implementation-plan reviews precede implementation. The +maintainer directed the avg-only implementation with matching window handling; +#789 tracks other functions as a v3.0.0 requirement. Claude reviews the final +diff before PRs are opened. No merge/release without explicit authorization. diff --git a/src/query/datalog/aggregate.rs b/src/query/datalog/aggregate.rs index b578a7caf3..c0ace7c20c 100644 --- a/src/query/datalog/aggregate.rs +++ b/src/query/datalog/aggregate.rs @@ -539,6 +539,8 @@ pub(crate) enum Agg { Count(i64), /// The sum, and the type of the first value that is not a number. Sum(Sum, Option<&'static str>), + /// The mean of numeric values, skipping nil and nonnumbers like window avg. + Avg(Sum), Min(Extreme), Max(Extreme), CountDistinct(BTreeSet), @@ -557,8 +559,7 @@ impl Agg { AggImpl::Builtin(b) => match b { Builtin::Count => Agg::Count(0), Builtin::Sum => Agg::Sum(Sum::default(), None), - // `avg` is a window function only (`PRS-027`). - Builtin::Avg => return Err(err_coded!(ErrorCode::Int029, func)), + Builtin::Avg => Agg::Avg(Sum::default()), Builtin::Min => Agg::Min(Extreme::default()), Builtin::Max => Agg::Max(Extreme::default()), Builtin::CountDistinct => Agg::CountDistinct(BTreeSet::new()), @@ -581,6 +582,26 @@ impl Agg { } } + /// Retained exact-sum allocation needed before stepping this value (#777). + pub(crate) fn exact_heap_growth(&self, v: &Value) -> usize { + match self { + Agg::Sum(s, _) | Agg::Avg(s) + if s.floats.is_none() && matches!(v, Value::Float(f) if f.is_finite()) => + { + size_of::() + } + _ => 0, + } + } + + /// Exact-sum heap bytes that move with this accumulator when finishing. + pub(crate) fn exact_heap_bytes(&self) -> usize { + match self { + Agg::Sum(s, _) | Agg::Avg(s) if s.floats.is_some() => size_of::(), + _ => 0, + } + } + /// Take one value; `nil` is skipped by every aggregate. Only a user /// aggregate fails (`QRY-014`). pub(crate) fn step(&mut self, v: &Value) -> Result<()> { @@ -594,6 +615,9 @@ impl Agg { *bad = Some(type_name(v)); } } + Agg::Avg(s) => { + s.add(v); + } Agg::Min(e) => e.add(v, false), Agg::Max(e) => e.add(v, true), Agg::CountDistinct(set) | Agg::SumDistinct(set) => { @@ -611,6 +635,7 @@ impl Agg { Agg::Count(n) => Value::Integer(n), Agg::Sum(_, Some(bad)) => return Err(err_coded!(ErrorCode::Int041, bad)), Agg::Sum(s, None) => s.value(), + Agg::Avg(s) => s.mean(), Agg::Min(e) | Agg::Max(e) => { let Some(best) = e.best else { return Ok(None); diff --git a/src/query/datalog/aggregate/tests.rs b/src/query/datalog/aggregate/tests.rs index 6d7b9b895c..1ab1b00eb0 100644 --- a/src/query/datalog/aggregate/tests.rs +++ b/src/query/datalog/aggregate/tests.rs @@ -299,10 +299,81 @@ fn count_skips_nil() { .flatten(), Some(i(2)) ); - assert!( - code(finish("avg", &[i(1)])) == "INT-029", - "avg is a window function" - ); +} + +fn average(vals: &[Value]) -> Value { + finish("avg", vals).expect("average").expect("keeps group") +} + +#[test] +fn ordinary_avg_skips_every_nonnumeric_kind() { + let ignored = vec![ + Value::Null, + Value::String("unknown".into()), + Value::Boolean(true), + kw(":unknown"), + Value::Ref(uuid::Uuid::nil()), + Value::Set(crate::graph::types::ValueSet::new([i(100)]).unwrap()), + ]; + assert_eq!(average(&[]), Value::Null); + assert_eq!(average(&ignored), Value::Null); + let mut vals = ignored; + vals.extend([i(10), f(20.0)]); + for p in [vals.clone(), vals.into_iter().rev().collect()] { + assert_eq!(average(&p), f(15.0)); + } + assert_eq!(average(&[i(10), i(20)]), f(15.0)); +} + +#[test] +fn ordinary_avg_uses_exact_unsaturated_totals_in_every_order() { + for (vals, expected) in [ + (vec![f(1e16), i(1), f(-1e16), f(1.0), f(0.5)], 0.5), + ( + vec![i(9_007_199_254_740_993), i(-9_007_199_254_740_992), i(2)], + 1.0, + ), + (vec![i(i64::MAX), i(i64::MAX)], i64::MAX as f64), + // Round the exact sum once, then divide, as window avg does. + ( + vec![i(9_007_199_254_740_992), i(1)], + 4_503_599_627_370_496.0, + ), + ] { + for vals in permutations(&vals) { + assert_eq!(average(&vals), f(expected)); + } + } +} + +#[test] +fn ordinary_avg_special_floats_and_underflow_match_window_mean() { + for (vals, expected) in [ + (vec![f(-0.0), f(-0.0)], -0.0f64), + (vec![f(-0.0), i(0)], 0.0), + (vec![f(-1.0), f(1.0)], 0.0), + (vec![f(-f64::from_bits(1)), f(0.0), f(0.0)], -0.0), + (vec![f(f64::from_bits(1)), f(0.0), f(0.0)], 0.0), + (vec![f(f64::MAX), f(f64::MAX)], f64::INFINITY), + (vec![f(-f64::MAX), f(-f64::MAX)], f64::NEG_INFINITY), + (vec![f(f64::INFINITY), i(1)], f64::INFINITY), + (vec![f(f64::NEG_INFINITY), i(1)], f64::NEG_INFINITY), + ] { + for vals in permutations(&vals) { + let Value::Float(actual) = average(&vals) else { + panic!("float average"); + }; + assert_eq!(actual.to_bits(), expected.to_bits()); + } + } + for vals in [ + vec![f(f64::NAN), i(1)], + vec![f(f64::INFINITY), f(f64::NEG_INFINITY)], + ] { + for vals in permutations(&vals) { + assert!(matches!(average(&vals), Value::Float(n) if n.is_nan())); + } + } } #[test] diff --git a/src/query/datalog/parser.rs b/src/query/datalog/parser.rs index 104f0402f2..a6322bf92c 100644 --- a/src/query/datalog/parser.rs +++ b/src/query/datalog/parser.rs @@ -694,7 +694,7 @@ fn parse_aggregate(elems: &[EdnValue]) -> Result { } }; - const WINDOW_ONLY: &[&str] = &["avg", "rank", "row-number"]; + const WINDOW_ONLY: &[&str] = &["rank", "row-number"]; if WINDOW_ONLY.contains(&func_name.as_str()) { bail_coded!(ErrorCode::Prs027, func_name); diff --git a/src/query/engine/aggregate.rs b/src/query/engine/aggregate.rs index 13b95c4f22..bb72ad0408 100644 --- a/src/query/engine/aggregate.rs +++ b/src/query/engine/aggregate.rs @@ -74,6 +74,10 @@ impl Acc { } Acc::Values(agg) => { let v = value(cell, ctx)?; + let growth = agg.exact_heap_growth(&v); + if growth > 0 { + held.charge(growth)?; + } if agg.would_add(&v) { held.charge(value_bytes(&v) + HASH_ENTRY_OVERHEAD)?; } @@ -83,15 +87,15 @@ impl Acc { Ok(()) } - /// The accumulator as an [`Agg`], whose `finish` gives its result. - /// The bytes `step` charged for the members of a set that stays in the - /// `Agg`: `sum-distinct`'s. A `count-distinct` set ends as its count. - fn member_bytes(&self) -> usize { + /// Heap bytes that stay in the `Agg`: sum-distinct members and exact sums. + /// A count-distinct set instead ends as its count. + fn retained_bytes(&self) -> usize { match self { Acc::Values(Agg::SumDistinct(set)) => set .iter() .map(|v| value_bytes(v) + HASH_ENTRY_OVERHEAD) .sum(), + Acc::Values(agg) => agg.exact_heap_bytes(), _ => 0, } } @@ -369,15 +373,14 @@ impl Aggregate { .iter() .map(|c| value(c, ctx)) .collect::>>()?; - let members: usize = group.accs.iter().map(Acc::member_bytes).sum(); + let retained: usize = group.accs.iter().map(Acc::retained_bytes).sum(); let n = 2 * (size_of::>() + values.iter().map(value_bytes).sum::()) + size_of::>() + group.accs.len() * size_of::() - + members + + retained + HASH_ENTRY_OVERHEAD; - // The members move from `held` to `finished`, so they are not - // counted twice. - self.held.uncharge(members); + // The retained heap moves from `held` to `finished`, counted once. + self.held.uncharge(retained); self.finished.charge(n)?; sorted += n; let accs = group.accs.into_iter().map(Acc::into_agg).collect(); diff --git a/src/query/engine/tests.rs b/src/query/engine/tests.rs index d899af78b3..3aa571970b 100644 --- a/src/query/engine/tests.rs +++ b/src/query/engine/tests.rs @@ -3933,6 +3933,107 @@ fn aggregate_finish_keeps_sum_distinct_members_charged_once() { ); } +/// Observe retained state between rows, before the finishing map is built. +#[test] +fn sum_and_avg_charge_exact_heap_once_and_release_it() { + use super::{ + Operator, + cell::{Cell, Scalar}, + }; + use std::sync::{Arc, Mutex}; + struct Numbers { + value: Scalar, + at: usize, + fail: bool, + meter: Arc, + observed: Arc>>, + } + impl Operator for Numbers { + fn next_batch(&mut self, _: &mut super::Ctx<'_>) -> anyhow::Result> { + self.observed.lock().unwrap().push(self.meter.used()); + if self.fail && self.at == 1 { + anyhow::bail!("injected child failure"); + } + if self.at == 2 { + return Ok(None); + } + self.at += 1; + let mut batch = super::Batch::new(1); + batch.push(&mut [Some(Cell::Val(self.value.clone()))]); + Ok(Some(batch)) + } + fn rewind(&mut self, _: &[Cell]) { + self.at = 0; + } + fn close(&mut self) { + self.at = 2; + } + fn counters(&self, _: &mut Vec) {} + } + for func in ["sum", "avg"] { + let build = |value, fail, meter: &Arc| { + let observed = Arc::new(Mutex::new(Vec::new())); + let op = super::aggregate::Aggregate::new( + Box::new(Numbers { + value, + at: 0, + fail, + meter: meter.clone(), + observed: observed.clone(), + }), + vec![], + vec![(func.into(), 0)], + false, + false, + meter, + ); + (op, observed) + }; + let meter = super::Meter::unlimited(); + let (mut ints, observed) = build(Scalar::Int(1), false, &meter); + pull_once(&mut ints, &meter) + .unwrap() + .expect("integer result"); + let base = observed.lock().unwrap()[1]; + assert_eq!(meter.used(), 0); + for value in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] { + let (mut op, observed) = build(Scalar::Float(value), false, &meter); + pull_once(&mut op, &meter) + .unwrap() + .expect("special float result"); + assert_eq!(&observed.lock().unwrap()[1..], &[base, base]); + assert_eq!(meter.used(), 0); + } + let (mut floats, observed) = build(Scalar::Float(1.0), false, &meter); + pull_once(&mut floats, &meter) + .unwrap() + .expect("float result"); + assert_eq!(&observed.lock().unwrap()[1..], &[base + 288, base + 288]); + assert_eq!(meter.used(), 0, "finished state released"); + + let limited = super::Meter::new(base + 287); + let (mut op, observed) = build(Scalar::Float(1.0), false, &limited); + let err = pull_once(&mut op, &limited).expect_err("first exact allocation exceeds limit"); + assert_eq!(crate::MinigrafError::from(err).code(), "QRY-012"); + assert_eq!( + observed.lock().unwrap().len(), + 1, + "fails before second pull" + ); + op.close(); + assert_eq!(limited.used(), 0); + + let (mut op, _) = build(Scalar::Float(1.0), true, &meter); + assert!(pull_once(&mut op, &meter).is_err()); + assert_eq!(meter.used(), base + 288); + op.rewind(&[]); + assert_eq!(meter.used(), 0); + assert!(pull_once(&mut op, &meter).is_err()); + op.close(); + assert_eq!(meter.used(), 0); + } +} + /// Every query boxes its `Stream` (`Answer::Stream`). glibc serves blocks of /// up to 1,032 bytes from a per-thread cache and larger ones through its /// locked bins, which cost a point query about 5% (#721 grew the stream to diff --git a/tests/error_codes_prs_test.rs b/tests/error_codes_prs_test.rs index 04730dbdc8..1fb5fa18a2 100644 --- a/tests/error_codes_prs_test.rs +++ b/tests/error_codes_prs_test.rs @@ -163,6 +163,10 @@ fn prs_codes_cover_with_and_aggregates() { #[test] fn prs_codes_cover_window_functions() { let cases: &[(&str, &str)] = &[ + ( + r#"(query [:find (row-number ?score) :where [?e :score ?score]])"#, + "PRS-027", + ), ( r#"(query [:find (rank ?score) :where [?e :score ?score]])"#, "PRS-027", diff --git a/tests/grammar/invalid/semantic/window_only_func_without_over.edn b/tests/grammar/invalid/semantic/window_only_func_without_over.edn index c9a3f5c0ca..65665e14c5 100644 --- a/tests/grammar/invalid/semantic/window_only_func_without_over.edn +++ b/tests/grammar/invalid/semantic/window_only_func_without_over.edn @@ -1,2 +1,2 @@ -(query [:find (avg ?salary) +(query [:find (rank ?salary) :where [?e :emp/salary ?salary]]) diff --git a/tests/grammar/valid/query_aggregate_avg.edn b/tests/grammar/valid/query_aggregate_avg.edn new file mode 100644 index 0000000000..10a9ffa70c --- /dev/null +++ b/tests/grammar/valid/query_aggregate_avg.edn @@ -0,0 +1 @@ +(query [:find (avg ?v) :where [?e :n ?v]]) diff --git a/tests/grammar/valid/query_aggregate_avg_grouped.edn b/tests/grammar/valid/query_aggregate_avg_grouped.edn new file mode 100644 index 0000000000..e1c161f91b --- /dev/null +++ b/tests/grammar/valid/query_aggregate_avg_grouped.edn @@ -0,0 +1 @@ +(query [:find ?g (avg ?v) :where [?e :group ?g] [?e :n ?v]]) diff --git a/tests/ordinary_avg_test.rs b/tests/ordinary_avg_test.rs new file mode 100644 index 0000000000..649d4e6992 --- /dev/null +++ b/tests/ordinary_avg_test.rs @@ -0,0 +1,232 @@ +//! Ordinary avg shares window avg's numeric population and arithmetic (#777). +use minigraf::{BindValue, Minigraf, QueryResult, Value}; + +fn result_rows(result: QueryResult) -> Vec> { + match result { + QueryResult::QueryResults { results, .. } => results, + _ => panic!("query results"), + } +} + +fn rows(db: &Minigraf, query: &str) -> Vec> { + result_rows(db.execute(query).expect("query")) +} + +const AVG: &str = "(query [:find (avg ?v) :where [?e :n ?v]])"; + +#[test] +fn ordinary_avg_reproduction_returns_float() { + let db = Minigraf::in_memory().unwrap(); + db.execute("(transact [[:a :n 10] [:b :n 20]])").unwrap(); + assert_eq!(rows(&db, AVG), vec![vec![Value::Float(15.0)]]); +} + +#[test] +fn ordinary_avg_groups_skip_nonnumbers_and_preserve_with_multiplicity() { + let db = Minigraf::in_memory().unwrap(); + db.execute( + r#"(transact [[:a :g "x"] [:a :n 10] [:b :g "x"] [:b :n 20] + [:c :g "x"] [:c :n "unknown"] [:d :g "y"] [:d :n nil] + [:e :g "z"] [:e :n true]])"#, + ) + .unwrap(); + let grouped = "(query [:find ?g (avg ?v) :where [?e :g ?g] [?e :n ?v]])"; + assert_eq!( + rows(&db, grouped), + vec![ + vec![Value::String("x".into()), Value::Float(15.0)], + vec![Value::String("y".into()), Value::Null], + vec![Value::String("z".into()), Value::Null], + ] + ); + let mut finer = rows(&db, "(query [:find (avg ?v) :with ?e :where [?e :n ?v]])"); + finer.sort(); + let mut expected = vec![ + vec![Value::Float(10.0)], + vec![Value::Float(20.0)], + vec![Value::Null], + vec![Value::Null], + vec![Value::Null], + ]; + expected.sort(); + assert_eq!(finer, expected); + assert_eq!(rows(&db, AVG), vec![vec![Value::Float(15.0)]]); + for find in ["(avg ?v)", "(count ?e) (avg ?v)", "?e (avg ?v)"] { + assert!( + rows( + &db, + &format!("(query [:find {find} :where [?e :absent ?v]])") + ) + .is_empty() + ); + } +} + +#[test] +fn ordinary_avg_preserves_other_aggregate_errors_and_dropped_groups() { + let db = Minigraf::in_memory().unwrap(); + db.execute(r#"(transact [[:a :n "unknown"] [:a :empty nil]])"#) + .unwrap(); + for find in ["(avg ?v) (sum ?v)", "(sum ?v) (avg ?v)"] { + let err = db + .execute(&format!("(query [:find {find} :where [?e :n ?v]])")) + .unwrap_err(); + assert_eq!(err.code(), "INT-041"); + } + assert!( + rows( + &db, + "(query [:find (avg ?v) (min ?n) :where [?e :n ?v] [?e :empty ?n]])" + ) + .is_empty() + ); +} + +#[test] +fn ordinary_avg_prepared_execute_and_cursor_use_numeric_count() { + let db = Minigraf::in_memory().unwrap(); + db.execute( + r#"(transact [[:a :g "x"] [:a :n 10] [:b :g "x"] [:b :n 20] + [:c :g "x"] [:c :n "unknown"] [:d :g "y"] [:d :n nil]])"#, + ) + .unwrap(); + let prepared = db + .prepare("(query [:find (avg ?v) :where [?e :g $group] [?e :n ?v]])") + .unwrap(); + for (group, expected) in [("x", Value::Float(15.0)), ("y", Value::Null)] { + let binds = [("group", BindValue::Val(Value::String(group.into())))]; + assert_eq!( + result_rows(prepared.execute(&binds).unwrap()), + vec![vec![expected.clone()]] + ); + let mut cursor = prepared.query(&binds).unwrap(); + assert_eq!( + cursor.next_batch(1).unwrap().unwrap().into_rows(), + vec![vec![expected]] + ); + assert!(cursor.next_batch(1).unwrap().is_none()); + } +} + +#[test] +fn ordinary_avg_matches_window_without_removing_nonnumeric_rows() { + let db = Minigraf::in_memory().unwrap(); + db.execute( + r#"(transact [[:a :n 10] [:a :ord 1] [:b :n "unknown"] [:b :ord 2] + [:c :n 20] [:c :ord 3]])"#, + ) + .unwrap(); + let mut window = rows( + &db, + "(query [:find ?o (avg ?v :over (:order-by ?o)) + (avg ?v :over (:order-by ?o :rows 1 preceding)) :where [?e :n ?v] [?e :ord ?o]])", + ); + window.sort(); + assert_eq!( + window, + vec![ + vec![Value::Integer(1), Value::Float(10.0), Value::Float(10.0)], + vec![Value::Integer(2), Value::Float(10.0), Value::Float(10.0)], + vec![Value::Integer(3), Value::Float(15.0), Value::Float(20.0)], + ] + ); + assert_eq!(rows(&db, AVG), vec![vec![Value::Float(15.0)]]); +} + +#[test] +fn ordinary_avg_recursive_rule_and_transaction_reads() { + let db = Minigraf::in_memory().unwrap(); + db.execute("(transact [[:a :next :b] [:b :next :c] [:b :n 10] [:c :n 20]])") + .unwrap(); + db.execute("(rule [(reachable ?x ?y) [?x :next ?y]])") + .unwrap(); + db.execute("(rule [(reachable ?x ?y) [?x :next ?z] (reachable ?z ?y)])") + .unwrap(); + let q = "(query [:find (avg ?v) :where (reachable :a ?e) [?e :n ?v]])"; + assert_eq!(rows(&db, q), vec![vec![Value::Float(15.0)]]); + let mut tx = db.begin_write().unwrap(); + tx.execute("(transact [[:c :next :d] [:d :n 30]])").unwrap(); + assert_eq!( + result_rows(tx.execute(q).unwrap()), + vec![vec![Value::Float(20.0)]] + ); + tx.rollback(); + assert_eq!(rows(&db, q), vec![vec![Value::Float(15.0)]]); +} + +#[test] +fn ordinary_avg_cursor_spans_batches_without_dropping_groups() { + let db = Minigraf::in_memory().unwrap(); + let facts: String = (0..1100) + .map(|n| format!("[:e{n} :n 10] [:e{n} :group {n}] ")) + .collect(); + db.execute(&format!("(transact [{facts}])")).unwrap(); + let mut cursor = db + .query("(query [:find ?g (avg ?v) :where [?e :group ?g] [?e :n ?v]])") + .unwrap(); + let mut got = Vec::new(); + while let Some(batch) = cursor.next_batch(17).unwrap() { + got.extend(batch.into_rows()); + } + let expected: Vec<_> = (0..1100) + .map(|n| vec![Value::Integer(n), Value::Float(10.0)]) + .collect(); + assert_eq!(got, expected); +} + +#[cfg(not(target_arch = "wasm32"))] +#[test] +fn ordinary_avg_pending_checkpointed_and_reopened_agree() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("avg.graph"); + let expected = vec![vec![Value::Float(1.0)]]; + { + let db = Minigraf::open(&path).unwrap(); + db.execute("(transact [[:a :n 9007199254740993] [:b :n -9007199254740992] [:c :n 2]])") + .unwrap(); + assert_eq!(rows(&db, AVG), expected); + db.checkpoint().unwrap(); + assert_eq!(rows(&db, AVG), expected); + db.execute(r#"(transact [[:d :n "unknown"] [:e :n #{1 2}]])"#) + .unwrap(); + assert_eq!(rows(&db, AVG), expected); + db.checkpoint().unwrap(); + } + assert_eq!(rows(&Minigraf::open(&path).unwrap(), AVG), expected); +} + +#[cfg(all(feature = "test-hooks", not(target_arch = "wasm32")))] +#[test] +fn ordinary_avg_is_independent_of_plan_choices() { + use minigraf::Subplans; + let dir = tempfile::tempdir().unwrap(); + let db = Minigraf::open(dir.path().join("plans.graph")).unwrap(); + db.execute( + r#"(transact [[:a :n 10000000000000000.0] [:a :g 1] + [:b :n 1.0] [:b :g 1] [:c :n -10000000000000000.0] [:c :g 1] + [:d :n 1.0] [:d :g 1] [:e :n 0.5] [:e :g 1] + [:f :n "unknown"] [:f :g 1] [:bad :n 999] [:bad :g 1] [:bad :excluded true]])"#, + ) + .unwrap(); + let q = "(query [:find ?g (avg ?v) :where [?e :g ?g] [?e :n ?v] + (not [?e :excluded true])])"; + for committed in [false, true] { + if committed { + db.checkpoint().unwrap(); + } + for reorder in [false, true] { + let _reorder = minigraf::__reorder(reorder); + for input_order in [false, true] { + let _input = minigraf::__input_order(input_order); + for mode in [Subplans::Choose, Subplans::Hash, Subplans::Correlated] { + let _mode = minigraf::__subplans(mode); + let expected = vec![vec![Value::Integer(1), Value::Float(0.5)]]; + assert_eq!(rows(&db, q), expected); + let mut cursor = db.query(q).unwrap(); + let got: Vec<_> = cursor.by_ref().collect::>().unwrap(); + assert_eq!(got, expected); + } + } + } + } +}