Skip to content

feat(engine): Aggregate on the streaming engine (#547 PR 2) - #645

Merged
adityamukho merged 5 commits into
v3from
feat/547-engine-aggregate
Oct 20, 2026
Merged

adityamukho merged 5 commits into
v3from
feat/547-engine-aggregate

Conversation

@adityamukho

Copy link
Copy Markdown
Collaborator

Part 5 of #473 (#547), PR 2 of 3: plan steps 9–12 of docs/superpowers/plans/2026-10-20-547-aggregate-window.md. Aggregates run on the streaming engine; windows stay on the executor until PR 3.

What changes

  • Routing (engine/plan.rs): a :find with aggregates (and :with) plans like any engine query, with a last Step::Aggregate { groups, aggs, zero_row }. Plan::find now names the column of each :find element in the last step's rows. Window queries return None (executor).
  • Aggregate (engine/aggregate.rs): a HashMap<Vec<JoinKey>, Group> keyed on the grouping cells (plain :find variables, then :with) as Value's Ord compares values (JoinKey::group: no keyword→ref normalising, -0.0 = 0.0, every NaN one key). Ungrouped = one entry, constant memory. count counts non-nil cells; count-distinct keeps a set of cell keys; other aggregates convert the cell to a Value and use aggregate::Agg from PR 1. At end of input: keys → values, sort by Value's Ord, finish in that order (first error fails, INT-042 drops the group), emit batches of 1,024. Zero row for a count-only query with no plain variable, as the executor.
  • Float group value (both engines, breaking, CHANGELOG streaming engine 5/9: Aggregate, Sort and Window (#473) #547 item 7): a group whose key floats are Ord-equal but distinct (0.0/-0.0, NaN payloads) returned the first one the plan met, so engine and executor disagreed (found by the oracle). It now returns the least by value_order (-0.0 when present). Confirmed with the maintainer. Docs site: v3: aggregate group value for 0.0 / -0.0 (#547) minigraf-docs#46.

Deviations from the plan

  • Step 9: plan::plan does not call check_find_functions: open_query already calls it before planning.
  • Step 10: no Cell::Value variant; aggregate results become cells through Cell::of_value (refs and keywords from registered aggregates get ids).
  • Envelope: added a count shape ([:find (count ?v) :where [?e ?a ?v]]); the envelope had only count_grouped.
  • Branch name feat/547-engine-aggregate (the plan's feat/547-aggregate was the merged spec PR's branch).

Tests

  • engine::tests: aggregates_match_executor (21 shapes × 6 temporals + a :db/tx-count shape, in memory/after checkpoint/after reopen, via the oracle and routing counter), ungrouped_aggregate_returns_one_row (counters: rows_in 3000, one batch of one row; zero row; empty answers), grouped_aggregate_orders_groups_and_fails_in_a_later_group; other_queries_stay_on_the_executor now uses windows.
  • tests/aggregation_test.rs::float_group_returns_least_value; tests/plan_independence_test.rs +2 float group shapes.
  • 1620 tests (1610 passing, 10 ignored); --features test-oracle: 1623 passed, 0 failed. Codex review: no findings.

Measurement (#394 envelope, same idle server, v3 234cc3e vs this branch, p50 / peak RSS)

Shape 100K 1M 10M
count 207 → 44 ms, 97 → 7 MiB 3.01 s → 457 ms, 728 → 7 MiB 29.6 s → 4.46 s, 6,619 → 7 MiB
count_grouped 31.4 → 5.9 ms, 14 → 5 MiB 397 → 61 ms, 97 → 7 MiB 4.51 s → 614 ms, 901 → 7 MiB

Every other shape within noise. Recorded in docs/BENCHMARKS.md and CHANGELOG.md.

Refs #547 (closed by hand after PR 3), #473.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PtCdzbJMSS1V8qEPaJ4jXz

adityamukho and others added 3 commits October 20, 2026 13:53
Aggregate queries and :with run on the engine: one hash-keyed group table
(one entry when ungrouped, so constant memory), count and count-distinct
on cells, other aggregates on values via aggregate::Agg; groups sorted by
value and finished in that order. A group of floats Value's Ord finds
equal (-0.0 and 0.0, NaNs) returns the least by value_order, on both
engines, so the answer does not depend on the plan. Windows stay on the
executor until PR 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PtCdzbJMSS1V8qEPaJ4jXz
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.04520% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/query/engine/aggregate.rs 94.94% 5 Missing ⚠️
src/query/engine/cell.rs 90.90% 1 Missing ⚠️
src/query/engine/filter.rs 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…#473)

1200 -> 1230 KiB on v3 until #551 removes the executor's query path;
#547 PR 2 brings the release binary to ~1205 KiB. Agreed with the
maintainer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PtCdzbJMSS1V8qEPaJ4jXz
@adityamukho

Copy link
Copy Markdown
Collaborator Author

Binary size: 1,208 KiB in CI against the 1,200 KiB gate. Shared group finishing with the executor (aggregate::finish_groups, Groups) and reused join.rs's HashMap<Vec<JoinKey>, usize> (−3 KiB). With the maintainer's agreement, the v3 gate is 1,230 KiB until #551 removes the executor's query path (noted on #551, PHILOSOPHY.md §4).

@adityamukho
adityamukho merged commit 58c19dd into v3 Oct 20, 2026
25 checks passed
@adityamukho
adityamukho deleted the feat/547-engine-aggregate branch October 20, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant