Skip to content

feat(query): joins on the streaming engine (#545 PR 2) - #593

Merged
adityamukho merged 3 commits into
v3from
feat/545-joins
Oct 14, 2026
Merged

adityamukho merged 3 commits into
v3from
feat/545-joins

Conversation

@adityamukho

Copy link
Copy Markdown
Collaborator

Part of #545 (#473 part 3), PR 2 of 3 in docs/superpowers/plans/2026-10-14-545-joins.md (steps 4–9). #545 stays open until PR 3.

What

Queries of several patterns with stored attributes, with or without expression clauses, now run on the streaming engine.

  • Planner v2 (plan.rs): clauses in optimizer::plan order. The first pattern is a Scan. Each later pattern is an IndexNestedLoopJoin when its entity is a variable already bound, otherwise a HashJoin (value and attribute positions, cross products), built on the side with the smaller branch_cost (ties build on the inner side). Pseudo-attribute patterns are still accepted only in single-pattern queries; across joins they wait for PR 3.
  • Joins (join.rs): the inner pattern is a Scan over its own slots, so a variable repeated inside it compares exactly, as it does in the executor's hash path. INLJ probes one entity range per outer row, committed and pending (Scan::probe; the attribute and value constants are resolved once). Both joins resume across output batches.
  • Join keys (cell.rs): JoinKey. Floats compare by IEEE == (-0.0 = 0.0, NaN equals nothing). A keyword equals a ref to the entity it names (IdSpace::keyword_eid), except between two attribute positions. Long strings have one form: Cell::of_value turns a string the value pages hold into Long, both for pending facts and for expression results.
  • Which cell a join keeps (decision 6): the planner runs the executor's Binders (now pub(crate)), so a variable keeps the value from its first pattern in the :where text.

Tests

  • The oracle matrix in query::engine::tests (every query also runs on the executor) gains 33 join shapes at all 6 temporals, in memory, across checkpoints and after reopen. The shapes cover INLJ via keyword, ref and UUID-named keyword, value joins of keyword against ref in both text orders, floats 0.0/-0.0, long strings, attribute joins from attributes and from entities, cross products, and expressions between joins. New test data: attributes that name entities.
  • engine_joins_read_many_batches covers one probe with more than a batch of matches, cross products, and tables built on either side, over committed and pending facts. planner_chooses_the_join and keyword_eid_is_the_named_entitys_id are new unit tests.
  • cargo test and cargo test --features test-oracle pass: 1493 passed, 10 ignored. Most multi-pattern queries in the integration suites now run through the oracle. Clippy is clean for native, --all-features, wasm32/browser and wasip1. The engine tests also pass with --features wasm.
  • Separate commit: join_rules_test::pseudo_attribute_pattern_never_replaces_a_value fails under --features wasm on v3 already. With source order, the pseudo-attribute pattern runs first and binds ?b. The test is now native-only. CI does not run --features wasm tests.

Measurements

#394 envelope on the idle bench server, A/B against eab15bb, two rounds each, p50 and peak RSS at 1M facts:

Shape v3 PR 2
join_2 796 ms, 322 MiB 50 µs, 4 MiB
join_3_selective 1.67 s, 392–462 MiB 148 ms, 14 MiB
join_3_scan 2.64 s, 468–480 MiB 1.12 s, 42 MiB

Point, :as-of, :valid-at, hot-entity and attribute-scan queries are unchanged. 100K rows are in docs/BENCHMARKS.md. Release binary: 1,165,920 → 1,182,944 bytes (+16.6 KiB, limit 1,200 KiB).

Known deviation

Two different keywords that name the same UUID (for example :<uuid> in upper and lower case), joined between two attribute positions, compare by ident here. The executor's hash path normalises them in that case, and its nested-loop path does not.

No API change and no new error code, so no binding cascade. The docs site and the CLAUDE.md module list beyond this PR follow with PR 3.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR

adityamukho and others added 3 commits October 14, 2026 18:50
#545)

Under the wasm feature the planner keeps source order, so
[?b :db/valid-at ?t] runs first and binds ?b as a ref; the test's premise
(the planner runs it last) holds only natively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
Planner v2 runs queries of several patterns with stored attributes and
expression clauses: the first pattern is a Scan, each later one an
IndexNestedLoopJoin when its entity is a bound variable (one EAVT range
per outer row, committed and pending) or a HashJoin otherwise, built on
the side with the smaller estimate. Cells join by JoinKey: floats by IEEE
==, a keyword with a ref to the entity it names (IdSpace::keyword_eid),
long strings as one form (pending strings the value pages hold become
Long). A join keeps the outer cell unless the executor's Binders say the
pattern gives the value (first binding in :where text order).
Pseudo-attribute patterns stay with single-pattern queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyLscCUuJ1GQpdAeX79EdR
@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.01140% with 14 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/query/engine/join.rs 94.79% 9 Missing ⚠️
src/query/engine/cell.rs 90.90% 2 Missing ⚠️
src/query/engine/plan.rs 98.01% 2 Missing ⚠️
src/query/engine/filter.rs 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@adityamukho
adityamukho merged commit 86ad20b into v3 Oct 14, 2026
25 checks passed
@adityamukho
adityamukho deleted the feat/545-joins branch October 14, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant