Skip to content

perf: narrow bound-entity point queries to the queried attribute (#323 mitigation) - #380

Merged
adityamukho merged 12 commits into
mainfrom
perf/323-point-query-history-scan
Sep 26, 2026
Merged

adityamukho merged 12 commits into
mainfrom
perf/323-point-query-history-scan

Conversation

@adityamukho

Copy link
Copy Markdown
Collaborator

Summary

Mitigates #323 on the v2.x line (file format v7, no API change).

A bound-entity point query [:e :attr ?v] used to range-scan and resolve every record the entity had ever written, across all its attributes. So an entity with one heavily retracted and reasserted attribute made every attribute on it slow.

  • Attribute-narrowed EAVT lookup: a new FactStorage::get_facts_by_entity_attribute_indexed scans exactly the range [(e, a), (e, a + "\0")), pending plus committed. selective_fact_fetch uses it when every pattern on a bound entity names a concrete attribute keyword. A variable or pseudo attribute keeps the whole-entity scan. If narrowing would exceed the lookup budget of 4, the query falls back to one scan per entity, so narrowing never turns a selective query into a full scan.
  • Dedup removed: the executor's (entity, attribute, tx_count, asserted) dedup was redundant, because its only consumer is net_asserted_facts, which ignores duplicate records. It was also lossy: it dropped rows when one WriteTransaction wrote an attribute in several valid-time windows. That's a real bug on main, now fixed and pinned by a test that fails on main.
  • net_asserted_facts: encodes each value once and uses borrowed keys instead of cloned ones. It deliberately keeps std's randomly keyed hasher: fact values are often untrusted text, and a fixed-seed fast hash (tried here, worth about 13%) would let crafted colliding values make queries quadratic.

Refs #323. Kept open: on v2.x, reading the churned attribute itself still costs time proportional to its history, because v7 keys carry neither the value nor the asserted flag. The structural fix needs v8 keys and is tracked in #379 (v3.0.0).

Benchmarks (criterion, point_query_chain_depth, on AC and idle)

Query Depth 2000: v2.0.1 This PR
[:e/hot :other ?v], sibling of a churned attribute 4.58 ms 19.0 µs, flat vs depth 1
[:e/hot :hash ?v], churned attribute 4.64 ms 3.36 ms (1.38×)
[?e :hash ?v], attribute scan 4.49 ms 3.33 ms (1.35×)

Depth 1 is unchanged. The full table is in docs/BENCHMARKS.md. The spec's original ≥3× target for the churned attribute was not reachable on v7 keys; the spec was revised and this ships as a mitigation.

Spec: docs/superpowers/specs/2026-09-25-point-query-history-scan-design.md · Plan: docs/superpowers/plans/2026-09-25-point-query-history-scan.md

Type of change

  • Bug fix (non-breaking) — performance plus the same-transaction multi-window row-drop fix
  • New feature (non-breaking)
  • Breaking change (requires migration or version bump)
  • Documentation / tests only

Checklist

  • cargo test passes (1165 passed, 8 ignored)
  • cargo clippy -- -D warnings is clean (--all-features)
  • cargo fmt --check passes
  • New code has tests (unit and/or integration)
  • Error paths are tested, not just happy paths
  • No unwrap() / expect() in library code paths
  • CHANGELOG.md updated under Unreleased
  • I have read PHILOSOPHY.md and this change aligns with it

Notes for reviewer

  • Result equivalence: tests/point_query_history_test.rs compares every selective-path query with the same query under :as-of 1000000000, which always takes the full-scan path. It covers prefix-sibling attributes, variable and pseudo attributes on a bound entity, not/or, more than 4 attributes, :valid-at, and file-backed databases across checkpoint and reopen.
  • net_asserted_facts rewrite: checked against the previous implementation, kept as a test-only reference, on 500 randomized histories. Duplicate-record idempotence, which the dedup removal relies on, is pinned by its own test.
  • Range end: a + "\0" is the exact successor of a. The older next_string_prefix returns None for attributes whose last UTF-8 byte is 0xBF or 0x7F (for example :丿), which would have left the scan unbounded. get_facts_by_attribute on main still has that weakness for attribute scans (results correct, just slow); that's out of scope here and can be a follow-up.
  • An executor unit test that asserted the old whole-entity fetch (2 facts) now asserts the narrowed fetch (1), plus a variable-attribute case (2).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy

adityamukho and others added 12 commits September 25, 2026 18:14
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
…eptance to measured

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
…indow fix (#323)

End the committed range at attribute + NUL (exact successor) instead of
incrementing the last byte, which returned None for attributes ending in
0x7F or a 0xBF UTF-8 byte and left the scan unbounded to the end of the
index. Also stops resolving prefix siblings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Fact values are often untrusted text (agent memory); a fixed-seed fast
hash would let crafted colliding values make queries quadratic. The
borrowed-key / encode-once rewrite stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
@adityamukho adityamukho added this to the v2.3.0 milestone Sep 26, 2026
@codecov

codecov Bot commented Sep 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.71795% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/graph/storage.rs 97.95% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@adityamukho
adityamukho merged commit 402790d into main Sep 26, 2026
17 checks passed
@adityamukho
adityamukho deleted the perf/323-point-query-history-scan branch September 26, 2026 04:21
adityamukho added a commit that referenced this pull request Sep 26, 2026
…382)

get_facts_by_attribute ended its AEVT range at next_string_prefix(attr),
which increments the last byte. For attributes ending in 0x7F or 0xBF
(e.g. `:丿`, `:ÿ`) that is not valid UTF-8, so the committed scan ran to
the end of the AEVT index. It also covered prefix siblings (`:ab` for `:a`).

End the range at `attribute + "\0"`, the exact successor, as #380 does for
the entity path. Remove the now-unused next_string_prefix.

Sparse `:丿` scan (100 of 40.1k facts, checkpointed): 45.4 ms -> 251 µs.

Closes #381
Refs #323


Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant