Repository navigation
perf: narrow bound-entity point queries to the queried attribute (#323 mitigation) - #380
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
… dedup (#323) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
…ixed (#323) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
…eptance to measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
…indow fix (#323) End the committed range at attribute + NUL (exact successor) instead of incrementing the last byte, which returned None for attributes ending in 0x7F or a 0xBF UTF-8 byte and left the scan unbounded to the end of the index. Also stops resolving prefix siblings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Fact values are often untrusted text (agent memory); a fixed-seed fast hash would let crafted colliding values make queries quadratic. The borrowed-key / encode-once rewrite stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Sep 26, 2026
adityamukho
added a commit
that referenced
this pull request
Sep 26, 2026
…382) get_facts_by_attribute ended its AEVT range at next_string_prefix(attr), which increments the last byte. For attributes ending in 0x7F or 0xBF (e.g. `:丿`, `:ÿ`) that is not valid UTF-8, so the committed scan ran to the end of the AEVT index. It also covered prefix siblings (`:ab` for `:a`). End the range at `attribute + "\0"`, the exact successor, as #380 does for the entity path. Remove the now-unused next_string_prefix. Sparse `:丿` scan (100 of 40.1k facts, checkpointed): 45.4 ms -> 251 µs. Closes #381 Refs #323 Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mitigates #323 on the v2.x line (file format v7, no API change).
A bound-entity point query
[:e :attr ?v]used to range-scan and resolve every record the entity had ever written, across all its attributes. So an entity with one heavily retracted and reasserted attribute made every attribute on it slow.FactStorage::get_facts_by_entity_attribute_indexedscans exactly the range[(e, a), (e, a + "\0")), pending plus committed.selective_fact_fetchuses it when every pattern on a bound entity names a concrete attribute keyword. A variable or pseudo attribute keeps the whole-entity scan. If narrowing would exceed the lookup budget of 4, the query falls back to one scan per entity, so narrowing never turns a selective query into a full scan.(entity, attribute, tx_count, asserted)dedup was redundant, because its only consumer isnet_asserted_facts, which ignores duplicate records. It was also lossy: it dropped rows when oneWriteTransactionwrote an attribute in several valid-time windows. That's a real bug on main, now fixed and pinned by a test that fails on main.net_asserted_facts: encodes each value once and uses borrowed keys instead of cloned ones. It deliberately keeps std's randomly keyed hasher: fact values are often untrusted text, and a fixed-seed fast hash (tried here, worth about 13%) would let crafted colliding values make queries quadratic.Refs #323. Kept open: on v2.x, reading the churned attribute itself still costs time proportional to its history, because v7 keys carry neither the value nor the asserted flag. The structural fix needs v8 keys and is tracked in #379 (v3.0.0).
Benchmarks (criterion,
point_query_chain_depth, on AC and idle)[:e/hot :other ?v], sibling of a churned attribute[:e/hot :hash ?v], churned attribute[?e :hash ?v], attribute scanDepth 1 is unchanged. The full table is in
docs/BENCHMARKS.md. The spec's original ≥3× target for the churned attribute was not reachable on v7 keys; the spec was revised and this ships as a mitigation.Spec:
docs/superpowers/specs/2026-09-25-point-query-history-scan-design.md· Plan:docs/superpowers/plans/2026-09-25-point-query-history-scan.mdType of change
Checklist
cargo testpasses (1165 passed, 8 ignored)cargo clippy -- -D warningsis clean (--all-features)cargo fmt --checkpassesunwrap()/expect()in library code pathsCHANGELOG.mdupdated underUnreleasedNotes for reviewer
tests/point_query_history_test.rscompares every selective-path query with the same query under:as-of 1000000000, which always takes the full-scan path. It covers prefix-sibling attributes, variable and pseudo attributes on a bound entity,not/or, more than 4 attributes,:valid-at, and file-backed databases across checkpoint and reopen.net_asserted_factsrewrite: checked against the previous implementation, kept as a test-only reference, on 500 randomized histories. Duplicate-record idempotence, which the dedup removal relies on, is pinned by its own test.a + "\0"is the exact successor ofa. The oldernext_string_prefixreturnsNonefor attributes whose last UTF-8 byte is 0xBF or 0x7F (for example:丿), which would have left the scan unbounded.get_facts_by_attributeon main still has that weakness for attribute scans (results correct, just slow); that's out of scope here and can be a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy