Repository navigation
perf: phase 7.4 — eliminate filter_facts_for_query index rebuild (~62-65% speedup) - #26
Merged
Merged
Conversation
- Add docs/superpowers/specs/2026-03-27-phase-7-4-query-optimizer-design.md covering profiling gate and filter_facts_for_query snapshot fix - Narrow phase 7.4 ROADMAP scope to profiling + snapshot fix only - Move optimizer extensions, rule eval optimization, and predicate push-down to post-1.0 backlog in ROADMAP.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace pprof/criterion with cargo flamegraph (Criterion version conflict) - Enumerate both matcher.rs get_asserted_facts() call sites (lines 27 and 222) - Clarify shared functions (apply_or_clauses, evaluate_not_join) across executor.rs and evaluator.rs with per-file call-site tables - Separate execute_query (line 263) and execute_query_with_rules (line 477) not-filter rows in evaluate_not_join table - Clarify §2.5 insertion point for accumulated_facts Arc computation - Move evaluate_not_join propagation note (line 921) to §2.4 where evaluate_branch is covered - Add per-join allocation note and out-of-scope entry Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Reframe gate as a proceed condition (not just a stop condition) - Add explicit Step 2 conditional header - Architecture Overview now states conditionality upfront Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
8-task TDD implementation plan for filter_facts_for_query Arc<[Fact]> return type change. Gated behind flamegraph profiling confirmation. Unit tests written in red state (Task 3) before implementation (Task 5).
Adds strip = "none" and debug = true to the bench profile so that cargo flamegraph --bench produces readable stack frames instead of opaque binary blobs. Profiling confirmed filter_facts_for_query index rebuild (Indexes::insert / BTreeMap) is dominant cost at 10K facts (~25% query, ~23% negation). Phase 7.4 profiling gate: PASSES.
Adds MatcherStorage enum with Owned(FactStorage) and Slice(Arc<[Fact]>) variants. from_slice constructor used by the executor snapshot path. Existing new(FactStorage) constructor and all evaluator call sites unchanged. Part of phase 7.4 snapshot fix.
Two unit tests for filter_facts_for_query in executor.rs #[cfg(test)]: net-asserted view (retracted facts excluded) and valid-time filtering. Tests currently fail to compile; Task 5 changes the return type to Arc<[Fact]> to make them pass. Phase 7.4. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Fact]> All call sites convert FactStorage → Arc at call time via get_asserted_facts(). No semantic change. Prepares for filter_facts_for_query returning Arc directly. Part of phase 7.4 snapshot fix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
filter_facts_for_query now returns Arc<[Fact]> instead of FactStorage, eliminating the O(N) BTreeMap rebuild (4 indexes x N facts) on every non-rules query call. execute_query constructs no FactStorage at all. execute_query_with_rules still converts Arc -> FactStorage for StratifiedEvaluator (deferred to post-profiling decision). Also fixes the test_filter_facts_for_query_valid_time_filter test: the "open-ended" fact now uses explicit valid_from=0 so it is visible at the test's query timestamps (1500, 3000) rather than at tx_id_now() which is epoch-ms and far larger. Phase 7.4. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion Replaces four separate accumulated.clone() + get_asserted_facts() calls with a single Arc<[Fact]> computed once and reused for PatternMatcher, apply_or_clauses, not-body matching, and evaluate_not_join. Part of phase 7.4 snapshot fix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
filter_facts_for_query returns Arc<[Fact]>; execute_query path constructs no FactStorage. PatternMatcher::from_slice added. 568 tests passing. ~62-65% speedup on non-rules queries at 10K facts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Update query latency point_entity/10K: 15.6 ms → 8.6 ms (-61.5% vs pre-fix baseline of 22 ms) - Update aggregation count_scale/10K: 23.4 ms → 9.7 ms (-64.7%) - Update negation not_scale/10K: 13.4 s → 7.0 s (-12.1% vs pre-fix baseline) - Update disjunction or_scale/10K: 73.7 s → 68.9 s (within noise, p=0.36) - Add Phase 7.3 → Phase 7.4 summary section with before/after table - Update Known Limitations: index rebuild eliminated for non-rules path - Fix Criterion version: 0.5 → 0.8 - Update header note to reference v0.13.1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hase references - Add named post-1.0 backlog entry for B+Tree Selective Lookup (range-scan predicate push-down): the get_all_facts() full scan is O(N) regardless of query predicates; fix is to use get_facts_by_entity/get_facts_by_attribute for bound patterns to achieve O(log N) selective fetch - Update BENCHMARKS.md query latency and known limitations sections to reference "post-1.0 backlog" instead of vague "future phase" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
filter_facts_for_queryon every query call. The function now returnsArc<[Fact]>instead of a throwawayFactStorage, avoiding the rebuild of EAVT/AEVT/AVET/VAET indexes thatPatternMatchernever used.execute_query(non-rules path): zeroFactStorageconstruction — pureArc<[Fact]>flow from filter through to all pattern match, or-clause, and not-join operations.query/point_entity/10k: 22ms → 8.6ms;aggregation/count_scale/10k: 28ms → 9.7ms). Negation improved ~12%; disjunction unchanged (dominated by known O(N²) inner loop, post-1.0 backlog).Changes
PatternMatcher: newMatcherStorageenum (Owned(FactStorage)/Slice(Arc<[Fact]>)),from_sliceconstructor,get_facts()helper. Existingnew(FactStorage)unchanged.filter_facts_for_query: return typeResult<FactStorage>→Result<Arc<[Fact]>>. Index rebuild removed.apply_or_clauses,evaluate_not_join,not_body_matches,evaluate_branch: signatures changed from&FactStoragetoArc<[Fact]>.execute_query_with_rules: still convertsArc<[Fact]>→FactStorageforStratifiedEvaluator(deferred; post-profiling TODO).accumulated_facts: Arc<[Fact]>computed once per rule iteration, reused for 4 call sites (was 4 separateget_asserted_facts()calls).matcher.rs, 2 inexecutor.rs). 568 tests passing.Test Plan
cargo test— 568 tests pass, 0 failurescargo clippy— cleancargo fmt --check— cleanIndexes::insert/BTreeMap as dominant cost (~25% query, ~23% negation) before changes🤖 Generated with Claude Code