Skip to content

fix(query): count window partition scratch against memory limit (#765) - #766

Merged
adityamukho merged 1 commit into
v3from
fix/765-window-memory
Oct 26, 2026
Merged

adityamukho merged 1 commit into
v3from
fix/765-window-memory

Conversation

@adityamukho

Copy link
Copy Markdown
Collaborator

Summary

  • Charge each window's partition row indices to the query meter while they overlap input rows and earlier window result columns.
  • Replace the allocating partition map and stable sort buffer with one fallibly reserved index vector and an in-place total sort. Preserve Value::Ord partition equality, peer ranks, row-number tie breaks, and lag/lead adjacency.
  • Release the temporary index charge after each window and on errors; allocator refusal returns QRY-012.

Fixes #765.

Verification

  • Test-first: the many-partition memory-limit and allocation-refusal regressions failed on origin/v3, then passed with this change.
  • cargo test --quiet and cargo test --quiet --features test-hooks on s1c44t88 (both exit 0; 77 successful suite summaries and no failures in each run).
  • cargo clippy --all-features -- -D warnings, cargo fmt --all --check, and git diff --check.
  • Bench machine s1c44t88, one NUMA node, 10k rows, Criterion 10 samples / 1-second warm-up / 2-second measurement: rank 17.57 ms baseline → 17.68 ms fix; 10k distinct partitions 19.25 ms baseline → 19.22 ms fix.
  • Claude Opus 5.5 final diff review: no actionable findings. Its assumptions about Held::charge, Held::uncharge, and the reserve-failure hook were checked against the code.

Scope

The meter charges a per-entry estimate of index bytes, consistent with its other holders. Rust's try_reserve_exact can receive excess capacity from the allocator, which is outside that estimate. The existing value_order comparison for Value::Set still builds temporary vectors. This change removes keyed.sort_by's merge buffer and the unmetered partition map; it does not change the set comparator outside window partitioning.

@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.53086% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/query/datalog/aggregate.rs 97.43% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@adityamukho
adityamukho merged commit deebeaf into v3 Oct 26, 2026
19 checks passed
@adityamukho
adityamukho deleted the fix/765-window-memory branch October 26, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant