From 23cde7d7433f5a615f4844a0dc099bd09e322cd0 Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Sat, 24 Oct 2026 09:18:29 +0530 Subject: [PATCH] docs: README envelope on the streaming engine; :limit (#306) is v3.0.0 scope README's performance table and text still had the pre-engine laptop run (scans that materialize, rule queries reading the whole database). They now use the #551 A/B figures from docs/BENCHMARKS.md. :limit (#306) is in the v3.0.0 milestone: ROADMAP and CLAUDE.md listed it under v3.next. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0169pNQsLcCEn48te9X9WuZW --- CLAUDE.md | 4 ++-- README.md | 19 ++++++++++--------- ROADMAP.md | 4 ++-- 3 files changed, 14 insertions(+), 13 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index cf73caa26..adb7db12f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -223,8 +223,8 @@ See `docs/TEST_COVERAGE.md` for the full per-file breakdown. Current plan (see `ROADMAP.md` and the milestones): - **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release. -- **v3.0.0** (format v8 and data integrity, tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`. -- **v3.next** (the release after v3.0.0) holds new features (query profiler #185, `:limit` #306, lag/lead #182, UniFFI additions). +- **v3.0.0** (format v8 and data integrity, plus `:limit` #306; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`. +- **v3.next** (the release after v3.0.0) holds new features (query profiler #185, lag/lead #182, UniFFI additions). - Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (`known-issue` label); `data-integrity`/`corruption`/`durability` issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased `v3` branch close on merge (closing keywords do not fire on `v3`, so close them with `gh issue close`) and stay listed in #421 (CONTRIBUTING.md). - Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2. diff --git a/README.md b/README.md index 09bac4093..5b86846ff 100644 --- a/README.md +++ b/README.md @@ -247,18 +247,19 @@ See [ROADMAP.md](ROADMAP.md) for planned work and current direction. ## Performance -Performance envelope on the `v3` branch (format v8), measured up to 10M facts on a laptop (Intel i7-1065G7, SSD, page cache warm; see [BENCHMARKS.md](docs/BENCHMARKS.md#performance-envelope-v300-394) for every shape, RSS and the method): +Performance envelope on the `v3` branch (format v8), measured up to 10M facts on an otherwise idle server, two rounds (see [BENCHMARKS.md](docs/BENCHMARKS.md#executor-removal-551) for every shape, RSS and the method): | Workload | 100K facts | 1M facts | 10M facts | |---|---:|---:|---:| -| Point lookup (bound entity + attribute, incl. `:as-of` / `:valid-at`), p50 / p99 | 13 µs / 16 µs | 12 µs / 48 µs | 12 µs / 46 µs | -| Entity with 1,000 versions, p50 / p99 | 0.26 / 0.30 ms | 0.26 / 0.29 ms | 0.27 / 0.30 ms | -| Open a file (RSS after open: 3 MiB) | 0.12 ms | 0.14 ms | 0.12 ms | -| Checkpoint after 1 new fact, p50 / p99 | 3.1 / 5.1 ms | 2.9 / 4.3 ms | 3.8 / 21 ms | -| Attribute scan (N/6 rows), p50 | 43 ms | 0.46 s | 4.8 s | -| Selective 2-clause join (1 row), p50 | 54 ms | 0.61 s | 6.5 s | - -Point lookups and time travel on one entity stay flat at every tested size. Scans, aggregations and joins grow linearly with the facts of the attributes they touch, because a query materializes them before joining (#473). They are interactive up to about 100K facts. Rule queries read the whole database (#529): 2.7 s at 100K facts and 40 s at 1M. Graphs over 10M facts are not tested. +| Point lookup (bound entity + attribute, incl. `:as-of` / `:valid-at`), p50 / p99 | 18–21 µs / 25–39 µs | 18–20 µs / 22–37 µs | 18–20 µs / 22–27 µs | +| Entity with 1,000 versions, p50 / p99 | 0.29–0.30 / 0.31–0.35 ms | 0.29–0.30 / 0.30–0.36 ms | 0.29–0.30 / 0.32–0.37 ms | +| Open a file (RSS after open: 3 MiB) | 0.12–0.16 ms | 0.25–0.27 ms | 0.13–0.23 ms | +| Checkpoint after 1 new fact, p50 / p99 | 0.7–0.9 / 1.4–1.6 ms | 0.9–1.8 / 2.7–2.8 ms | 1.1–1.5 / 2.2–2.9 ms | +| Attribute scan (N/6 rows) drained in batches of 1,000, p50 | 23 ms | 0.25 s | 2.5 s | +| Selective 2-clause join (1 row), p50 | 38 µs | 37 µs | 37 µs | +| Rule query with a constant (magic sets, 5–7 rows), p50 | 1.1 ms | 1.5 ms | 1.9 ms | + +Point lookups, time travel on one entity, and joins and rule queries that a constant makes selective stay flat at every tested size. Queries that read a whole attribute (scans, aggregates, joins with nothing selective) grow linearly with the facts they read, in bounded memory: the 1.67M-row scan at 10M facts holds one batch, 7 MiB. Windows and whole transitive closures hold their rows, and `OpenOptions::query_memory_limit` stops a query that passes it with `QRY-012`. Graphs over 10M facts are not tested. File-backed databases limit string values to 4,068 bytes (`WAL-003`). In-memory databases have no limit. diff --git a/ROADMAP.md b/ROADMAP.md index 648ec55de..aaf26a707 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -93,7 +93,8 @@ Scope — format and storage integrity: Scope — query API: -- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`). The binding cursor objects follow. Behind the cursor, the streaming engine (#473) runs every query and produces rows as they are pulled: patterns, expressions and pseudo-attributes (#544), joins and `:db/valid-at` (#545), `not` and `or` (#546), aggregates and windows (#547), rules (#548), and transaction reads (#550); the executor's query path is gone (#551; done on `v3`). `:limit` (#306) is the one #473 acceptance item left, in v3.next. Operators do not spill: memory follows operator state, with an opt-in per-query limit (`QRY-012`, #549). +- Cursor API: `Minigraf::query` and `PreparedQuery::query` return an owned `Cursor` that delivers rows in batches, with the answer fixed at open (#432; Rust API done on `v3`). The binding cursor objects follow. Behind the cursor, the streaming engine (#473) runs every query and produces rows as they are pulled: patterns, expressions and pseudo-attributes (#544), joins and `:db/valid-at` (#545), `not` and `or` (#546), aggregates and windows (#547), rules (#548), and transaction reads (#550); the executor's query path is gone (#551; done on `v3`). Operators do not spill: memory follows operator state, with an opt-in per-query limit (`QRY-012`, #549). +- Native `:limit N` / `:offset` for plain `:find` queries (#306), with the 10M-fact acceptance test moved from #473. - Fact-log iterator: `Minigraf::fact_log(&FactFilter)` streams every fact version (assertions and retractions, with `tx_count`, `tx_id` and valid-time bounds) in transaction order or storage order, filtered by attribute, attribute prefix, entity and tx range, with memory bounded by a window rather than the database (#430; Rust API done on `v3`). An open log pins its generation: checkpoints run but do not reuse its pages (#543; `API-013` deprecated). A raw log writer (#431) completes the extract-transform-load path; the binding objects follow with #462. - Read-only open: `OpenOptions::read_only` takes a shared lock, so several readers can hold a file at once, writes nothing to the file or its WAL (a WAL is applied in memory, a v7 file is read without migrating), and refuses writes with `API-014` (#429; Rust API done on `v3`). The bindings' `open_with_options` follows. - Raw log writer: `LogWriter` builds a new database from `FactRecord`s with their `tx_count`, `tx_id`, valid-time windows and assertion flags kept, gaps allowed (a purged transaction stays a hole for `:as-of`), at `.partial` renamed into place by `finish` (#431; Rust API done on `v3`). Completes extract (#429, #430), transform, load. The binding objects follow. @@ -119,7 +120,6 @@ Additive features, kept out of v3.0.0 so the #371 fix is not held up by them: - Rule persistence — design first (#241) - Query profiler (#185) -- Native `:limit N` / `:offset` for plain `:find` queries (#306) - Set membership predicates (#316) - `lag`/`lead` window functions (#182) - Sliding row frames — `:rows N preceding` (#183; builds on #182)