From 9baa86390bcb8bc6af1eede96ca48720ed751292 Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Sat, 24 Oct 2026 09:38:53 +0530 Subject: [PATCH 1/5] docs(plan): UDF registration over UniFFI (#180), with #734 first Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015NBT2Yx9JUQKYvv2CvC9py --- .../plans/2026-10-24-180-udf-uniffi.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-24-180-udf-uniffi.md diff --git a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md new file mode 100644 index 0000000000..ea4a6af737 --- /dev/null +++ b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md @@ -0,0 +1,60 @@ +# UDF registration over UniFFI (#180): plan + +Spec: `docs/superpowers/specs/2026-10-24-udf-uniffi-design.md` (merged as #730), called "the spec" below. +Core PRs target `v3` and merge with a merge commit. The `main` backport is squashed. No PR uses a closing keyword: issues fixed on `v3` are closed by hand. #734 stays open until its v2.x patch is published (`main` backport), and #180 closes after the last binding PR. + +| PR | Steps | Repo, base | Branch | +|---|---|---|---| +| 0 | 1–4: `INT-001` guards (#734, spec §5) | minigraf `v3`, then a `main` backport | `fix/734-tx-guards`, `fix/734-tx-guards-main` | +| 1 | 5–13: fallible UDFs, `QRY-014`, registry snapshot, re-entry (spec §3–§5, §7, §8) | minigraf `v3` (+ minigraf-docs `main`) | `feat/180-fallible-udfs` | +| 2 | 14–16: the FFI in the template (spec §6) | minigraf-binding-template `v3` | `feat/180-udf-ffi` | +| 3–6 | 17–20, once per binding | minigraf-python, -java, -android, -swift `v3` | `feat/180-udf-ffi` | + +Tests and coverage run on the bench machine first (`PATH=$HOME/.cargo/bin`). Each step is test first: the test fails for the stated reason before the code changes. Every PR gets a Codex review (`codex exec -s read-only … < /dev/null`) before merge is asked for. + +## Call sites (v3 at 8f28bb4a) + +- Registry holders, all `Arc>`: `db.rs:427` (`Inner`), `executor.rs:126/152/183`, `prepared.rs:85/173`, `browser/mod.rs:30`, `engine/mod.rs:111/163/202/263`. +- Registry read locks on the query path: `engine/mod.rs:235` (`open_rules`, held through `recursive::evaluate`) and `engine/mod.rs:370` (`Stream::pull`, held for one batch). Registration write locks: `db.rs` `register_aggregate_inner`, `register_predicate_inner`. +- Predicate calls: `executor.rs:808` in `eval_expr`. Its only non-test caller outside itself is `engine/filter.rs:159`. +- UDF aggregate calls: `aggregate.rs` `Udf::of` (389), `Udf::add` (399), `Udf::value` (404). Callers: `Agg::new` (440), `Agg::add` (473), `Agg::finish` (514), `Running::new` (574), `Running::add` (590), `Running::value` (600), `windows` (615…, 664, 681), `finish_groups` (528), `engine/aggregate.rs:41`, `engine/window.rs:99`. +- Locks a plain read holds across a UDF call, other than the registry: none. `rules` is read-locked only while `open_rule_query` plans (`executor.rs:372–386`, released at the end of the block), and `FactStorage::snapshot` drops its `data`/`pending` read guards before returning. Step 11 re-checks this against the finished code. +- Unguarded methods (#734): `checkpoint_inner` (`db.rs:1102`), `verify_inner` (1138), `rebuild_indexes_inner` (1183), `prepare_inner` (1329), `PreparedQuery::execute`/`query` (`prepared.rs:106`/`120`). + +## PR 0: #734 + +Worktree `.worktrees/180-udf-ffi`, branch `fix/734-tx-guards` from `origin/v3` (this plan is its first commit). + +1. **Tests** in a new `tests/write_tx_guard_test.rs` (`#![cfg(not(target_arch = "wasm32"))]`). There is one test per method: `checkpoint`, `verify`, `rebuild_indexes`, `prepare`, `PreparedQuery::execute`, `PreparedQuery::query`. Each runs on a file database, with a `WriteTransaction` open on the test's thread, and expects `ErrorCode::Int001`. Each scenario runs in a spawned thread, and the test waits on a channel with `recv_timeout(Duration::from_secs(30))`, so a deadlock fails the test instead of hanging CI. Today `checkpoint`/`verify`/`rebuild_indexes` time out, and the prepared ones return `Ok`. A further test checks that the transaction still commits after each refusal. +2. **Code**: an `is_write_tx_active()` → `bail_coded!(ErrorCode::Int001)` check at the top of each method in step 1's list. It goes before `reject_if_read_only`, matching `execute_inner`. `is_write_tx_active` becomes `pub(crate)` for `prepared.rs`. +3. **Docs**: `docs/ERROR_REFERENCE.md` INT-001 lists every method that raises it. `CHANGELOG.md` v3.0.0 gets a `Fixed` entry. `docs/TEST_COVERAGE.md` and the `CLAUDE.md` test counts are updated. The #421 entry gets "Fix: merged on `v3` in PR #…". +4. **Backport to `main`**: worktree `.worktrees/734-tx-guards-main` from `origin/main`. On v2.x the guard goes on `checkpoint` and the prepared methods (`verify`/`rebuild_indexes` do not exist there), with step 1's tests for those. CHANGELOG `Unreleased` → `Fixed`. #734 stays open until the v2.x patch with it is published (support policy: v2.0.x gets bug fixes until v3.0.0). + +## PR 1: fallible UDFs and the registry snapshot + +Worktree `.worktrees/180-udf-ffi` (reused), branch `feat/180-fallible-udfs` from `origin/v3` after PR 0. + +5. **`UdfError` and `QRY-014`.** `UdfError` (spec §3) goes in `src/query/datalog/functions.rs` and is re-exported from `lib.rs` next to `MinigrafError`. `ErrorCode::Qry014` gets its message ``user function `{}` failed: {}`` and category `Query`, in the registry list and the code↔doc tests (`error_codes_foundation_test.rs`), plus `docs/ERROR_REFERENCE.md`. First check that #306's branch or any open v3 PR has not claimed `QRY-014` (`git grep QRY-014 origin/v3 $(git branch -r --list 'origin/feat/*')`). Test: the code's string, category, and doc entry. +6. **Fallible descriptors.** `PredicateDesc::f: Arc Result + Send + Sync>`, and `UdfOps::init`, `UdfStepFn`, `UdfFinaliseFn` likewise return `Result`. `register_predicate_inner`/`register_aggregate_inner` take the fallible closures. `register_*` wrap the user's closure in `Ok`, and the new `try_register_*` (spec §3) pass it through. The built-in sentinels (`|_| false`) become `|_| Ok(false)`. Test (`udf_test.rs`): every existing UDF test also runs through `try_register_*` with never-failing closures and gets the same rows (one parametrised helper). +7. **Predicate failures.** `pub(crate) enum EvalError { NoValue, Udf(anyhow::Error) }`. `eval_expr` returns `Result`: today's `Err(())` becomes `NoValue`, and a failing predicate becomes `Udf(err_coded!(Qry014, name, e.message()))`. The `?` chains carry both. `filter::eval` maps `NoValue` → `Ok(None)` and `Udf(e)` → `Err(e)`. The unit tests on `eval_expr` (`executor.rs:1994…`) compare against `Err(EvalError::NoValue)` (`EvalError: PartialEq` by variant, under `cfg(test)`). Tests (`udf_test.rs`): a `try_` predicate that fails on one value, with an `AtomicUsize` call counter, in `:where`, a rule body, a `not` body and an `or` branch, through `execute`, `query` + `next_batch` (rows before the failure come first, then the error, then `None`), and `PreparedQuery::execute`. Each asserts `Qry014`, the name and message in the text, and that the counter stops at the failing call. +8. **Aggregate failures.** `Udf::of` → `Result>` (init's error is `QRY-014`), and `Udf::add`/`value` → `Result`. `Agg::add` and `Running::add`/`value` → `Result<…>`, with built-in arms `Ok(…)`. `windows`, `finish_groups`, `engine/aggregate.rs`, `engine/window.rs` propagate with `?`. A failure in `step` stops the query at that value. Tests: `try_` aggregates failing in `init`, `step` and `finalise`, each as a `:find` aggregate and as an `:over` window. Each asserts `Qry014` and that no call follows the failing one. +9. **Registry snapshot.** `pub(crate) struct Functions(RwLock>)`, with `snapshot() -> Result>` (read lock for the clone only; poisoned → `QRY-008`) and `register(|reg| …)` (write lock, `(*arc).clone()`, mutate, swap; poisoned → `API-008`). `FunctionRegistry: Clone`. Every holder in "Call sites" takes `Arc`. The engine's `Stream` and `open`/`open_rules` take the `Arc` snapshot instead. The executor takes it once in `open_query`/`open_rule_query`, before the `QRY-007`/`QRY-010`/`QRY-011` checks, and uses that one snapshot for the checks and the stream. `pull` and `open_rules` no longer lock. Tests: a predicate that calls `db.register_predicate` mid-query returns, where today it deadlocks (step 1's timeout harness). A cursor opened before a registration does not see the new function (`QRY-007` stays the open-time answer; a later query sees it). +10. **Re-entry tests**, one per row of spec §5's table, for both columns, on a file database with pending and committed facts. From inside a predicate: query `execute`, `query` drained, `prepare` + `execute`, `fact_log`, `transact`, `retract`, `rule`, `begin_write` + commit, `checkpoint`, `verify`, `rebuild_indexes`, `register_*`, and a nested query that calls a second UDF. Plain-read column: each runs. A write is not seen by the outer query, and the outer query's rows equal those of the same query without the callback's side effect. Write-transaction column: `INT-001` for each row that says so, and registration runs. All of them go through the timeout harness. +11. **Lock audit.** Check that no `Mutex`/`RwLock` guard is alive at any UDF call: grep `\.read()\|\.write()\|\.lock()` under `src/query`, `src/graph/storage.rs`, `src/storage/cache.rs`, `src/storage/reader.rs`, and trace each guard's scope against `filter::eval` and the aggregate/window call sites. Record the result (file:line per guard, and why it is not held) in the PR body. A guard found alive gets a fix and a re-entry test. +12. **Docs.** `db.rs` doc comments on the four registration methods: the failure rule, the snapshot rule, and re-entry (spec §5 table, as prose). `CHANGELOG.md` v3.0.0 `Added`: the `try_` methods and `UdfError`, `QRY-014`, re-entrant UDFs, and the snapshot rule. `CLAUDE.md` module notes (`functions.rs`: `Functions` snapshot, `UdfError`) and test counts. `docs/TEST_COVERAGE.md`. Docs site: worktree `../minigraf-docs/.worktrees/180`, the UDF section of `content/datalog-reference.md` (and `content/architecture.md` if it names the registry) inside ``. That PR merges after PR 1. +13. **Acceptance.** On the bench: full suite plain and `--features test-hooks`, the coverage gates (`coverage-thresholds.toml`; a new file's floors added if needed), and the release binary size against v3 (spec: a few KiB). Codex review. CI green, then ask to merge. + +## PR 2: the template + +Branch `feat/180-udf-ffi` from `minigraf-binding-template` `v3`. + +14. **Pin** `minigraf` to PR 1's merge commit (git rev, as the other v3 pins). +15. **FFI code** (spec §6) in `src/lib.rs`: `MiniGrafUdfError` with `From`, the three traits under `#[uniffi::export]`, and `MiniGrafDb::register_predicate`/`register_aggregate` over `try_register_*` with `Acc = Arc`. A `Ref` from `finish` that is not a UUID becomes `UdfError` with `to_core`'s message. Rust tests with Rust implementations of the traits: a predicate, an aggregate in `:find` and `:over`, a failing `step` → `MiniGrafError::Query` containing `QRY-014`, a bad `Ref` from `finish`, and a predicate that queries the same `MiniGrafDb`. +16. **Bindgen check**: generate Python, Kotlin and Swift from the built cdylib (README step 5) and confirm that each has an implementable interface for all three traits (Python: a class to subclass; Kotlin: an `interface`; Swift: a `protocol`), that throwing `MiniGrafUdfError` is declared, and note Swift's `Sendable` requirement if the protocol carries one. Put the findings in the README's §6.1 section (lifecycle, threading, cost, re-entry), which every binding copies. + +## PRs 3–6: one per binding (python, java, android, swift) + +17. **Pin** `minigraf` to PR 1's merge. **Copy** step 15's code and Rust tests from the template. +18. **Language tests** in the binding's existing test harness (pytest; the Kotlin/JVM tests in java and android; XCTest in swift): a predicate; an aggregate in `:find` and `:over`; a callback that throws its own exception → the language's `Query` error with `QRY-014`; a callback that runs a query on the same database. Python adds two threads querying with one shared predicate (GIL release and reacquire, spec §6.1). +19. **Consumer test**: `consumer-test/run.sh`'s contract gains one predicate and one aggregate, so the packaged generated code is exercised (`decision_binding_consumer_tests`). **README**: a predicate and an aggregate example, plus the template's §6.1 text. +20. **CI green** on the PR, including package + consumer test. Merge on approval (each binding's merge rule). After the last binding, tick #180's acceptance criteria and close it by hand. From e931b3e16212f8b330c22508908a2a9b936d030a Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Sat, 24 Oct 2026 09:45:37 +0530 Subject: [PATCH 2/5] =?UTF-8?q?docs(spec):=20#180=20covers=20every=20bindi?= =?UTF-8?q?ng:=20C,=20Node=20and=20BrowserDb=20(=C2=A76.2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015NBT2Yx9JUQKYvv2CvC9py --- .../specs/2026-10-24-udf-uniffi-design.md | 50 ++++++++++++++++++- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md b/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md index 97ce0334b3..aaa7a8cb3f 100644 --- a/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md +++ b/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md @@ -1,7 +1,7 @@ # UDF registration over UniFFI (#180) **Milestone:** v3.0.0, branch `v3` (moved from v3.next, 2026-10-24). No change to the `.graph` format or the WAL format. The public Rust API gains two methods and one type; nothing is removed or changed. -**Bindings:** every UniFFI binding: `minigraf-binding-template`, `minigraf-python`, `minigraf-java`, `minigraf-android`, `minigraf-swift`, each on its `v3` branch. `minigraf-c`, `minigraf-node` and `minigraf-wasm` do not use UniFFI and are not covered here. +**Bindings:** every binding, each on its `v3` branch. The UniFFI ones are `minigraf-binding-template`, `minigraf-python`, `minigraf-java`, `minigraf-android` and `minigraf-swift` (§6). The others are `minigraf-c` (cbindgen), `minigraf-node` (napi) and the browser `BrowserDb` (wasm-bindgen, in this repo's `src/browser/`, re-exported by `minigraf-wasm`) (§6.2). The scope was widened from UniFFI only on 2026-10-24. **Status:** draft 2026-10-24; in review. ## 1. Decisions @@ -59,8 +59,9 @@ A Rust closure that panics is not a failing call. It unwinds as it does today. T | Code | Name | Category | Message | |------|------|----------|---------| -| `QRY-014` (new) | User function failed | Query | ``[QRY-014] user function `` failed: `` | +| `QRY-014` (new; see note) | User function failed | Query | ``[QRY-014] user function `` failed: `` | +- Code number: #316's plan (step 18) also claims `QRY-014`. Whichever PR lands second takes the next free `QRY` code and updates this spec. "`QRY-014`" below means the code PR 1 lands with. - The error is raised at the first failing call. No later call of any UDF is made for that query. `Minigraf::execute` and `PreparedQuery::execute` return it. A cursor treats it as any other read error (`cursor.rs` `next_batch`): rows gathered before the failure are returned first, the error comes on the next call, and after it the cursor returns `None`. - Rows already returned stay returned. A cursor's earlier batches are not undone. - Today, an unknown UDF predicate name gives no row. It stays that way, and `QRY-007` at open already rejects an unknown predicate. Today's other "no value" cases are also unchanged: unbound variable, type mismatch, division by zero. @@ -147,6 +148,47 @@ impl MiniGrafDb { - Each call crosses the FFI once, so a UDF over many rows costs one foreign call per value. The README says so. - What a callback may call on the database is §5's table. +### 6.2 Bindings without UniFFI + +Each binding's values cross in its existing value form, the same `{"type", "value"}` shape its fact-log records use. Failures follow the binding's existing error path. Every rule in §6.1 applies. The differences are below. + +**C (`minigraf-c`)** follows SQLite's `sqlite3_create_function`: a function pointer, a `void *user_data` and an optional destructor. Results and errors are set through a context, so no string ownership crosses the boundary. + +```c +typedef struct MiniGrafUdfContext MiniGrafUdfContext; +typedef void (*minigraf_destroy_fn)(void *user_data); + +/* 1 true, 0 false, -1 failed (after minigraf_udf_error). */ +typedef int (*minigraf_predicate_fn)(MiniGrafUdfContext *ctx, void *user_data, const char *value_json); +/* A new accumulator, or NULL when failed. */ +typedef void *(*minigraf_acc_new_fn)(MiniGrafUdfContext *ctx, void *user_data); +/* 0 ok, -1 failed. */ +typedef int (*minigraf_acc_step_fn)(MiniGrafUdfContext *ctx, void *acc, const char *value_json); +/* 0 ok after minigraf_udf_result, -1 failed. */ +typedef int (*minigraf_acc_finish_fn)(MiniGrafUdfContext *ctx, void *acc, int64_t count); + +void minigraf_udf_error(MiniGrafUdfContext *ctx, const char *message); /* copied */ +void minigraf_udf_result(MiniGrafUdfContext *ctx, const char *value_json); /* copied */ + +int minigraf_register_predicate(MiniGrafDb *db, const char *name, minigraf_predicate_fn f, + void *user_data, minigraf_destroy_fn destroy); +int minigraf_register_aggregate(MiniGrafDb *db, const char *name, minigraf_acc_new_fn init, + minigraf_acc_step_fn step, minigraf_acc_finish_fn finish, + minigraf_destroy_fn acc_free, void *user_data, + minigraf_destroy_fn destroy); +``` + +- Both registration calls return 0 or -1, and `minigraf_last_error` gives the message, as the other `int` calls do. `destroy(user_data)` runs when the database drops the function. `acc_free(acc)` runs when the query drops the accumulator: after `finish`, on an error, or on an early `minigraf_cursor_free`. +- A callback that returns -1 without calling `minigraf_udf_error` fails with the message `user function failed`. A `finish` that returns 0 without calling `minigraf_udf_result`, or with JSON that is not a value, fails as well. A `ctx` is valid only during the call it was passed to. +- The core needs `Send + Sync`. The shim's wrapper of `(fn, user_data)` is `unsafe impl Send + Sync`, so the header states the caller's contract: the callbacks and `user_data` must be safe to call from any thread that queries the database. + +**Node (`minigraf-node`)**: `db.registerPredicate(name, (value) => boolean)` and `db.registerAggregate(name, { init: () => acc, step: (acc, value) => acc, finish: (acc, count) => value })`. Accumulator state is a JS value that `step` returns (functional, with no `this`), held by the binding as a napi reference. A thrown exception is a failing call, with its message (or `String(e)`). +- JS can run only on the thread that owns its `napi_env`. Every binding method is synchronous on that thread, so every UDF call happens there. The binding stores the function as a napi reference wrapped in a type marked `Send + Sync`, and records the owning thread's id at registration. A call from any other thread is a failing call, ``called off the JavaScript thread``, and never touches the env. No current code path reaches that check: it guards future async APIs. It is tested by calling the wrapper from a spawned Rust thread in the binding's Rust tests. +- Worker threads each have their own env, and a database object belongs to the env that opened it. A function registered on one is called only on that env's thread. + +**Browser (`BrowserDb`, this repo)**: `registerPredicate(name, fn)` and `registerAggregate(name, { init, step, finish })`, with the same JS shapes as Node and `js_sys::Function`. wasm32 without atomics has one thread, so the wrapper is `unsafe impl Send + Sync` under `cfg(all(target_arch = "wasm32", not(target_feature = "atomics")))`, and a build with atomics does not compile it. `BrowserDb.execute` is `async` only for the IndexedDB flush, and the query and its UDF calls run synchronously before any `await`. Re-entry differs from §5 here. A query holds `inner.borrow()` (`browser/mod.rs` ~125) while it runs. Every method that takes `borrow_mut` (`checkpoint`, `importGraph`, `apply_write` under `execute`) is `async`, and wasm-bindgen-futures runs its body from a microtask, after the outer query has returned. So from inside a UDF, `query` runs synchronously (it takes only `borrow()`), and `registerPredicate`/`registerAggregate` run synchronously (snapshot rule). `execute`, `checkpoint` and `importGraph` return a Promise that starts only after the outer query, and the UDF cannot wait for it. The plan checks that no synchronous method takes `borrow_mut`, with a browser test per method. +- Values reach JS as `{type, value}` objects, built with `js_sys::Object` (no new dependency). `BrowserDb`'s registry (`browser/mod.rs:30`) moves to `Functions` with the core's in PR 1. + ## 7. Tests **Core (`tests/udf_test.rs`)**: @@ -160,6 +202,8 @@ impl MiniGrafDb { **Each binding**: Rust unit tests in `src/lib.rs` with Rust implementations of the three traits. Language tests in pytest, Kotlin/JVM (java and android) and XCTest (swift), covering at least: one predicate, one aggregate (`:find` and `:over`), a callback that throws (→ `Query` error with `QRY-014`), and a callback that runs a query on the same database. The consumer-test contract (`consumer-test/run.sh`) gains one predicate and one aggregate, so the published package's generated callback code is exercised. +**C, Node, browser**: the same language-level cases. C: a C test program in the binding's test harness, plus `destroy`/`acc_free` counts, including an early `minigraf_cursor_free`. Node: the binding's JS tests and the off-thread check. Browser: `#[wasm_bindgen_test]` in `src/browser/` (headless browsers, `wasm.yml`), and the `minigraf-wasm` consumer test. Each consumer test gains one predicate and one aggregate. + ## 8. Documentation - `docs/ERROR_REFERENCE.md`: `QRY-014`. `src/error.rs` registry and tests. @@ -175,5 +219,7 @@ impl MiniGrafDb { 1. **Core, on `v3`**: §3–§5, core tests, `QRY-014`, docs. Docs-site PR merged after it. 2. **`minigraf-binding-template`**, on `v3`: §6 and its Rust tests. 3. **`minigraf-python`, `minigraf-java`, `minigraf-android`, `minigraf-swift`**, each on `v3`: bump the git-rev pin to PR 1's merge, add §6, the language tests and the README section. One PR per binding. +4. **Browser `BrowserDb`**, on `v3` in this repo (Tier 1 from v3.0.0, so first of the non-UniFFI ones), then `minigraf-wasm`'s pin, consumer test and README. +5. **`minigraf-c`**, then **`minigraf-node`**, each on `v3` (§6.2). No change to `.graph`, WAL or the binary-size budget is expected beyond a few KiB. The size gate checks it. From 8c417283b53d66e6d6a53751177dc439351514c2 Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Sat, 24 Oct 2026 09:45:54 +0530 Subject: [PATCH 3/5] =?UTF-8?q?docs(plan,roadmap):=20#180=20in=20every=20b?= =?UTF-8?q?inding=20=E2=80=94=20PRs=207=E2=80=939=20(BrowserDb,=20C,=20Nod?= =?UTF-8?q?e)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015NBT2Yx9JUQKYvv2CvC9py --- CLAUDE.md | 2 +- ROADMAP.md | 2 +- .../plans/2026-10-24-180-udf-uniffi.md | 17 +++++++++++++++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 671f0ae7f8..a60373b3ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -223,7 +223,7 @@ See `docs/TEST_COVERAGE.md` for the full per-file breakdown. Current plan (see `ROADMAP.md` and the milestones): - **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release. -- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, query profiler #185, set membership predicates #316 and UniFFI `PreparedQuery`/UDF registration #181/#180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`. +- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, query profiler #185, set membership predicates #316 and UniFFI `PreparedQuery` #181, UDF registration in every binding #180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`. - **v3.next** (the release after v3.0.0) holds new features (lag/lead #182, sliding row frames #183, rule persistence #241, OPFS `BrowserDb` #441). - Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (`known-issue` label); `data-integrity`/`corruption`/`durability` issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased `v3` branch close on merge (closing keywords do not fire on `v3`, so close them with `gh issue close`) and stay listed in #421 (CONTRIBUTING.md). - Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2. diff --git a/ROADMAP.md b/ROADMAP.md index 3d42eb7652..35a09501ea 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -104,7 +104,7 @@ Scope — features: - Query profiler (#185) - Set membership predicates (#316) - `PreparedQuery` over UniFFI (#181) -- UDF registration over UniFFI (#180) +- UDF registration in every binding: the UniFFI ones, C, Node and the browser `BrowserDb` (#180) - `OpenOptions` exposed over UniFFI through `open_with_options`, so embedders can suppress the close-time checkpoint (#322; done on `v3` in every binding with #465) Scope — release-gate testing and documentation: diff --git a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md index ea4a6af737..05ca15c0a3 100644 --- a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md +++ b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md @@ -1,4 +1,4 @@ -# UDF registration over UniFFI (#180): plan +# UDF registration in every binding (#180): plan Spec: `docs/superpowers/specs/2026-10-24-udf-uniffi-design.md` (merged as #730), called "the spec" below. Core PRs target `v3` and merge with a merge commit. The `main` backport is squashed. No PR uses a closing keyword: issues fixed on `v3` are closed by hand. #734 stays open until its v2.x patch is published (`main` backport), and #180 closes after the last binding PR. @@ -9,6 +9,11 @@ Core PRs target `v3` and merge with a merge commit. The `main` backport is squas | 1 | 5–13: fallible UDFs, `QRY-014`, registry snapshot, re-entry (spec §3–§5, §7, §8) | minigraf `v3` (+ minigraf-docs `main`) | `feat/180-fallible-udfs` | | 2 | 14–16: the FFI in the template (spec §6) | minigraf-binding-template `v3` | `feat/180-udf-ffi` | | 3–6 | 17–20, once per binding | minigraf-python, -java, -android, -swift `v3` | `feat/180-udf-ffi` | +| 7 | 21–22: browser `BrowserDb` (spec §6.2) | minigraf `v3` (+ minigraf-wasm `v3`) | `feat/180-browser-udfs` | +| 8 | 23: C (spec §6.2) | minigraf-c `v3` | `feat/180-udf-ffi` | +| 9 | 24: Node (spec §6.2) | minigraf-node `v3` | `feat/180-udf-ffi` | + +Other sessions: #316 (set values) claims `QRY-014` in its step 18 and touches `functions.rs`, `filter.rs` and `error.rs`, and a session on #181 (prepared queries in the bindings) adds to the same binding `src/lib.rs` files and `src/browser/mod.rs`. Whichever PR lands second rebases, and on a code clash takes the next free `QRY` code (spec §4 note). Tests and coverage run on the bench machine first (`PATH=$HOME/.cargo/bin`). Each step is test first: the test fails for the stated reason before the code changes. Every PR gets a Codex review (`codex exec -s read-only … < /dev/null`) before merge is asked for. @@ -57,4 +62,12 @@ Branch `feat/180-udf-ffi` from `minigraf-binding-template` `v3`. 17. **Pin** `minigraf` to PR 1's merge. **Copy** step 15's code and Rust tests from the template. 18. **Language tests** in the binding's existing test harness (pytest; the Kotlin/JVM tests in java and android; XCTest in swift): a predicate; an aggregate in `:find` and `:over`; a callback that throws its own exception → the language's `Query` error with `QRY-014`; a callback that runs a query on the same database. Python adds two threads querying with one shared predicate (GIL release and reacquire, spec §6.1). 19. **Consumer test**: `consumer-test/run.sh`'s contract gains one predicate and one aggregate, so the packaged generated code is exercised (`decision_binding_consumer_tests`). **README**: a predicate and an aggregate example, plus the template's §6.1 text. -20. **CI green** on the PR, including package + consumer test. Merge on approval (each binding's merge rule). After the last binding, tick #180's acceptance criteria and close it by hand. +20. **CI green** on the PR, including package + consumer test. Merge on approval (each binding's merge rule). + +## PRs 7–9: bindings without UniFFI (spec §6.2) + +21. **Browser, core repo.** `BrowserDb::register_predicate`/`register_aggregate` (`js_name` `registerPredicate`/`registerAggregate`) over `try_register_*`'s internals on `BrowserDbInner.functions` (now `Functions`, PR 1). Values become `{type, value}` objects via `js_sys::Object`/`Reflect`, in the shape `minigraf-wasm`'s docs give for records. A thrown `JsValue` becomes `UdfError` (its `message` property, else `String(e)`). The `Send + Sync` wrapper sits under `cfg(all(target_arch = "wasm32", not(target_feature = "atomics")))`. `#[wasm_bindgen_test]`s: a predicate; an aggregate in `:find` and `:over`; a throw → `QRY-014` in the rejection; `query` from inside a UDF; `registerPredicate` from inside a UDF; `execute`/`checkpoint` started from inside a UDF resolve after the outer query and leave both answers correct. Audit: no synchronous `BrowserDb` method takes `borrow_mut` (grep, with the result in the PR body). Docs-site browser page. +22. **`minigraf-wasm`**: pin to PR 7's merge, consumer test + README (one predicate, one aggregate). +23. **C.** The spec §6.2 functions in `src/lib.rs` (cbindgen regenerates `include/minigraf.h`; the header is never hand-edited). The wrapper over `(fn, user_data, destroy)` is `unsafe impl Send + Sync`, and `Drop` calls `destroy`. An accumulator box calls `acc_free` on drop. `MiniGrafUdfContext` is a stack value holding `Option` result/error slots, valid for one call. Rust tests drive the C ABI with `extern "C"` test callbacks: true/false/-1 with and without `minigraf_udf_error`, `finish` without a result, bad JSON, `destroy`/`acc_free` counts (including early `minigraf_cursor_free`), and re-entry through `minigraf_query` from a callback. Consumer test: a C program with one predicate and one aggregate. Header comments state the threading contract. +24. **Node.** `registerPredicate`/`registerAggregate` per spec §6.2 with napi `FunctionRef`s, plus the owning thread's id. The off-thread check returns `UdfError("called off the JavaScript thread")` without touching the env, tested from a spawned Rust thread. JS tests: a predicate, an aggregate in `:find`/`:over` (functional accumulator), a throw → `QRY-014`, a query from inside a UDF. Consumer test + README. +25. **Close-out.** After PR 9: tick #180's acceptance criteria (one line per binding, with PR links) and close it by hand. From dd34ee6681f128312cd36362810515bd44f97b04 Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Sat, 24 Oct 2026 09:46:15 +0530 Subject: [PATCH 4/5] docs(plan): #734 branch starts after the plan merges Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015NBT2Yx9JUQKYvv2CvC9py --- docs/superpowers/plans/2026-10-24-180-udf-uniffi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md index 05ca15c0a3..a58dd0883e 100644 --- a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md +++ b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md @@ -28,7 +28,7 @@ Tests and coverage run on the bench machine first (`PATH=$HOME/.cargo/bin`). Eac ## PR 0: #734 -Worktree `.worktrees/180-udf-ffi`, branch `fix/734-tx-guards` from `origin/v3` (this plan is its first commit). +Worktree `.worktrees/180-udf-ffi`, branch `fix/734-tx-guards` from `origin/v3` after this plan merges. 1. **Tests** in a new `tests/write_tx_guard_test.rs` (`#![cfg(not(target_arch = "wasm32"))]`). There is one test per method: `checkpoint`, `verify`, `rebuild_indexes`, `prepare`, `PreparedQuery::execute`, `PreparedQuery::query`. Each runs on a file database, with a `WriteTransaction` open on the test's thread, and expects `ErrorCode::Int001`. Each scenario runs in a spawned thread, and the test waits on a channel with `recv_timeout(Duration::from_secs(30))`, so a deadlock fails the test instead of hanging CI. Today `checkpoint`/`verify`/`rebuild_indexes` time out, and the prepared ones return `Ok`. A further test checks that the transaction still commits after each refusal. 2. **Code**: an `is_write_tx_active()` → `bail_coded!(ErrorCode::Int001)` check at the top of each method in step 1's list. It goes before `reject_if_read_only`, matching `execute_inner`. `is_write_tx_active` becomes `pub(crate)` for `prepared.rs`. From 15027be34578991b0ae01e9f9b75426ef1c096ce Mon Sep 17 00:00:00 2001 From: Aditya Mukhopadhyay Date: Sat, 24 Oct 2026 09:47:46 +0530 Subject: [PATCH 5/5] docs(spec,plan): #180 review: binding handle locks, C destroy timing Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015NBT2Yx9JUQKYvv2CvC9py --- docs/superpowers/plans/2026-10-24-180-udf-uniffi.md | 6 +++--- docs/superpowers/specs/2026-10-24-udf-uniffi-design.md | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md index a58dd0883e..b5ac784eaf 100644 --- a/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md +++ b/docs/superpowers/plans/2026-10-24-180-udf-uniffi.md @@ -54,7 +54,7 @@ Worktree `.worktrees/180-udf-ffi` (reused), branch `feat/180-fallible-udfs` from Branch `feat/180-udf-ffi` from `minigraf-binding-template` `v3`. 14. **Pin** `minigraf` to PR 1's merge commit (git rev, as the other v3 pins). -15. **FFI code** (spec §6) in `src/lib.rs`: `MiniGrafUdfError` with `From`, the three traits under `#[uniffi::export]`, and `MiniGrafDb::register_predicate`/`register_aggregate` over `try_register_*` with `Acc = Arc`. A `Ref` from `finish` that is not a UUID becomes `UdfError` with `to_core`'s message. Rust tests with Rust implementations of the traits: a predicate, an aggregate in `:find` and `:over`, a failing `step` → `MiniGrafError::Query` containing `QRY-014`, a bad `Ref` from `finish`, and a predicate that queries the same `MiniGrafDb`. +15. **FFI code** (spec §6) in `src/lib.rs`: `MiniGrafUdfError` with `From`, the three traits under `#[uniffi::export]`, and `MiniGrafDb::register_predicate`/`register_aggregate` over `try_register_*` with `Acc = Arc`. A `Ref` from `finish` that is not a UUID becomes `UdfError` with `to_core`'s message. The handle lock change (spec §6.1 "Handle locks"): every `MiniGrafDb` method locks only to clone the `Minigraf`. Rust tests with Rust implementations of the traits: a predicate, an aggregate in `:find` and `:over`, a failing `step` → `MiniGrafError::Query` containing `QRY-014`, a bad `Ref` from `finish`, and a predicate that queries the same `MiniGrafDb`. 16. **Bindgen check**: generate Python, Kotlin and Swift from the built cdylib (README step 5) and confirm that each has an implementable interface for all three traits (Python: a class to subclass; Kotlin: an `interface`; Swift: a `protocol`), that throwing `MiniGrafUdfError` is declared, and note Swift's `Sendable` requirement if the protocol carries one. Put the findings in the README's §6.1 section (lifecycle, threading, cost, re-entry), which every binding copies. ## PRs 3–6: one per binding (python, java, android, swift) @@ -68,6 +68,6 @@ Branch `feat/180-udf-ffi` from `minigraf-binding-template` `v3`. 21. **Browser, core repo.** `BrowserDb::register_predicate`/`register_aggregate` (`js_name` `registerPredicate`/`registerAggregate`) over `try_register_*`'s internals on `BrowserDbInner.functions` (now `Functions`, PR 1). Values become `{type, value}` objects via `js_sys::Object`/`Reflect`, in the shape `minigraf-wasm`'s docs give for records. A thrown `JsValue` becomes `UdfError` (its `message` property, else `String(e)`). The `Send + Sync` wrapper sits under `cfg(all(target_arch = "wasm32", not(target_feature = "atomics")))`. `#[wasm_bindgen_test]`s: a predicate; an aggregate in `:find` and `:over`; a throw → `QRY-014` in the rejection; `query` from inside a UDF; `registerPredicate` from inside a UDF; `execute`/`checkpoint` started from inside a UDF resolve after the outer query and leave both answers correct. Audit: no synchronous `BrowserDb` method takes `borrow_mut` (grep, with the result in the PR body). Docs-site browser page. 22. **`minigraf-wasm`**: pin to PR 7's merge, consumer test + README (one predicate, one aggregate). -23. **C.** The spec §6.2 functions in `src/lib.rs` (cbindgen regenerates `include/minigraf.h`; the header is never hand-edited). The wrapper over `(fn, user_data, destroy)` is `unsafe impl Send + Sync`, and `Drop` calls `destroy`. An accumulator box calls `acc_free` on drop. `MiniGrafUdfContext` is a stack value holding `Option` result/error slots, valid for one call. Rust tests drive the C ABI with `extern "C"` test callbacks: true/false/-1 with and without `minigraf_udf_error`, `finish` without a result, bad JSON, `destroy`/`acc_free` counts (including early `minigraf_cursor_free`), and re-entry through `minigraf_query` from a callback. Consumer test: a C program with one predicate and one aggregate. Header comments state the threading contract. -24. **Node.** `registerPredicate`/`registerAggregate` per spec §6.2 with napi `FunctionRef`s, plus the owning thread's id. The off-thread check returns `UdfError("called off the JavaScript thread")` without touching the env, tested from a spawned Rust thread. JS tests: a predicate, an aggregate in `:find`/`:over` (functional accumulator), a throw → `QRY-014`, a query from inside a UDF. Consumer test + README. +23. **C.** Handle locks (spec §6.1): every `minigraf_*` call clones the `Minigraf` under `handle.db.lock()` and releases the lock before calling the core. Then the spec §6.2 functions in `src/lib.rs` (cbindgen regenerates `include/minigraf.h`; the header is never hand-edited). The wrapper over `(fn, user_data, destroy)` is `unsafe impl Send + Sync`, and `Drop` calls `destroy`. An accumulator box calls `acc_free` on drop. `MiniGrafUdfContext` is a stack value holding `Option` result/error slots, valid for one call. Rust tests drive the C ABI with `extern "C"` test callbacks: true/false/-1 with and without `minigraf_udf_error`, `finish` without a result, bad JSON, `destroy`/`acc_free` counts (including early `minigraf_cursor_free`), and re-entry through `minigraf_query` from a callback. Consumer test: a C program with one predicate and one aggregate. Header comments state the threading contract. +24. **Node.** Handle locks (spec §6.1): every method clones the `Minigraf` out of `inner` and releases the lock first; `close()` takes the `Option`. Then `registerPredicate`/`registerAggregate` per spec §6.2 with napi `FunctionRef`s, plus the owning thread's id. The off-thread check returns `UdfError("called off the JavaScript thread")` without touching the env, tested from a spawned Rust thread. JS tests: a predicate, an aggregate in `:find`/`:over` (functional accumulator), a throw → `QRY-014`, a query from inside a UDF. Consumer test + README. 25. **Close-out.** After PR 9: tick #180's acceptance criteria (one line per binding, with PR links) and close it by hand. diff --git a/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md b/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md index aaa7a8cb3f..87a85fe316 100644 --- a/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md +++ b/docs/superpowers/specs/2026-10-24-udf-uniffi-design.md @@ -147,6 +147,7 @@ impl MiniGrafDb { - The language runtime's rules apply. Python: UniFFI's ctypes calls release the GIL and the callback re-acquires it. The plan checks this with a test that runs two threads. - Each call crosses the FFI once, so a UDF over many rows costs one foreign call per value. The README says so. - What a callback may call on the database is §5's table. +- **Handle locks.** Every binding today holds its handle's mutex for the whole core call: Python/UniFFI `inner: Arc>`, C `handle.db.lock()` in `minigraf_execute`, Node `self.inner.lock()` in `execute`. A callback that calls back into the same handle would then deadlock in the binding, before the core's rules apply. So each binding locks its handle only to clone the `Minigraf` (it is `Clone`, an `Arc` clone) and calls the core on the clone with no binding lock held. That applies to every method, not only the ones that run queries. Where the handle holds `Option` for `close()` (Node), `close()` sets it to `None`. A call already running keeps its clone, and the file closes when the last clone drops, as with an open cursor today. Each binding's re-entry test covers this. ### 6.2 Bindings without UniFFI @@ -178,7 +179,7 @@ int minigraf_register_aggregate(MiniGrafDb *db, const char *name, minigraf_acc_n minigraf_destroy_fn destroy); ``` -- Both registration calls return 0 or -1, and `minigraf_last_error` gives the message, as the other `int` calls do. `destroy(user_data)` runs when the database drops the function. `acc_free(acc)` runs when the query drops the accumulator: after `finish`, on an error, or on an early `minigraf_cursor_free`. +- Both registration calls return 0 or -1, and `minigraf_last_error` gives the message, as the other `int` calls do. `destroy(user_data)` runs when the last reference to the function drops: after `minigraf_close` and after every cursor, fact log and prepared query opened from that database is freed (each holds a registry snapshot, §5). Until then `user_data` must stay valid. `acc_free(acc)` runs when the query drops the accumulator: after `finish`, on an error, or on an early `minigraf_cursor_free`. - A callback that returns -1 without calling `minigraf_udf_error` fails with the message `user function failed`. A `finish` that returns 0 without calling `minigraf_udf_result`, or with JSON that is not a value, fails as well. A `ctx` is valid only during the call it was passed to. - The core needs `Send + Sync`. The shim's wrapper of `(fn, user_data)` is `unsafe impl Send + Sync`, so the header states the caller's contract: the callbacks and `user_data` must be safe to call from any thread that queries the database.