Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

Current plan (see `ROADMAP.md` and the milestones):
- **v2.x** (`main`, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release.
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, query profiler #185, set membership predicates #316 and UniFFI `PreparedQuery`/UDF registration #181/#180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.0.0** (format v8 and data integrity, plus `:limit` #306, query profiler #185, set membership predicates #316 and UniFFI `PreparedQuery` #181, UDF registration in every binding #180; tracker #383) is developed on the long-lived `v3` branch. All v3 PRs target `v3`; `main` stays untouched. CI runs on PRs to `v3` the same as on PRs to `main`. Do not merge `main` into `v3` routinely: `main` only gets backports of fixes already on `v3`. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge `main` into `v3` once (merge commit; source conflicts resolved to `v3`, `main`'s v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current `main` is copied to `v2` and `v3` becomes the new `main`.
- **v3.next** (the release after v3.0.0) holds new features (lag/lead #182, sliding row frames #183, rule persistence #241, OPFS `BrowserDb` #441).
- Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (`known-issue` label); `data-integrity`/`corruption`/`durability` issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased `v3` branch close on merge (closing keywords do not fire on `v3`, so close them with `gh issue close`) and stay listed in #421 (CONTRIBUTING.md).
- Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2.
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Scope — features:
- Query profiler (#185)
- Set membership predicates (#316)
- `PreparedQuery` over UniFFI (#181)
- UDF registration over UniFFI (#180)
- UDF registration in every binding: the UniFFI ones, C, Node and the browser `BrowserDb` (#180)
- `OpenOptions` exposed over UniFFI through `open_with_options`, so embedders can suppress the close-time checkpoint (#322; done on `v3` in every binding with #465)

Scope — release-gate testing and documentation:
Expand Down
73 changes: 73 additions & 0 deletions docs/superpowers/plans/2026-10-24-180-udf-uniffi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# UDF registration in every binding (#180): plan

Spec: `docs/superpowers/specs/2026-10-24-udf-uniffi-design.md` (merged as #730), called "the spec" below.
Core PRs target `v3` and merge with a merge commit. The `main` backport is squashed. No PR uses a closing keyword: issues fixed on `v3` are closed by hand. #734 stays open until its v2.x patch is published (`main` backport), and #180 closes after the last binding PR.

| PR | Steps | Repo, base | Branch |
|---|---|---|---|
| 0 | 1–4: `INT-001` guards (#734, spec §5) | minigraf `v3`, then a `main` backport | `fix/734-tx-guards`, `fix/734-tx-guards-main` |
| 1 | 5–13: fallible UDFs, `QRY-014`, registry snapshot, re-entry (spec §3–§5, §7, §8) | minigraf `v3` (+ minigraf-docs `main`) | `feat/180-fallible-udfs` |
| 2 | 14–16: the FFI in the template (spec §6) | minigraf-binding-template `v3` | `feat/180-udf-ffi` |
| 3–6 | 17–20, once per binding | minigraf-python, -java, -android, -swift `v3` | `feat/180-udf-ffi` |
| 7 | 21–22: browser `BrowserDb` (spec §6.2) | minigraf `v3` (+ minigraf-wasm `v3`) | `feat/180-browser-udfs` |
| 8 | 23: C (spec §6.2) | minigraf-c `v3` | `feat/180-udf-ffi` |
| 9 | 24: Node (spec §6.2) | minigraf-node `v3` | `feat/180-udf-ffi` |

Other sessions: #316 (set values) claims `QRY-014` in its step 18 and touches `functions.rs`, `filter.rs` and `error.rs`, and a session on #181 (prepared queries in the bindings) adds to the same binding `src/lib.rs` files and `src/browser/mod.rs`. Whichever PR lands second rebases, and on a code clash takes the next free `QRY` code (spec §4 note).

Tests and coverage run on the bench machine first (`PATH=$HOME/.cargo/bin`). Each step is test first: the test fails for the stated reason before the code changes. Every PR gets a Codex review (`codex exec -s read-only … < /dev/null`) before merge is asked for.

## Call sites (v3 at 8f28bb4a)

- Registry holders, all `Arc<RwLock<FunctionRegistry>>`: `db.rs:427` (`Inner`), `executor.rs:126/152/183`, `prepared.rs:85/173`, `browser/mod.rs:30`, `engine/mod.rs:111/163/202/263`.
- Registry read locks on the query path: `engine/mod.rs:235` (`open_rules`, held through `recursive::evaluate`) and `engine/mod.rs:370` (`Stream::pull`, held for one batch). Registration write locks: `db.rs` `register_aggregate_inner`, `register_predicate_inner`.
- Predicate calls: `executor.rs:808` in `eval_expr`. Its only non-test caller outside itself is `engine/filter.rs:159`.
- UDF aggregate calls: `aggregate.rs` `Udf::of` (389), `Udf::add` (399), `Udf::value` (404). Callers: `Agg::new` (440), `Agg::add` (473), `Agg::finish` (514), `Running::new` (574), `Running::add` (590), `Running::value` (600), `windows` (615…, 664, 681), `finish_groups` (528), `engine/aggregate.rs:41`, `engine/window.rs:99`.
- Locks a plain read holds across a UDF call, other than the registry: none. `rules` is read-locked only while `open_rule_query` plans (`executor.rs:372–386`, released at the end of the block), and `FactStorage::snapshot` drops its `data`/`pending` read guards before returning. Step 11 re-checks this against the finished code.
- Unguarded methods (#734): `checkpoint_inner` (`db.rs:1102`), `verify_inner` (1138), `rebuild_indexes_inner` (1183), `prepare_inner` (1329), `PreparedQuery::execute`/`query` (`prepared.rs:106`/`120`).

## PR 0: #734

Worktree `.worktrees/180-udf-ffi`, branch `fix/734-tx-guards` from `origin/v3` after this plan merges.

1. **Tests** in a new `tests/write_tx_guard_test.rs` (`#![cfg(not(target_arch = "wasm32"))]`). There is one test per method: `checkpoint`, `verify`, `rebuild_indexes`, `prepare`, `PreparedQuery::execute`, `PreparedQuery::query`. Each runs on a file database, with a `WriteTransaction` open on the test's thread, and expects `ErrorCode::Int001`. Each scenario runs in a spawned thread, and the test waits on a channel with `recv_timeout(Duration::from_secs(30))`, so a deadlock fails the test instead of hanging CI. Today `checkpoint`/`verify`/`rebuild_indexes` time out, and the prepared ones return `Ok`. A further test checks that the transaction still commits after each refusal.
2. **Code**: an `is_write_tx_active()` → `bail_coded!(ErrorCode::Int001)` check at the top of each method in step 1's list. It goes before `reject_if_read_only`, matching `execute_inner`. `is_write_tx_active` becomes `pub(crate)` for `prepared.rs`.
3. **Docs**: `docs/ERROR_REFERENCE.md` INT-001 lists every method that raises it. `CHANGELOG.md` v3.0.0 gets a `Fixed` entry. `docs/TEST_COVERAGE.md` and the `CLAUDE.md` test counts are updated. The #421 entry gets "Fix: merged on `v3` in PR #…".
4. **Backport to `main`**: worktree `.worktrees/734-tx-guards-main` from `origin/main`. On v2.x the guard goes on `checkpoint` and the prepared methods (`verify`/`rebuild_indexes` do not exist there), with step 1's tests for those. CHANGELOG `Unreleased` → `Fixed`. #734 stays open until the v2.x patch with it is published (support policy: v2.0.x gets bug fixes until v3.0.0).

## PR 1: fallible UDFs and the registry snapshot

Worktree `.worktrees/180-udf-ffi` (reused), branch `feat/180-fallible-udfs` from `origin/v3` after PR 0.

5. **`UdfError` and `QRY-014`.** `UdfError` (spec §3) goes in `src/query/datalog/functions.rs` and is re-exported from `lib.rs` next to `MinigrafError`. `ErrorCode::Qry014` gets its message ``user function `{}` failed: {}`` and category `Query`, in the registry list and the code↔doc tests (`error_codes_foundation_test.rs`), plus `docs/ERROR_REFERENCE.md`. First check that #306's branch or any open v3 PR has not claimed `QRY-014` (`git grep QRY-014 origin/v3 $(git branch -r --list 'origin/feat/*')`). Test: the code's string, category, and doc entry.
6. **Fallible descriptors.** `PredicateDesc::f: Arc<dyn Fn(&Value) -> Result<bool, UdfError> + Send + Sync>`, and `UdfOps::init`, `UdfStepFn`, `UdfFinaliseFn` likewise return `Result`. `register_predicate_inner`/`register_aggregate_inner` take the fallible closures. `register_*` wrap the user's closure in `Ok`, and the new `try_register_*` (spec §3) pass it through. The built-in sentinels (`|_| false`) become `|_| Ok(false)`. Test (`udf_test.rs`): every existing UDF test also runs through `try_register_*` with never-failing closures and gets the same rows (one parametrised helper).
7. **Predicate failures.** `pub(crate) enum EvalError { NoValue, Udf(anyhow::Error) }`. `eval_expr` returns `Result<Value, EvalError>`: today's `Err(())` becomes `NoValue`, and a failing predicate becomes `Udf(err_coded!(Qry014, name, e.message()))`. The `?` chains carry both. `filter::eval` maps `NoValue` → `Ok(None)` and `Udf(e)` → `Err(e)`. The unit tests on `eval_expr` (`executor.rs:1994…`) compare against `Err(EvalError::NoValue)` (`EvalError: PartialEq` by variant, under `cfg(test)`). Tests (`udf_test.rs`): a `try_` predicate that fails on one value, with an `AtomicUsize` call counter, in `:where`, a rule body, a `not` body and an `or` branch, through `execute`, `query` + `next_batch` (rows before the failure come first, then the error, then `None`), and `PreparedQuery::execute`. Each asserts `Qry014`, the name and message in the text, and that the counter stops at the failing call.
8. **Aggregate failures.** `Udf::of` → `Result<Option<Udf>>` (init's error is `QRY-014`), and `Udf::add`/`value` → `Result`. `Agg::add` and `Running::add`/`value` → `Result<…>`, with built-in arms `Ok(…)`. `windows`, `finish_groups`, `engine/aggregate.rs`, `engine/window.rs` propagate with `?`. A failure in `step` stops the query at that value. Tests: `try_` aggregates failing in `init`, `step` and `finalise`, each as a `:find` aggregate and as an `:over` window. Each asserts `Qry014` and that no call follows the failing one.
9. **Registry snapshot.** `pub(crate) struct Functions(RwLock<Arc<FunctionRegistry>>)`, with `snapshot() -> Result<Arc<FunctionRegistry>>` (read lock for the clone only; poisoned → `QRY-008`) and `register(|reg| …)` (write lock, `(*arc).clone()`, mutate, swap; poisoned → `API-008`). `FunctionRegistry: Clone`. Every holder in "Call sites" takes `Arc<Functions>`. The engine's `Stream` and `open`/`open_rules` take the `Arc<FunctionRegistry>` snapshot instead. The executor takes it once in `open_query`/`open_rule_query`, before the `QRY-007`/`QRY-010`/`QRY-011` checks, and uses that one snapshot for the checks and the stream. `pull` and `open_rules` no longer lock. Tests: a predicate that calls `db.register_predicate` mid-query returns, where today it deadlocks (step 1's timeout harness). A cursor opened before a registration does not see the new function (`QRY-007` stays the open-time answer; a later query sees it).
10. **Re-entry tests**, one per row of spec §5's table, for both columns, on a file database with pending and committed facts. From inside a predicate: query `execute`, `query` drained, `prepare` + `execute`, `fact_log`, `transact`, `retract`, `rule`, `begin_write` + commit, `checkpoint`, `verify`, `rebuild_indexes`, `register_*`, and a nested query that calls a second UDF. Plain-read column: each runs. A write is not seen by the outer query, and the outer query's rows equal those of the same query without the callback's side effect. Write-transaction column: `INT-001` for each row that says so, and registration runs. All of them go through the timeout harness.
11. **Lock audit.** Check that no `Mutex`/`RwLock` guard is alive at any UDF call: grep `\.read()\|\.write()\|\.lock()` under `src/query`, `src/graph/storage.rs`, `src/storage/cache.rs`, `src/storage/reader.rs`, and trace each guard's scope against `filter::eval` and the aggregate/window call sites. Record the result (file:line per guard, and why it is not held) in the PR body. A guard found alive gets a fix and a re-entry test.
12. **Docs.** `db.rs` doc comments on the four registration methods: the failure rule, the snapshot rule, and re-entry (spec §5 table, as prose). `CHANGELOG.md` v3.0.0 `Added`: the `try_` methods and `UdfError`, `QRY-014`, re-entrant UDFs, and the snapshot rule. `CLAUDE.md` module notes (`functions.rs`: `Functions` snapshot, `UdfError`) and test counts. `docs/TEST_COVERAGE.md`. Docs site: worktree `../minigraf-docs/.worktrees/180`, the UDF section of `content/datalog-reference.md` (and `content/architecture.md` if it names the registry) inside `<!-- @since v3.0.0 -->`. That PR merges after PR 1.
13. **Acceptance.** On the bench: full suite plain and `--features test-hooks`, the coverage gates (`coverage-thresholds.toml`; a new file's floors added if needed), and the release binary size against v3 (spec: a few KiB). Codex review. CI green, then ask to merge.

## PR 2: the template

Branch `feat/180-udf-ffi` from `minigraf-binding-template` `v3`.

14. **Pin** `minigraf` to PR 1's merge commit (git rev, as the other v3 pins).
15. **FFI code** (spec §6) in `src/lib.rs`: `MiniGrafUdfError` with `From<uniffi::UnexpectedUniFFICallbackError>`, the three traits under `#[uniffi::export]`, and `MiniGrafDb::register_predicate`/`register_aggregate` over `try_register_*` with `Acc = Arc<dyn MiniGrafAccumulator>`. A `Ref` from `finish` that is not a UUID becomes `UdfError` with `to_core`'s message. The handle lock change (spec §6.1 "Handle locks"): every `MiniGrafDb` method locks only to clone the `Minigraf`. Rust tests with Rust implementations of the traits: a predicate, an aggregate in `:find` and `:over`, a failing `step` → `MiniGrafError::Query` containing `QRY-014`, a bad `Ref` from `finish`, and a predicate that queries the same `MiniGrafDb`.
16. **Bindgen check**: generate Python, Kotlin and Swift from the built cdylib (README step 5) and confirm that each has an implementable interface for all three traits (Python: a class to subclass; Kotlin: an `interface`; Swift: a `protocol`), that throwing `MiniGrafUdfError` is declared, and note Swift's `Sendable` requirement if the protocol carries one. Put the findings in the README's §6.1 section (lifecycle, threading, cost, re-entry), which every binding copies.

## PRs 3–6: one per binding (python, java, android, swift)

17. **Pin** `minigraf` to PR 1's merge. **Copy** step 15's code and Rust tests from the template.
18. **Language tests** in the binding's existing test harness (pytest; the Kotlin/JVM tests in java and android; XCTest in swift): a predicate; an aggregate in `:find` and `:over`; a callback that throws its own exception → the language's `Query` error with `QRY-014`; a callback that runs a query on the same database. Python adds two threads querying with one shared predicate (GIL release and reacquire, spec §6.1).
19. **Consumer test**: `consumer-test/run.sh`'s contract gains one predicate and one aggregate, so the packaged generated code is exercised (`decision_binding_consumer_tests`). **README**: a predicate and an aggregate example, plus the template's §6.1 text.
20. **CI green** on the PR, including package + consumer test. Merge on approval (each binding's merge rule).

## PRs 7–9: bindings without UniFFI (spec §6.2)

21. **Browser, core repo.** `BrowserDb::register_predicate`/`register_aggregate` (`js_name` `registerPredicate`/`registerAggregate`) over `try_register_*`'s internals on `BrowserDbInner.functions` (now `Functions`, PR 1). Values become `{type, value}` objects via `js_sys::Object`/`Reflect`, in the shape `minigraf-wasm`'s docs give for records. A thrown `JsValue` becomes `UdfError` (its `message` property, else `String(e)`). The `Send + Sync` wrapper sits under `cfg(all(target_arch = "wasm32", not(target_feature = "atomics")))`. `#[wasm_bindgen_test]`s: a predicate; an aggregate in `:find` and `:over`; a throw → `QRY-014` in the rejection; `query` from inside a UDF; `registerPredicate` from inside a UDF; `execute`/`checkpoint` started from inside a UDF resolve after the outer query and leave both answers correct. Audit: no synchronous `BrowserDb` method takes `borrow_mut` (grep, with the result in the PR body). Docs-site browser page.
22. **`minigraf-wasm`**: pin to PR 7's merge, consumer test + README (one predicate, one aggregate).
23. **C.** Handle locks (spec §6.1): every `minigraf_*` call clones the `Minigraf` under `handle.db.lock()` and releases the lock before calling the core. Then the spec §6.2 functions in `src/lib.rs` (cbindgen regenerates `include/minigraf.h`; the header is never hand-edited). The wrapper over `(fn, user_data, destroy)` is `unsafe impl Send + Sync`, and `Drop` calls `destroy`. An accumulator box calls `acc_free` on drop. `MiniGrafUdfContext` is a stack value holding `Option<String>` result/error slots, valid for one call. Rust tests drive the C ABI with `extern "C"` test callbacks: true/false/-1 with and without `minigraf_udf_error`, `finish` without a result, bad JSON, `destroy`/`acc_free` counts (including early `minigraf_cursor_free`), and re-entry through `minigraf_query` from a callback. Consumer test: a C program with one predicate and one aggregate. Header comments state the threading contract.
24. **Node.** Handle locks (spec §6.1): every method clones the `Minigraf` out of `inner` and releases the lock first; `close()` takes the `Option`. Then `registerPredicate`/`registerAggregate` per spec §6.2 with napi `FunctionRef`s, plus the owning thread's id. The off-thread check returns `UdfError("called off the JavaScript thread")` without touching the env, tested from a spawned Rust thread. JS tests: a predicate, an aggregate in `:find`/`:over` (functional accumulator), a throw → `QRY-014`, a query from inside a UDF. Consumer test + README.
25. **Close-out.** After PR 9: tick #180's acceptance criteria (one line per binding, with PR links) and close it by hand.
Loading
Loading