Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions .github/workflows/property-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,19 @@ permissions:

jobs:
proptest:
name: Property tests
name: Property tests (${{ matrix.test }})
runs-on: ubuntu-latest
timeout-minutes: 360
strategy:
fail-fast: false
matrix:
include:
- test: property_test
cases: 8000000
# Model-based storage test (#385): each case runs up to 50 steps
# against a file, ~0.1 s per case.
- test: model_based_test
cases: 30000

steps:
- uses: actions/checkout@v4
Expand All @@ -26,9 +36,9 @@ jobs:
shell: bash
run: |
set -o pipefail
cargo test --test property_test -- --nocapture 2>&1 | tee proptest-output.txt
cargo test --test ${{ matrix.test }} -- --nocapture 2>&1 | tee proptest-output.txt
env:
PROPTEST_CASES: 8000000
PROPTEST_CASES: ${{ matrix.cases }}
RUST_BACKTRACE: 1

- name: Record property-test failure
Expand All @@ -37,14 +47,16 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
ISSUE_TITLE: "Nightly property-test failure"
TEST_NAME: ${{ matrix.test }}
CASES: ${{ matrix.cases }}
run: |
report_file="property-test-failure.md"
{
echo "The nightly property-test workflow failed."
echo "The nightly property-test workflow failed in \`$TEST_NAME\`."
echo
echo "- Run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
echo "- Commit: $GITHUB_SHA"
echo "- Reproduce: \`PROPTEST_CASES=8000000 cargo test --test property_test -- --nocapture\`"
echo "- Reproduce: \`PROPTEST_CASES=$CASES cargo test --test $TEST_NAME -- --nocapture\`"
echo
echo "## Test output"
echo
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Tests

- **Golden-file compatibility corpus (#391).** `tests/golden/` holds v7 files written by minigraf 2.0.3, each with a JSON manifest of its CRC32, `tx_count` and expected query results. They cover one checkpoint, several checkpoints, indexes rebuilt on open, a WAL left by a crash, a WAL whose entries were already checkpointed, and same-transaction multi-values (#371). `tests/golden_corpus_test.rs` opens a copy of each file and checks its manifest after open, after a checkpoint and reopen, and after one more write. Golden files are never regenerated, only added; a new `golden-corpus` job in `policy.yml` fails a pull request that changes one. The generator is a standalone crate pinned to `minigraf = "=2.0.3"` (`tests/golden/gen/v7/`). On `v3`, each v7 file must also carry a v8 meta page after its first open, and seven v8 files are frozen alongside them, written by the `v3` branch at `23aa57c` (`tests/golden/gen/v8/`): one checkpoint, seven copy-on-write checkpoints, a crash-left v2 WAL, an already-checkpointed v2 WAL, same-transaction multi-values, a v7 file migrated in place, and long values (a 4,068-byte string stored twice, the 64/65-byte inline boundary, a 1,000-byte attribute and keyword). All #371 queries now pass.
- **Model-based storage test (#385).** `tests/model_based_test.rs` runs random sequences of `transact` (multi-valued batches, transaction- and fact-level valid-time windows, a triple repeated with another window), `retract`, write transactions (committed or rolled back, retracting their own assertions), checkpoints, reopens (default, a 4-page cache, auto-checkpoint every 2 writes) and simulated crashes (a copy of the `.graph` file and WAL taken while the handle is open, so the WAL is replayed) against a file in a temp dir. After every step, `current_tx_count`, `verify`, full rows with windows and tx counts, and entity-, entity+attribute-, attribute-, attribute+value-, value-bound and full-scan queries, under `:as-of N` and `:valid-at T`, must match a plain reference model (one valid-time window per triple, #435; API-011 for two windows of one triple in one transaction). Normal CI runs 24 cases; the nightly property-test workflow runs 30,000.

## v2.0.3 — 2026-10-06

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1347 tests** (1338 passing, 9 ignored; unit + integration + doc).
**1348 tests** (1339 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1347 tests
cargo test # run 1348 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Scope — release-gate testing and documentation:

- Golden-file compatibility corpus for every format version (#391; done on `v3`: six v7 files from 2.0.3 checked after migration, seven v8 files frozen at `23aa57c`)
- SIGKILL crash test that checks committed data through EAVT, AEVT and full scan (#384)
- Model-based test of transact/retract/checkpoint/reopen (#385)
- Model-based test of transact/retract/checkpoint/reopen (#385; done on `v3`: `tests/model_based_test.rs`, 30,000 cases nightly)
- Wider property tests: file-backed, batched, retracts, temporal, joins (#386)
- Persistent fuzz corpus and an operation-sequence target (#387)
- Fault injection across save/WAL/recovery (#390)
Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

87 changes: 87 additions & 0 deletions docs/superpowers/specs/2026-10-07-model-based-test-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Model-based (stateful) storage test — Design

**Issue:** #385, tracker #383
**Milestone:** v3.0.0, branch `v3` (test and CI only; no library change)

## 1. Goal

Run random sequences of writes, checkpoints and reopens against a file-backed
database and compare every read with a plain in-memory reference model. The
storage and index bugs of 2026 (#285, #287, #323, #370, #371) only showed up after
such sequences.

## 2. Shape

`tests/model_based_test.rs`, one `proptest!` over `Vec<Step>`. A step is an
operation plus a probe (the random inputs of the checks that follow it). Plain
proptest replaces `proptest-state-machine`: a `Vec<Step>` already shrinks by
dropping and simplifying steps, every operation is valid in every state, and no
dependency is added.

Pools are small so that steps collide: 4 keyword entities, 3 attributes (one
non-ASCII), 18 values (integers 0–2, `i64::MIN`, booleans, a keyword, a float, short
strings, two 80-byte strings sharing a 40-byte prefix (the v8 long-string key
prefix is 32 bytes), a 3,000-byte string for the value pages, and refs to the
entities). A third of the triples are one hot triple, so that its history grows
past the on-disk scan's seek threshold (8 entries). A case has 1–49 steps.

## 3. Operations

| Op | API | Model |
|---|---|---|
| `Transact` | `execute("(transact {window} [[e a v {window}] …])")`, 1–6 facts, sometimes a repeated triple with another window | one tx; API-011 when a triple gets two windows |
| `Retract` | `execute("(retract [[e a v] …])")` | one tx (always, even if nothing matches) |
| `WriteTx` | `begin_write`, 0–4 transact/retract statements (half the time plus a retraction of a triple it asserts, before or after), then `commit` or `rollback` | one tx if committed and non-empty (a same-tx retraction wins); API-011 as above; nothing on rollback |
| `Checkpoint` | `checkpoint()`, then `verify()` | none |
| `Reopen` | drop and `open_with_options`: default, `page_cache_size(4)`, or `wal_checkpoint_threshold(2)`; then `verify()` | none |
| `Crash` | copy the `.graph` file and WAL while the handle is open, drop it, open the copy (same modes); then `verify()` | none |

`Crash` is the only way to reach WAL replay in-process: dropping a handle always
saves (`PersistentFactStorage::drop` saves when dirty, even with
`wal_checkpoint_threshold(usize::MAX)`), and the file lock forbids leaking it. On
Windows, where locks are mandatory, the copy is taken after the drop.

All handles use `SyncMode::Normal`: the test checks results, not fsync.

Valid-time windows come from a grid: `:valid-from` in {2000, 2002, 2004, 2006},
`:valid-to` in {2003, 2005, 2007, 2200} (January 1st UTC), each optional at the
transaction and the fact level. A fact window that would be empty or inverted
(#436) is rewritten to start in 2000, so every generated window is valid.

## 4. Model

A log of records `(tx, e, a, v, valid_from or tx-time, valid_to, asserted)` and a
transaction counter. At `as-of N`, a triple is live when its newest assertion
with `tx ≤ N` is newer than its newest retraction; its window is that
assertion's (one window per triple, #435). A window that starts at transaction
time is before every grid instant up to 2008, after 2100, and before "now".

## 5. Checks after each step

- `current_tx_count()` equals the model counter.
- Full rows `?e ?a ?v ?vf ?vt ?tc ?ti` with `:any-valid-time`, current and at a
random `:as-of N` (N up to counter + 1): windows and tx counts match the model,
a tx-time `valid_from` equals the fact's tx id, and tx count → tx id is a
function.
- Entity-bound, entity+attribute-bound, attribute-bound, attribute+value-bound,
value-bound and full-scan queries, each under: default (now), `:valid-at T`,
`:as-of N :valid-at T`, `:as-of N`. T comes from a grid that hits every window
bound plus 1999, 2008, 2100 and 2300.

Results are compared as sorted lists of rendered rows; entity UUIDs render as
their pool index, so failure messages carry no `Uuid` (CodeQL rule).

## 6. CI

Normal CI runs 24 cases (default, overridden by `PROPTEST_CASES`). The nightly
property-test job becomes a matrix over `property_test` (8,000,000 cases) and
`model_based_test` (30,000 cases, ~0.12 s each locally), so the two run in
parallel within their own 6-hour limits; the failure issue names the test.

Mutation checks during development: the test fails on each of ignored
retractions, an off-by-one `:as-of`, inclusive `:valid-to`, exclusive
`:valid-from`, a same-tx retraction that loses, a missing API-011, on-disk
retractions ignored, on-disk older transactions not hidden, a seek that skips the
`:as-of` transaction, a WAL replay that skips an entry, and a transaction counter
not restored after replay. Minimized failures become named regression
tests in the same file.
Loading
Loading