Repository navigation
Description
Summary
A write (transact) against a large, actively-growing graph file failed with:
msg='Page 284994 out of bounds (total pages: 284993)'
The attempted page index is exactly one past the current total — a page-allocation/growth off-by-one, not a corrupted-file or out-of-range read. The fault is transient: subsequent writes to the same file succeeded immediately afterward with no repair action taken, and simply retrying the failed operation (via temporal_reasoning's resume-from-watermark) completed successfully.
Exact reproduction state
Encountered during real git-history ingestion (temporal_reasoning, arangodb repo, ~52k commits) at:
- File size at failure: 1,167,343,616 bytes exactly.
- Page size: confirmed 4096 bytes (1,167,343,616 / 4096 = 284,996 total pages, zero remainder).
- Layout: 3 header pages + 284,993 data pages = 284,996 total pages — arithmetic is exact, no rounding.
- Attempted access: data page 284994 (i.e.,
current_total_pages + 1) — exactly the next page that growth would need to allocate.
Why this looks like an off-by-one in growth, not a size-of-write issue
The triggering commit was trivial — a 1-line change to a single file (fixed https version of test, UnitTests/HttpInterface/api-admin-spec.rb, 1 insertion, 1 deletion). This rules out "the write was unusually large" as the cause; the failure correlates with the absolute file size / page count crossing a boundary, not with the size of the individual transact.
Evidence of a torn write
The commit's core entity was partially persisted before the fault: querying immediately after showed [:commit/01d654e9238b :date ...] and :hash already present ([:find ?d ?h :where [:commit/01d654e9238b :date ?d] [:commit/01d654e9238b :hash ?h]] succeeded), while the calling client's (temporal_reasoning) own progress counter never advanced for this commit (its _ingest_progress["processed"] counter is only incremented after a commit's entire multi-transact sequence completes without exception) — consistent with the fault occurring partway through a commit's sequence of transacts (commit metadata, then code-structure triples, then dependency edges), after some of them had already succeeded.
Confirmed transient, not permanent
- Immediately after the failure, an unrelated diagnostic
transact(a trivial decision fact) succeeded without issue. - Resuming ingestion via the client's normal resume-from-watermark path succeeded and proceeded past the previously-failing commit and several more with no repeat of the error.
- File size did not need to change for the diagnostic write to succeed (same 1,167,343,616 bytes before and after), so this specific probe didn't confirm growth itself now works — but the subsequent full ingestion resume, which necessarily involved further growth past this exact page count, completed cleanly.
Impact
Low-frequency but real: an in-progress, multi-hour ingestion job hit a hard failure requiring the calling application to detect the terminal error state and manually re-invoke ingestion to resume — data wasn't lost (already-committed facts up to the fault point are intact and queryable), but the affected commit's write was left torn (partial), and the fault is currently unexplained/unpredictable from the caller's side — there's no way to know in advance that a given write might hit this, since it wasn't correlated with write size.
Suggested investigation focus
Since the reported bound is exactly current_total_pages + 1, look at whatever code path handles extending file/mmap size when a write needs a page beyond the current allocation — likely a sequencing issue where the new page index is computed and used before the corresponding file/mmap extension is guaranteed to have completed, or a bounds check using a stale cached total-pages value from just before a concurrent(or same-thread but not-yet-synced) size update.
Activity
adityamukho commentedon Jul 11, 2026
Second occurrence — different page gap, same tool/repo
Hit this same class of failure again, independently, in another
temporal_reasoningingestion run against the arangodb repo (~53k commits):Key difference from the original report
The original report reproduced an exact
N+1pattern (attempted page = current total pages + 1). This occurrence has a gap of 3 pages, not 1:423786 − 423783 = 3This is much larger in absolute page count than the original report's
284994 / 284993(~423k vs ~285k pages), and the non-uniform gap (+3 here vs +1 there) suggests this isn't a strict single-page off-by-one at a fixed boundary. It's more consistent with the original report's own hypothesis: a bounds check or growth step using a stale total-pages value while a multi-page write (e.g. a commit whose multi-transact sequence — metadata, code-structure triples, dependency edges — needed to grow the file by more than one page at once) was still in flight.Context at failure
1624c6a1b87798b62cba2f8284c90904d8d627b1processed_this_run: 22for that server session)Reporting mainly because the variable gap size (+3 vs the original +1) seems like a useful data point for narrowing down the growth/bounds-check race — happy to share more of the client-side (
temporal_reasoning) ingestion logs around this commit if helpful.adityamukho commentedon Jul 11, 2026
Negative result: did not reproduce at smaller scale despite deliberately adversarial conditions
Tried specifically to reproduce this in a small, fast, controlled setting rather than requiring a multi-hour real-repo ingestion. Two attempts, both against fresh scratch graph files, direct
MiniGrafDbFFI calls (bypassingtemporal_reasoning):Both the original report (
284994/284993, ~285k pages / ~1.1GB) and our own independent occurrence reported over in the linked comment above (423786/423783, ~424k pages / ~1.6GB) happened at roughly an order of magnitude more pages than we reached here. So: either this needs substantially larger file sizes to manifest, or it needs sustained duration (multi-hour real ingestion) rather than just short-burst adversarial concurrency, or there's some other condition (specific growth-arithmetic edge case, page-table/index structure that only grows large enough to hit this at scale, etc.) that small/fast repros like ours can't trigger. Flagging as a data point in case it's useful for narrowing down the reproduction conditions — happy to try a longer-duration/larger-scale run if that would help isolate it further.adityamukho commentedon Jul 11, 2026
Third occurrence — real-world recurrence during this same investigation
This bug just recurred a third time on the same real, live ingestion job (arangodb repo, ~53k commits) discussed in my two earlier comments on this issue:
State at failure: 17,775 / 52,948 commits processed,
processed_this_run: 2640, status went straight to a permanent"error"with no auto-retry (the #106 behavior).Comparing all three observed gap patterns on this one job
The third data point is the most unusual: the attempted page index isn't past the reported total by any margin — it's numerically equal to it. If pages are 0-indexed (so a total of 392400 pages means valid indices are 0..392399), then requesting index 392400 is indeed exactly one past the end, consistent with the original's
N / N-1framing just expressed with an off-by-one in which number the error message reports as "total" vs. "attempted" — but if pages are 1-indexed instead, this would mean the header's own recorded total-page-count was already stale/wrong by the time the growth check ran, which is a meaningfully different failure (a stale-metadata read racing a to a size actually already grown to 392400 pages) rather than a pure allocate-one-more-than-you-have miscalculation.Given all three occurrences now show different gaps (+1, +3, +0/exact-equal) on what should be the same underlying arithmetic each time, this continues to look less like a single fixed off-by-one constant and more like a race whose exact manifestation depends on how much concurrent write/growth activity is happening at the moment — consistent with the multi-writer-contention hypothesis from my first comment, and possibly related to project-minigraf/temporal_reasoning#110 (a separate but adjacent concurrency issue I found this session, where concurrent reads during active ingestion writes occasionally observed transiently-inconsistent state).
Recovery: same as before —
minigraf_ingest_git(manual restart, no args) was called again and the job is running normally as of this comment, continuing from the last watermark. No data loss observed.adityamukho commentedon Jul 12, 2026
Fourth occurrence — gap scales dramatically with single-commit write-batch size
A fourth occurrence on the same real ingestion job discussed in my three earlier comments (arangodb repo):
Gap comparison across all four observed occurrences on this one job
This latest gap is roughly four orders of magnitude larger than any previous occurrence, and it happened on a specific, identifiable trigger: the commit being processed at the time was "renamed V8 to V8-3.29.59", a single historical commit touching 16,968 files — by far the largest single-commit write batch encountered in this repo's ~53k-commit history (the next-largest we'd separately noted was a ~6,982-file commit, which processed slowly but without triggering this error).
This is a meaningful new data point for the growth-race hypothesis from my earlier comments: a commit requiring the file to grow by roughly 84,000 pages in one write batch (rather than the handful of pages an ordinary commit needs) produced a gap of matching order of magnitude, rather than the small, near-constant gaps (+1, +3, ~0) seen on ordinary-sized commits. That's consistent with the growth/bounds-check miscalculation scaling with how much the file needs to grow in a single batch, rather than being a fixed small offset — i.e. whatever computes "how many new pages do we need to allocate for this write" may be accumulating error proportional to the requested growth size itself, rather than a constant off-by-one/few.
If useful, this suggests a promising angle for isolating the bug deterministically: construct a single transact (or commit-processing sequence) that requires a similarly large one-shot page-growth (tens of thousands of pages at once) against a freshly-initialized, otherwise-idle database, and see if the gap size in the resulting error scales predictably with the requested growth size. I attempted something in this spirit at smaller scale in my second comment (30-worker concurrent test forcing near-constant page growth, reaching ~11,483 pages) without reproducing it — but that test never forced a single huge growth batch the way this real commit did; it was many small, individually-modest growth events under concurrency, not one enormous one. That distinction (many small growth events vs. one huge one) may be the more important variable to control for in a future repro attempt, more so than total file size or concurrency level.
adityamukho commentedon Aug 9, 2026
A 37-page occurrence — and why it contradicts the size hypothesis
Hit this in
temporal_reasoning's CI (GitHub Actions,test (3.10); 3.11 and 3.12 passed in the same workflow):37 pages. ~151 KB. Gap
+0(attempted index equals total), matching the third-occurrence pattern above.That is four orders of magnitude below every occurrence reported here — and, more importantly, roughly 1,100x smaller than the negative-result stress test in the second comment, which reached 41,868 pages under 30-way concurrency with near-max payloads and produced zero occurrences.
Those two facts together are hard to reconcile with a size- or growth-magnitude-driven cause: the deliberately adversarial large test doesn't reproduce it, and a trivial 151 KB graph does.
Suggestion: this issue may be tracking two different bugs
The four gaps reported here split into two families that look mechanically distinct:
Our 37-page
+0belongs to the first family. Worth separating, because the fourth occurrence is the most dramatic data point and may pull investigation toward the arithmetic path while the small-gap race — the one that fails CI on a trivial graph — goes unexamined. The second comment's repro attempt targeted the concurrency family; the fourth occurrence points at the arithmetic family. They may need different reproductions.A caller-side trigger candidate, which may give you a deterministic repro
temporal_reasoninghas an unserialized write that its own code documents but does not guard:Its
_db_native_lockserializesexecuteandcheckpoint, but not handle destruction. Its ingestion loop sets_db = Noneonce per commit to release the file lock between commits — so a full ingestion performs one unserialized, page-table-invalidating write per commit, interleaved with locked writes on other handles in the same process. The.locksidecar doesn't help here, since it guards across processes, not within one.That fits your own stated hypothesis — "a bounds check using a stale cached total-pages value from just before a concurrent (or same-thread but not-yet-synced) size update" — and it would explain why direct-FFI stress testing with clean per-op release doesn't reproduce it while real ingestion does.
Tracking the caller-side fix as project-minigraf/temporal_reasoning#253. If serializing that drop suppresses the failure, that gives you a precise trigger condition rather than another occurrence report. I'll report the result either way.
Stated as a hypothesis. No one has yet instrumented a failure to confirm it lands on a drop/reopen boundary, and a targeted 20-run A/B on the two failing tests produced zero events on both legs — no signal, not a null result.
One further data point on transience, consistent with everything above: on the PR where this surfaced, commit A passed CI and commit B failed, where the diff between them was two markdown files and one JSON — zero executable code. Re-running the identical failing commit went green. So the outcome flips on scheduling alone.
Make the crash-simulation lock handling portable
fix(storage): never treat a lock held by our own PID as stale (#314)
docs: bring TEST_COVERAGE.md fully current (998 tests)
fix: add close() so a file-backed db can be released deterministically (
chore: release v1.2.3 (core unchanged)
Adopt minigraf 1.2.3 and fix the handle leaks it exposes
Adopt minigraf 1.2.3 and fix the handle leaks behind the flaky page-b…
docs: credit @ocasazza in the CHANGELOG for #317
docs: credit @ocasazza in the CHANGELOG for #317
fix: kernel file locking replaces the PID sidecar (#317, #304) (#332)