Repository navigation
Description
Summary
Two related concurrency-fairness issues observed while stress-testing MiniGrafDb under many concurrent writer processes against one shared graph file:
- Total starvation with long-held connections. If one process opens a
MiniGrafDbhandle and holds it for an extended series of operations (e.g. a long loop, or — realistically — an MCP server that opens once at startup and keeps the handle for its whole session, astemporal_reasoning'smcp_server.pydoes outside of its explicit release windows), other processes trying to open the same file can be starved completely for the holder's entire lifetime, with no fairness/queueing to give them a turn. - High timeout rate even with correct short-lived connections. When every process does the right thing — open, do one operation, release, immediately loop — sustained concurrency still produces a large fraction of simple lock-acquisition timeouts rather than fair queued access.
Neither of these causes corruption or data loss (verified — see below), but both represent real scalability/fairness limits that surprised us in a very ordinary usage pattern: multiple independent processes/sessions (e.g. multiple agent sessions, as documented in project-minigraf/temporal_reasoning#108) legitimately sharing one graph file.
Reproduction 1: total starvation (long-held connections)
24 worker processes, each opening MiniGrafDb.open(path) once and then looping 2000 mixed transact/query/retract operations on that single held handle (i.e. never releasing until the whole run finishes) — a plausible pattern for anyone treating the handle like a normal DB connection pool member.
Result: 23 of 24 workers got zero operations done. Each blocked in its own open-retry loop (with a generous ~38s cumulative backoff budget) until it gave up, never once acquiring the lock, because the 24th worker held it continuously for its entire 2000-op run. Only that one worker's operations succeeded (matching almost exactly its expected ~1500 transacts at a 75% transact-mix ratio: 1506 actual commits). The 23 starved workers logged exactly one "open"-phase lock-timeout error each (23 total), never reaching their operation loop at all.
Reproduction 2: timeout rate under correct usage
Same 24 workers, but fixed to open/execute-one-op/release/repeat (matching the pattern temporal_reasoning's own code documents as necessary — see its comments "release file lock while enumerating commits", "release file lock between commits"), 1200 ops each:
- 14,656 operations succeeded, 9,122 timed out waiting for the lock (each worker's own retry budget: exponential backoff up to ~8s max wait).
- That's a ~38% timeout rate under 24-way concurrency — no corruption, no lost data (0/14,656 committed facts missing on final integrity check), just a high failure rate for legitimate, well-behaved callers.
- A separate 30-worker run specifically using near-max-size payloads (forcing page growth on almost every write) saw a similar pattern: 3,702 succeeded, 798 timed out (~18%).
Impact
- Any application that (reasonably) assumes a
MiniGrafDbhandle behaves like a typical pooled DB connection — safe to hold across multiple logical operations — can inadvertently starve every other process touching the same file, with no error or warning surfaced to the holder, only silent timeouts on everyone else. - Even correctly-behaved short-lived-connection callers see a substantial failure rate under realistic multi-agent/multi-session concurrency (e.g. several Claude Code sessions in the same project directory, each running their own
temporal_reasoningMCP server against one shared graph file — a scenario we hit for real, see temporal_reasoning#108). - This compounds with perf: WAL fsync-per-write is a full fsync() with no batching/relaxed-durability escape hatch #302 (WAL fsync-per-write, no batching) — every serialized operation pays a full fsync, so the lock is held longer per-op than it might otherwise need to be, worsening contention.
Suggested investigation focus
- Consider fair (FIFO) lock queueing instead of whatever the current acquire/backoff scheme is, so waiters are served in order rather than however the OS/backoff timing happens to shake out.
- Consider a documented, enforced maximum hold time or an explicit warning/telemetry hook when a handle is held across many operations without releasing, so calling applications can detect they're accidentally starving others.
- This may be a natural companion to perf: WAL fsync-per-write is a full fsync() with no batching/relaxed-durability escape hatch #302 — reducing per-op lock hold time (e.g. batched/relaxed-durability fsync) would directly reduce contention windows for every other waiter.
Environment
- Reproduced via direct
minigraf.MiniGrafDbPython FFI bindings, bypassingtemporal_reasoningentirely. - All scratch files, page size 4096 bytes confirmed via file-size arithmetic.
- Related: Auto-ingest should check for an existing live lock holder before starting, not race for it temporal_reasoning#108 (the real-world scenario — two live sessions sharing one graph file — that motivated this stress test), perf: WAL fsync-per-write is a full fsync() with no batching/relaxed-durability escape hatch #302 (WAL fsync cost, likely compounds contention duration).
Activity
adityamukho commentedon Aug 25, 2026
Minigraf's documented concurrency model is single-writer, one process at a time (see
docs/ERROR_REFERENCE.mdSTG-026), with the kernel-locking work in #317/#304 (and its NFS/Docker/PID-namespace test suites) scoped to making sequential cross-process access safe — never to making concurrent, many-way access fair or high-throughput. PHILOSOPHY.md explicitly calls out designing for client-server coordination as an anti-pattern for an embedded library, and building FIFO fairness/queueing into the file-lock layer would be exactly that.The repro here (24 independent processes contending directly on one shared file) stress-tests a pattern the docs already advise against — STG-026's existing resolution is: "If two services genuinely need the same file, put one in front of the other. Minigraf is embedded, not a server."
Closing as by-design. The real fix for the motivating case (project-minigraf/temporal_reasoning#108, multiple MCP server sessions sharing one graph file) belongs upstream in
temporal_reasoning: an arbiter/single-owner process in front of the shared file, not fairness engineering inside minigraf's lock acquisition path.#302 (WAL fsync batching) already shipped and reduces per-op lock hold time regardless.