Skip to content

Lock acquisition is unfair under concurrency: long-held connections can fully starve others; even correct per-op release sees ~38% timeout rate at 24 writers #309

Closed

Description

@adityamukho

Summary

Two related concurrency-fairness issues observed while stress-testing MiniGrafDb under many concurrent writer processes against one shared graph file:

  1. Total starvation with long-held connections. If one process opens a MiniGrafDb handle and holds it for an extended series of operations (e.g. a long loop, or — realistically — an MCP server that opens once at startup and keeps the handle for its whole session, as temporal_reasoning's mcp_server.py does outside of its explicit release windows), other processes trying to open the same file can be starved completely for the holder's entire lifetime, with no fairness/queueing to give them a turn.
  2. High timeout rate even with correct short-lived connections. When every process does the right thing — open, do one operation, release, immediately loop — sustained concurrency still produces a large fraction of simple lock-acquisition timeouts rather than fair queued access.

Neither of these causes corruption or data loss (verified — see below), but both represent real scalability/fairness limits that surprised us in a very ordinary usage pattern: multiple independent processes/sessions (e.g. multiple agent sessions, as documented in project-minigraf/temporal_reasoning#108) legitimately sharing one graph file.

Reproduction 1: total starvation (long-held connections)

24 worker processes, each opening MiniGrafDb.open(path) once and then looping 2000 mixed transact/query/retract operations on that single held handle (i.e. never releasing until the whole run finishes) — a plausible pattern for anyone treating the handle like a normal DB connection pool member.

Result: 23 of 24 workers got zero operations done. Each blocked in its own open-retry loop (with a generous ~38s cumulative backoff budget) until it gave up, never once acquiring the lock, because the 24th worker held it continuously for its entire 2000-op run. Only that one worker's operations succeeded (matching almost exactly its expected ~1500 transacts at a 75% transact-mix ratio: 1506 actual commits). The 23 starved workers logged exactly one "open"-phase lock-timeout error each (23 total), never reaching their operation loop at all.

Reproduction 2: timeout rate under correct usage

Same 24 workers, but fixed to open/execute-one-op/release/repeat (matching the pattern temporal_reasoning's own code documents as necessary — see its comments "release file lock while enumerating commits", "release file lock between commits"), 1200 ops each:

  • 14,656 operations succeeded, 9,122 timed out waiting for the lock (each worker's own retry budget: exponential backoff up to ~8s max wait).
  • That's a ~38% timeout rate under 24-way concurrency — no corruption, no lost data (0/14,656 committed facts missing on final integrity check), just a high failure rate for legitimate, well-behaved callers.
  • A separate 30-worker run specifically using near-max-size payloads (forcing page growth on almost every write) saw a similar pattern: 3,702 succeeded, 798 timed out (~18%).

Impact

  • Any application that (reasonably) assumes a MiniGrafDb handle behaves like a typical pooled DB connection — safe to hold across multiple logical operations — can inadvertently starve every other process touching the same file, with no error or warning surfaced to the holder, only silent timeouts on everyone else.
  • Even correctly-behaved short-lived-connection callers see a substantial failure rate under realistic multi-agent/multi-session concurrency (e.g. several Claude Code sessions in the same project directory, each running their own temporal_reasoning MCP server against one shared graph file — a scenario we hit for real, see temporal_reasoning#108).
  • This compounds with perf: WAL fsync-per-write is a full fsync() with no batching/relaxed-durability escape hatch #302 (WAL fsync-per-write, no batching) — every serialized operation pays a full fsync, so the lock is held longer per-op than it might otherwise need to be, worsening contention.

Suggested investigation focus

  • Consider fair (FIFO) lock queueing instead of whatever the current acquire/backoff scheme is, so waiters are served in order rather than however the OS/backoff timing happens to shake out.
  • Consider a documented, enforced maximum hold time or an explicit warning/telemetry hook when a handle is held across many operations without releasing, so calling applications can detect they're accidentally starving others.
  • This may be a natural companion to perf: WAL fsync-per-write is a full fsync() with no batching/relaxed-durability escape hatch #302 — reducing per-op lock hold time (e.g. batched/relaxed-durability fsync) would directly reduce contention windows for every other waiter.

Environment

Activity

  1. modified the milestones: , v1.3.0 on Aug 24, 2026
  2. adityamukho commented on Aug 25, 2026

    @adityamukho
    CollaboratorAuthor

    Minigraf's documented concurrency model is single-writer, one process at a time (see docs/ERROR_REFERENCE.md STG-026), with the kernel-locking work in #317/#304 (and its NFS/Docker/PID-namespace test suites) scoped to making sequential cross-process access safe — never to making concurrent, many-way access fair or high-throughput. PHILOSOPHY.md explicitly calls out designing for client-server coordination as an anti-pattern for an embedded library, and building FIFO fairness/queueing into the file-lock layer would be exactly that.

    The repro here (24 independent processes contending directly on one shared file) stress-tests a pattern the docs already advise against — STG-026's existing resolution is: "If two services genuinely need the same file, put one in front of the other. Minigraf is embedded, not a server."

    Closing as by-design. The real fix for the motivating case (project-minigraf/temporal_reasoning#108, multiple MCP server sessions sharing one graph file) belongs upstream in temporal_reasoning: an arbiter/single-owner process in front of the shared file, not fairness engineering inside minigraf's lock acquisition path.

    #302 (WAL fsync batching) already shipped and reduces per-op lock hold time regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions