Skip to content

FileLock refuses stale lock under PID namespaces (Kubernetes): every container is PID 1 #317

Closed

Description

@ocasazza

Summary

FileLock::acquire (src/storage/backend/file.rs) guards a .graph database with a sidecar <db>.graph.lock file containing the holder's PID. To fix #304, acquire refuses when the recorded holder PID equals its own PID, and only reclaims the lock when the holder is a different, dead process (is_process_alive checks /proc/<pid>).

Under Linux PID namespaces this heuristic breaks: in Kubernetes every container's main process is PID 1. When a pod dies and a replacement pod mounts the same persistent volume, the stale lock file says "PID 1" and the new container is also PID 1, so:

  1. pid == our_pid → acquire bails with "Database is already open in this process", and
  2. is_process_alive(1) is always true, so the lock never self-heals.

The same aliasing applies to any in-namespace PID number, not just 1: a dead holder's PID can collide with a live process in the new container.

Impact

The database file is permanently bricked after any pod restart on a persistent volume, until someone manually deletes the .graph.lock file. We hit this in production: every rollout or crash of a stateful pod embedding minigraf required manual intervention to delete the lock before the new pod could start serving.

Root cause

A PID number identifies a process only within one PID namespace. The lock check conflates "same PID number" with "same live process", which is only valid within a single namespace lifetime. There is no per-process-incarnation token recorded in the lock, so a dead namespaced predecessor is indistinguishable from a genuinely live same-process handle (the #304 case, which must keep being refused).

Proposed fix

Record the holder's process start time (field 22 of /proc/<pid>/stat, clock ticks since boot) alongside the PID, as <pid>:<starttime>, on procfs platforms. Start times are unique across process incarnations sharing a PID number, so:

  • In the pid == our_pid branch: a recorded start time that differs from our own means the holder is a dead namespaced predecessor → treat as stale (remove + retry, same as the dead-foreign-PID path). A matching, absent, or unreadable start time keeps the Transient 'Page N out of bounds (total pages: N-1)' write failure at large file sizes #304 refusal exactly as-is.
  • A live foreign PID whose start time differs from the recorded one is likewise not the lock's author → stale.
  • Lock files with a bare <pid> (written by older minigraf versions) still parse, and non-procfs platforms (macOS/Windows) keep today's conservative behavior.

A PR implementing this, with regression tests covering all three cases (wrong start time → reclaimed, own start time → refused, legacy bare PID → refused), is incoming.

Activity

  1. added theissue type on Aug 19, 2026
  2. adityamukho commented on Aug 21, 2026

    @adityamukho
    Collaborator

    Your report is correct. I accept this issue.

    I examined the code in src/storage/backend/file.rs. The acquire function writes only the PID to the lock file (line 47). The function then refuses the lock when pid == our_pid (lines 55 to 68). A container that receives SIGKILL does not run Drop. Thus the lock file stays on the disk and contains the value 1. The replacement container also has PID 1. The function therefore refuses the lock permanently.

    Your statement about #304 is also correct. That refusal exists because #304 was a second open of one file in one process.

    One correction: cause (2) is not necessary. In the pid == our_pid branch, the code stops before it reaches is_process_alive. Cause (2) applies only when the two PIDs are different. This does not change your conclusion.

    I discuss the fix in #318. I will not merge that mechanism, because the start time cannot show liveness across PID namespaces. This issue stays open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions