Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 40 additions & 1 deletion crates/diagnostics/src/procs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,28 @@ async fn triage_pids(markers: &[&str], always: &[u32]) -> Result<Vec<(u32, bool)
Ok(pids)
}

/// Re-pins a [`triage_pids`] result immediately before its per-pid `/proc`
/// state is read. A pid from the caller-supplied `always` set
/// (`from_snapshot == false`) is trusted unchanged — it is the caller's own
/// process and its identity is not in question. A pid drawn from the table
/// snapshot (`from_snapshot == true`) is re-checked against `markers` and
/// dropped if it no longer matches: the snapshot is stale the moment it is
/// taken, and a pid recycled in between would otherwise attribute an unrelated
/// process's open sockets to our family. This is the same gate
/// [`hang_triage_including`] applies inline; [`socket_join`] shares it so the
/// fd→socket join cannot splice in a foreign process's sockets.
#[cfg(target_os = "linux")]
async fn repin_live(pids: Vec<(u32, bool)>, markers: &[&str]) -> Vec<u32> {
let mut live = Vec::with_capacity(pids.len());
for (pid, from_snapshot) in pids {
if from_snapshot && !still_matches(pid, markers).await {
continue;
}
live.push(pid);
}
live
}

/// `<dest>/proc/sockets.txt`: which of our processes holds which socket, in
/// which state — every `socket:[inode]` fd of the family joined against the
/// `/proc/net` tables.
Expand Down Expand Up @@ -408,6 +430,11 @@ async fn socket_join<W: BundleSink>(
w.skip(path, "no marker-matched processes holding sockets");
return Ok(());
}
// Re-pin before reading any fds: a snapshot pid recycled since the table
// would otherwise splice an unrelated process's sockets into the join. The
// caller's own `always` pids are trusted without a re-read — same gate as
// `hang_triage_including`.
let pids = repin_live(pids, markers).await;
Comment on lines +433 to +437

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Handle an empty set after re-pinning.

When all snapshot PIDs are stale, repin_live returns an empty vector, but socket_join still builds /proc/net tables and emits a header-only file instead of calling w.skip(...). Add the same empty check immediately after Line 437.

Proposed fix
     let pids = repin_live(pids, markers).await;
+    if pids.is_empty() {
+        w.skip(path, "no marker-matched processes holding sockets");
+        return Ok(());
+    }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Re-pin before reading any fds: a snapshot pid recycled since the table
// would otherwise splice an unrelated process's sockets into the join. The
// caller's own `always` pids are trusted without a re-read — same gate as
// `hang_triage_including`.
let pids = repin_live(pids, markers).await;
// Re-pin before reading any fds: a snapshot pid recycled since the table
// would otherwise splice an unrelated process's sockets into the join. The
// caller's own `always` pids are trusted without a re-read — same gate as
// `hang_triage_including`.
let pids = repin_live(pids, markers).await;
if pids.is_empty() {
w.skip(path, "no marker-matched processes holding sockets");
return Ok(());
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/diagnostics/src/procs.rs` around lines 433 - 437, After the repin_live
call in socket_join, check whether pids is empty and immediately call
w.skip(...) using the existing skip behavior, returning before building
/proc/net tables or emitting a header-only file. Preserve the normal socket-join
flow when any PID remains.

let mut tables = Vec::with_capacity(crate::net::PROC_NET_SOCKET_TABLES.len());
for table in crate::net::PROC_NET_SOCKET_TABLES {
if let Ok(text) = tokio::fs::read_to_string(format!("/proc/net/{table}")).await {
Expand All @@ -417,7 +444,7 @@ async fn socket_join<W: BundleSink>(
let sockets = index_by_inode(&tables);

let mut text = String::from("pid\tfd\tinode\ttable\tentry\n");
for (pid, _) in pids {
for pid in pids {
let Ok(mut entries) = tokio::fs::read_dir(format!("/proc/{pid}/fd")).await else {
let _ = writeln!(text, "{pid}\t-\t-\t-\t<fd directory unreadable>");
continue;
Expand Down Expand Up @@ -748,4 +775,16 @@ ffff8880: 00000002 00000000 00010000 0001 01 41002 /run/minimald/ssh.sock
.unwrap();
assert_eq!(pids, vec![(4242, false)]);
}

#[cfg(target_os = "linux")]
#[tokio::test]
async fn repin_live_trusts_caller_pids_and_drops_stale_snapshot_pids() {
// A caller-named (`always`) pid arrives as (pid, false): it is kept
// without any /proc re-read, so pid 4242 — which need not exist —
// survives, proving its identity was never re-pinned. A snapshot pid
// arrives as (pid, true); u32::MAX exceeds any pid_max, so its
// /proc/<pid>/cmdline is unreadable, the re-pin fails, and it drops.
let live = repin_live(vec![(4242, false), (u32::MAX, true)], MARKERS).await;
assert_eq!(live, vec![4242]);
}
}
54 changes: 37 additions & 17 deletions crates/minimald/src/diag.rs
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,15 @@ const MAX_SESSION_FILE_BYTES: u64 = 8 * 1024 * 1024;
const LOG_FILES_MAX: usize = 5;
/// Entry cap for the recursive state-dir listing.
const LISTING_MAX_ENTRIES: usize = 100_000;
/// Ceiling on the whole client-facing transfer. Every collector is deadlined
/// individually, but those deadlines only bound work — a client that holds the
/// channel open and stops reading it parks the pump instead, and through the
/// bounded duplex that stalls collection and finalization behind it. This is
/// the bound on the peer rather than on ourselves.
const STREAM_TIMEOUT: Duration = Duration::from_secs(300);
/// How long a single write to the client may stall before the stream is
/// abandoned. This bounds a *non-reading* peer, not total transfer time: the
/// pump resets it on every write that makes progress, so a slow-but-advancing
/// build — up to fifteen 30s [`diagnostics::COLLECTOR_TIMEOUT`] collectors,
/// ~450s worst case — streams to completion, while a client that stops
/// draining the channel (backpressure fills the duplex and parks the next
/// write) trips it. A whole-transfer wall-clock cap, by contrast, would kill a
/// legitimate slow run and wrongly blame the client for it.
const STREAM_IDLE_TIMEOUT: Duration = Duration::from_secs(60);
Comment on lines +64 to +72

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

cargo tree -i tokio
rg -n 'poll_write|WriteAll' \
  "${CARGO_HOME:-$HOME/.cargo}/registry/src" \
  --glob '*/tokio-*/src/io/util/write_all.rs'

Repository: gominimal/minimal

Length of output: 147


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- crates/minimald/src/diag.rs (relevant lines) ---\n'
sed -n '150,220p' crates/minimald/src/diag.rs

printf '\n--- tokio version in this repo ---\n'
cargo metadata --no-deps --format-version 1 | python3 - <<'PY'
import json,sys
m=json.load(sys.stdin)
for p in m["packages"]:
    if p["name"]=="tokio":
        print(p["name"], p["version"])
PY

printf '\n--- tokio write_all source if present in cargo registry ---\n'
python3 - <<'PY'
import os,glob,sys
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
paths=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
print("\n".join(paths) if paths else "NOT_FOUND")
PY

Repository: gominimal/minimal

Length of output: 246


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the exact call site and surrounding logic.
rg -n 'write_all|timeout\(STREAM_IDLE_TIMEOUT|STREAM_IDLE_TIMEOUT' crates/minimald/src/diag.rs -n -A8 -B8

# Find any local guidance on Tokio or diagnostics behavior.
rg -n 'diagnostics::COLLECTOR_TIMEOUT|STREAM_IDLE_TIMEOUT|write_all' crates -g '!**/target/**'

Repository: gominimal/minimal

Length of output: 22091


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read Tokio's write_all implementation from the local cargo registry if present.
python3 - <<'PY'
import os,glob
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
matches=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
if not matches:
    print("NO_TOKIO_SOURCE")
else:
    print(matches[0])
PY

Repository: gominimal/minimal

Length of output: 171


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- diag.rs excerpt ---'
sed -n '160,210p' crates/minimald/src/diag.rs

echo
echo '--- tokio version from Cargo.lock ---'
rg -n '^name = "tokio"|^version = ' Cargo.lock -A1 -B1

echo
echo '--- local tokio write_all implementation path(s) ---'
python3 - <<'PY'
import os,glob
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
paths=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
for p in paths:
    print(p)
PY

Repository: gominimal/minimal

Length of output: 50373


🌐 Web query:

Tokio AsyncWriteExt::write_all implementation partial writes loop timeout semantics source

💡 Result:

Tokio's AsyncWriteExt::write_all implementation functions by continuously calling the write method on the underlying AsyncWrite type until the entire buffer is exhausted or an error occurs [1][2][3][4]. Regarding your specific areas of inquiry: Partial Writes and Loops The implementation essentially wraps write in a loop [1][2][4]. Because AsyncWrite::poll_write is readiness-based, it may return a partial result (e.g., writing fewer bytes than the length of the provided buffer) [5]. write_all handles this by tracking the remaining data and continuing to call write in subsequent polls until all bytes are written [2][4]. Timeout Semantics write_all does not have built-in timeout semantics [1][2]. If you need a timeout, you must wrap the write_all future in a tokio::time::timeout call. Because write_all is not cancellation-safe, if a timeout occurs, the operation will be aborted mid-progress [1][2][4]. Cancellation Safety write_all is explicitly documented as not cancellation-safe [1][2][4]. If it is used within a tokio::select! block and another branch completes first, the write_all future will be dropped [6]. Any data that was successfully written to the underlying stream remains there, but the future will not resume where it left off [1][2][4]. Subsequent calls to write_all on the same writer will start over from the beginning of the buffer, which can lead to duplicated data if the previous partial write was not accounted for [1][2][4]. Important Considerations - I/O Completion vs. Readiness: Tokio's I/O traits are readiness-based [5]. Even when write_all returns Poll::Ready(Ok(())), it only guarantees that the data has been accepted by the underlying writer (e.g., buffered by the kernel or the runtime), not necessarily that it has been fully persisted to the destination (e.g., disk) [7][8][5]. For files or buffered writers, an explicit flush may be required to ensure completion [7][8][5]. - Errors: write_all will return the first error encountered during its internal loop [1][2][3]. Top results: [1][2][7][6][5]

Citations:


Reset the deadline per write.
At crates/minimald/src/diag.rs:179-180, timeout(STREAM_IDLE_TIMEOUT, writer.write_all(...)) measures the whole buffer write, so a slow-but-progressing client can still be dropped after 60s. Loop on writer.write(...), restart the timeout after each successful partial write, and treat WriteZero as an error.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/diag.rs` around lines 64 - 72, Update the streaming write
logic near the diagnostic pump to loop over the buffer using writer.write rather
than a single timeout-wrapped write_all. Apply timeout(STREAM_IDLE_TIMEOUT, ...)
separately to each write attempt so the deadline resets after every successful
partial write, continue until all bytes are sent, and return an error when a
write reports zero bytes.

/// Duplex buffer between the bundle writer and the channel pump.
const PIPE_BUF: usize = 256 * 1024;
/// The gvproxy management endpoint, reachable only from inside the switch
Expand Down Expand Up @@ -155,18 +158,35 @@ async fn stream_diag_bundle(s: &ServerStateHandle, c: &mut RuChannel<Msg>) -> Re
};

let mut writer = c.make_writer();
// Bounded here rather than around the build: cancelling the pump is safe,
// cancelling the builder is not (see `build_bundle`). On expiry the `drop`
// below turns the build task's next write into `BrokenPipe`, so it unwinds
// through its own error path instead of parking on the full duplex.
let copy_result =
match tokio::time::timeout(STREAM_TIMEOUT, tokio::io::copy(&mut rx, &mut writer)).await {
Ok(r) => r,
Err(_) => Err(std::io::Error::new(
std::io::ErrorKind::TimedOut,
format!("client stopped reading the bundle for {STREAM_TIMEOUT:?}"),
)),
// A hand-rolled pump so the deadline is per-write, not whole-transfer:
// reading from the build task is left unbounded (its collectors self-bound
// via `COLLECTOR_TIMEOUT`, so a slow-but-progressing build must not be
// killed), while each write to the client is bounded. A stalled write means
// the client stopped draining the channel — backpressure that fills the
// duplex and parks collection behind it — which is the non-reading-client
// DoS this bound exists for. The bound is on the pump rather than the build
// because cancelling the pump is safe and cancelling the builder is not
// (see `build_bundle`): on a stall the `drop` below turns the build task's
// next write into `BrokenPipe`, so it unwinds through its own error path
// instead of parking on the full duplex.
let mut buf = vec![0u8; PIPE_BUF];
let copy_result = loop {
let n = match rx.read(&mut buf).await {
Ok(0) => break Ok(()), // EOF: the build task finished and dropped tx.
Ok(n) => n,
Err(e) => break Err(e),
};
match tokio::time::timeout(STREAM_IDLE_TIMEOUT, writer.write_all(&buf[..n])).await {
Ok(Ok(())) => {}
Ok(Err(e)) => break Err(e),
Err(_) => {
break Err(std::io::Error::new(
std::io::ErrorKind::TimedOut,
format!("client read no bundle data for {STREAM_IDLE_TIMEOUT:?}"),
));
}
}
};
let shutdown_result = writer.shutdown().await;
// The read half must be gone before waiting on the build task: if the copy
// failed mid-stream (client vanished) the task is still writing, and only a
Expand Down