Repository navigation
sandbox2 + op: CS-compatible nested-sandbox plumbing - #228
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe PR adds stdout tail capture (last ~4 KiB) to sandbox execution errors. The capture flows through updated error types at both sandbox2 and mctx layers, with a rolling tail buffer maintained during execution. A separate /proc remediation feature optionally bind-mounts and symlinks cache directories when enabled. ChangesSandbox stdout capture and /proc remediation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/mctx/src/error.rs (1)
17-27:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRender the captured streams on the default reporting path.
report_to()still falls back toDisplayforError::Execution, andDisplayonly prints the exit summary. As written, the newstdouttail never reaches the main user-facing error path, and the existingstderris still hidden too.Proposed fix
- Error::Execution { - idx, code, reason, .. - } if *code == 125 => write!(f, "invocation {} failed: {}", idx, reason,), - Error::Execution { idx, code, .. } => { - write!(f, "invocation {} failed: exit code {}", idx, code,) - } + Error::Execution { + idx, + code, + reason, + stderr, + stdout, + } => { + if *code == 125 { + write!(f, "invocation {idx} failed: {reason}")?; + } else { + write!(f, "invocation {idx} failed: exit code {code}")?; + } + if !stderr.is_empty() { + write!(f, "\nstderr:\n{stderr}")?; + } + if !stdout.is_empty() { + write!(f, "\nstdout:\n{stdout}")?; + } + Ok(()) + }Also applies to: 235-246
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mctx/src/error.rs` around lines 17 - 27, The Execution variant's captured stdout/stderr tails are never shown because Error::Execution is only rendered via Display which prints only the exit summary; update the Display impl for Error (or the report_to() match arm that handles Error::Execution) to include the Execution struct's stderr and stdout fields (e.g., Execution.stderr and Execution.stdout) on the default reporting path, labeling them (e.g., "stderr:" / "stdout:") and only emitting non-empty tails (or truncating long tails) so that both the existing stderr and the new stdout tail reach the main user-facing error output.
🧹 Nitpick comments (1)
crates/orchestrator/src/local_backend.rs (1)
432-450: 🏗️ Heavy liftReplace the 11-argument constructor with a typed config/builder API.
Lines 438-450 are now highly order-sensitive and require lint suppression. A config struct/builder would make optional sandbox injection explicit and safer at call sites.
As per coding guidelines,
**/*.rs: "Use the builder pattern for types with more than a couple of fields or any optional config, in the owned style:pub fn with_x(mut self, x: T) -> Self" and "Justify any#[allow(...)]attributes with a comment explaining why the lint is suppressed".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/orchestrator/src/local_backend.rs` around lines 432 - 450, The constructor new_orchestrator_with_env has too many positional arguments and uses #[allow(clippy::too_many_arguments)]; replace it with a typed config/builder: introduce a new OrchestratorConfig (or NewOrchestratorBuilder) struct capturing fields (output_base, remote_cache, sf, num_concurrent_builds, graph, cache, log_sink, ot, cancel, extra_env_vars, extra_rootfs) and implement owned-style builder methods like with_remote_cache(...), with_extra_env_vars(...), with_extra_rootfs(...), etc., and a build_orchestrator(self) -> Result<Orchestrator<Self>, Error> that performs the construction currently in new_orchestrator_with_env; remove the clippy allow and keep public API ergonomic by providing a convenience Orchestrator::new_with_config(config) that delegates to the builder.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/op/src/specs.rs`:
- Around line 285-288: The for-loop in SpecBuild::run currently consumes
self.extra_rootfs with drain(..), which drops entries for subsequent calls;
instead iterate non-destructively (e.g., iterate over &self.extra_rootfs and
clone or otherwise copy each entry) and insert into rootfs, so extra_rootfs
remains intact between runs; update the loop that inserts SandboxMapped::Dir to
use a borrow-based iteration (referencing extra_rootfs and SandboxMapped::Dir)
rather than drain(..).
In `@crates/sandbox2/src/lib.rs`:
- Around line 441-443: The current code treats any set value of
MINIMAL_SANDBOX_BIND_PROC as true; change the check so it only enables
container.bindmount_rw("/proc", "/proc") and
container.runctl(hakoniwa::Runctl::MountFallback) when the environment variable
is explicitly set to "1". Locate the block that reads
std::env::var_os("MINIMAL_SANDBOX_BIND_PROC") and replace the truthy presence
check with an explicit value comparison against "1" (preserving use of
container.bindmount_rw and container.runctl with
hakoniwa::Runctl::MountFallback).
---
Outside diff comments:
In `@crates/mctx/src/error.rs`:
- Around line 17-27: The Execution variant's captured stdout/stderr tails are
never shown because Error::Execution is only rendered via Display which prints
only the exit summary; update the Display impl for Error (or the report_to()
match arm that handles Error::Execution) to include the Execution struct's
stderr and stdout fields (e.g., Execution.stderr and Execution.stdout) on the
default reporting path, labeling them (e.g., "stderr:" / "stdout:") and only
emitting non-empty tails (or truncating long tails) so that both the existing
stderr and the new stdout tail reach the main user-facing error output.
---
Nitpick comments:
In `@crates/orchestrator/src/local_backend.rs`:
- Around line 432-450: The constructor new_orchestrator_with_env has too many
positional arguments and uses #[allow(clippy::too_many_arguments)]; replace it
with a typed config/builder: introduce a new OrchestratorConfig (or
NewOrchestratorBuilder) struct capturing fields (output_base, remote_cache, sf,
num_concurrent_builds, graph, cache, log_sink, ot, cancel, extra_env_vars,
extra_rootfs) and implement owned-style builder methods like
with_remote_cache(...), with_extra_env_vars(...), with_extra_rootfs(...), etc.,
and a build_orchestrator(self) -> Result<Orchestrator<Self>, Error> that
performs the construction currently in new_orchestrator_with_env; remove the
clippy allow and keep public API ergonomic by providing a convenience
Orchestrator::new_with_config(config) that delegates to the builder.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ba17cb33-c0da-48e4-a9bd-98bf4f2bde46
📒 Files selected for processing (6)
crates/mctx/src/error.rscrates/op/src/patched.rscrates/op/src/specs.rscrates/orchestrator/src/local_backend.rscrates/sandbox2/src/error.rscrates/sandbox2/src/lib.rs
6c62ce9 to
f2038c4
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
crates/sandbox2/src/lib.rs (1)
441-444:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winHonor the documented
=1opt-in.The check
std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some()enables the/procbind mount for any value, soMINIMAL_SANDBOX_BIND_PROC=0orMINIMAL_SANDBOX_BIND_PROC=falsestill flips behavior. The comment on line 410-411 documents=1as the expected opt-in.Proposed fix
- if std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() { + if matches!( + std::env::var("MINIMAL_SANDBOX_BIND_PROC").as_deref(), + Ok("1") + ) { container.bindmount_rw("/proc", "/proc"); container.runctl(hakoniwa::Runctl::MountFallback); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/sandbox2/src/lib.rs` around lines 441 - 444, The current check uses std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() which treats any set value (including "0" or "false") as enabled; change it to only opt-in when the env var equals "1". Update the condition around the container.bindmount_rw("/proc", "/proc") and container.runctl(hakoniwa::Runctl::MountFallback) calls to explicitly compare the environment variable value to "1" (e.g., read the var and compare string-wise) so only MINIMAL_SANDBOX_BIND_PROC=1 enables the /proc bind mount.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@crates/sandbox2/src/lib.rs`:
- Around line 441-444: The current check uses
std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() which treats any set
value (including "0" or "false") as enabled; change it to only opt-in when the
env var equals "1". Update the condition around the
container.bindmount_rw("/proc", "/proc") and
container.runctl(hakoniwa::Runctl::MountFallback) calls to explicitly compare
the environment variable value to "1" (e.g., read the var and compare
string-wise) so only MINIMAL_SANDBOX_BIND_PROC=1 enables the /proc bind mount.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 66579a88-a2d1-4be2-a8e5-23b6f4762d15
📒 Files selected for processing (4)
crates/mctx/src/error.rscrates/minimal/src/cmd_run.rscrates/sandbox2/src/error.rscrates/sandbox2/src/lib.rs
f2038c4 to
ab9ea9f
Compare
|
Thanks for the careful review! Addressed all 4 comments — force-pushed 1. Display impl rendering ( 2. 3. 4. 11-positional-arg constructor / builder pattern ( Mirrored these same fixes on the local |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/sandbox2/src/lib.rs (1)
726-731: ⚡ Quick winDon’t silently drop reader-thread failures during cancellation.
The
if letjoin pattern drops both panic andErr(Error)outcomes, which hides teardown failures when cancellation happens.As per coding guidelines: "Never swallow errors; do not use `let _ = result;` or `.ok()` discards without a comment justifying it".Proposed fix
- if let Ok(Ok((f, _tail))) = stdout_thread.join() { - self.stdout = f; - } - if let Ok(Ok((f, _tail))) = stderr_thread.join() { - self.stderr = f; - } + match stdout_thread.join() { + Ok(Ok((f, _tail))) => self.stdout = f, + Ok(Err(e)) => tracing::warn!("stdout reader thread failed during cancellation: {e}"), + Err(_) => tracing::warn!("stdout reader thread panicked during cancellation"), + } + match stderr_thread.join() { + Ok(Ok((f, _tail))) => self.stderr = f, + Ok(Err(e)) => tracing::warn!("stderr reader thread failed during cancellation: {e}"), + Err(_) => tracing::warn!("stderr reader thread panicked during cancellation"), + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/sandbox2/src/lib.rs` around lines 726 - 731, The current join logic for stdout_thread and stderr_thread silently ignores both panics and reader Err results; change the two if-let blocks to explicitly match the join outcomes (match stdout_thread.join() and match stderr_thread.join()) and handle all arms: Ok(Ok((f, _))) should set self.stdout, Ok(Err(e)) should propagate or return that error (or at minimum log it with context), and Err(panic) should be converted into a visible error (return or log) instead of being dropped; do the same for stderr_thread and ensure the function returns an Err or reports the failure so teardown errors are not swallowed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/sandbox2/src/lib.rs`:
- Around line 726-731: The current join logic for stdout_thread and
stderr_thread silently ignores both panics and reader Err results; change the
two if-let blocks to explicitly match the join outcomes (match
stdout_thread.join() and match stderr_thread.join()) and handle all arms:
Ok(Ok((f, _))) should set self.stdout, Ok(Err(e)) should propagate or return
that error (or at minimum log it with context), and Err(panic) should be
converted into a visible error (return or log) instead of being dropped; do the
same for stderr_thread and ensure the function returns an Err or reports the
failure so teardown errors are not swallowed.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3ae55776-82d2-4f95-b761-f1ce166f753b
📒 Files selected for processing (7)
crates/mctx/src/error.rscrates/minimal/src/cmd_run.rscrates/op/src/patched.rscrates/op/src/specs.rscrates/orchestrator/src/local_backend.rscrates/sandbox2/src/error.rscrates/sandbox2/src/lib.rs
✅ Files skipped from review due to trivial changes (1)
- crates/op/src/patched.rs
|
Overall, if we need to make changes to support confidential computing, I would prefer to scope them to exactly just our builder (i.e. no new configurables, just an undocumented flag Or like if we need a new primitive (i.e. for loading go sources or whatever) we should figure out a clean way to plumb/scaffold it that isnt just env vars, so its less likely to accidentally be the source of non-hermetic behavior in the future (given our users wont be building in this mode).
I'm confused about this, minimal-in-minimal? Shouldnt that not work at all because of NO_NEW_PRIVS?
Why is
SGTM. |
| // GCP Confidential Space (and other containerd-default runtimes) | ||
| // mask sub-paths of /proc (/proc/kcore, /proc/scsi, ...) via | ||
| // tmpfs overlays. The kernel then refuses any procfsmount in a | ||
| // nested namespace to prevent unmasking attacks, EPERM. The |
There was a problem hiding this comment.
Is there no way to turn this off?
There was a problem hiding this comment.
No, unfortunately part of Confidential Spaces, here is Claude's explanation,
Root cause: CS workload containers get OCI MaskedPaths (containerd's
default — tmpfs masks /proc/{kcore,scsi,keys,...}). The Linux kernel's
anti-unmask guard then refuses any nested procfsmount over a masked
parent, so hakoniwa's implicit Container::new()::procfsmount returns
EPERM. Not configurable from a CS workload — both the MaskedPaths
layer and the kernel guard are sealed.
| // any other value) would still enable the bind-mount. | ||
| if std::env::var("MINIMAL_SANDBOX_BIND_PROC").as_deref() == Ok("1") { | ||
| container.bindmount_rw("/proc", "/proc"); | ||
| container.runctl(hakoniwa::Runctl::MountFallback); |
There was a problem hiding this comment.
Also worth noting this applies to all mountpoints in the sandbox, not just /proc. Might not be avoidable if we cant turn off the weird behavior of the CS kernel.
There was a problem hiding this comment.
Yeah, this is a hakoniwa limitation, all or nothing on the runctl mount options. I'm renmaing this behind the MINIMAL_INTERNAL_CS_BUILD flag.
Thanks for the thorough review, putting behind a flag and reducing the changes sounds like a good plan. This was a bit of a mess to get the builder working, I'm going to redesign and put behind flag. I'll drop the extra_env_vars and extra_rootfs API additions
Yeah, this isn't accurate, the builder is using minimal as a library inside the Confidential Spaces setup, not shelling from minimal. This is a CS issue, not a minimal-in-minimal.
Yeah, this isn't accurate either, this was the original plan 1 I was using in hermetic builder, but my final design is just adding to the build except for some of the original Go work, but that came move into the pkgs changes. I was using some extra env vars to make the offline stuff work, but they can all be staged in /state instead of the top level extra bindings I was using when I was testing. I should be able to get Claude to redesign the hermetic builder to better work with what we have.
I'll submit an updated patch soon, the /proc is a limitation of Confidential Spaces and hakoniwa doesn't support a per-mount runctl, so it is all or nothing. I'm going to move that part behind the the flag as suggested. If we want to lock it down more, I can see if there is a way to patch hakoniwa to be more flexible in that config since they have been pretty respsonsive. |
|
Gotcha, so it sounds like the two broad strokes we need are:
|
…DBOX_BIND_PROC) Per @twitchyliquid64 review on #228: scope CS-specific support to exactly our trusted builder rather than expose generic configurables that could leak into non-hermetic behavior for other callers. This commit collapses three earlier surface-area additions into one undocumented "I am the trusted CS builder" flag: - (was) MINIMAL_SANDBOX_BIND_PROC env var — renamed + semantically expanded. - (was) `BackendShared.extra_env_vars` + `extra_rootfs` public fields — dropped entirely (separate revert). - (was) `new_orchestrator_with_env` 11-arg constructor — dropped entirely (separate revert). When MINIMAL_INTERNAL_CS_BUILD=1: 1. Bind-mount the outer /proc + enable Runctl::MountFallback. CS workload containers get OCI MaskedPaths (containerd's default — tmpfs over /proc/{kcore,scsi,keys,...}). The Linux kernel anti-unmask guard then refuses any nested procfsmount over a masked parent, so hakoniwa's implicit Container::new() ::procfsmount returns EPERM. Workaround: bind the outer (already-masked) /proc instead of trying to mount a fresh procfs. MountFallback is required for the mandatory MS_REMOUNT that hakoniwa emits after every bind — without it the remount also fails on the masked /proc. Caveat noted by @twitchyliquid64 (line 446 inline): MountFallback is a container-global runctl, not per-mount. It applies to every bind in the sandbox — a sandbox that wanted to assert e.g. NOEXEC may silently end up with the source's existing flags instead. Acceptable inside the CS attested boundary where outer isolation handles the actual security property; the inner hakoniwa is for build-script reproducibility, not isolation. Will see if hakoniwa upstream takes a per-mount runctl patch that would let us scope this to /proc only. 2. Symlink the hermetic-builder's ecosystem cache paths. minimalmertic's hermetic-builder-rs stages caches at <state>/cs-mirror/{cargo-vendor,npm-cache,pnpm-store,bun-cache, pip-wheels,rust-stage0}/. The /state dir is bind-mounted into every sandbox, so the caches are accessible at /state/cs-mirror/ already. Pkg build.shs reference /cargo-vendor, /npm-cache, etc. as hardcoded paths (per the if-d-cargo-vendor offline-cache idiom); these symlinks bridge the two without requiring every build.sh to know about /state/cs-mirror/. Dangling symlinks (when a given cache wasn't staged for the current pkg) are safe — `[ -d /<name> ]` returns false through a dangling symlink, so build.sh's online-fallback branch is correctly taken. Inert when the env var is unset; default `minimal package` users see zero behavior change. GOPROXY/GOSUMDB do NOT get auto-injected by minimal — each Go pkg's build.sh sets them itself (gated on the existence of /state/cs-mirror/goproxy), keeping minimal ecosystem-agnostic. Per @twitchyliquid64's "can we just set the environment variables in build.sh ?" suggestion. Tested: cargo check -p sandbox2 --tests passes in Linux container (Rust 1.x, Debian 13, protobuf-compiler). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mesa's build.sh uses `pip install foo 2>/dev/null || true` so when the python pkgs fail to install, stderr is silenced — and any diagnostic from the subsequent `meson setup` step that fails because of missing python pkgs only appears on stdout. The InvocationFailed error has carried only stderr; the operator was getting empty stderr strings with no clue what failed. Mirror the existing stderr_thread rolling-tail logic in stdout_thread: last 4 KiB kept, surfaced via a new `stdout` field on InvocationFailed. mctx::Error::Execution propagates the same field so consumers like hermetic-builder-rs's queue_mode get both streams in the FailureDetail message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ab9ea9f to
86bbec1
Compare
|
Force-pushed v2 implementing the redesign:
PR collapses from 3 commits to 2; new public API surface is zero. |
) PR #228's MINIMAL_INTERNAL_CS_BUILD bundle had the right intent but the cache-delivery half didn't survive integration. The container.symlink("/state/cs-mirror/<cache>", "/<cache>") calls assumed /state was shared across all sandboxes — but /state inside each per-build sandbox defaults to base_dir.join("state"), a per-build location not shared with the orchestrator's outer state dir. The symlinks all dangled, and `[ -d /cargo-vendor ]` checks in package build.shs took their online-fallback branch (which fails in CS because there's no egress). Fix: when the flag is set, hardlink contents from the convention path /root/.cache/minimal/cs-mirror into the sandbox rootfs root. This is the same mechanism the (since-removed) public extra_rootfs field used; here it's triggered behind the existing CS flag with a hardcoded internal-convention path, so no new public API surface. - Still one env var: MINIMAL_INTERNAL_CS_BUILD - Still boolean semantics (== "1") - /root/.cache/minimal/cs-mirror is project-internal convention (same category as /state and /run — convention, not configuration) The /proc bind-mount + MountFallback runctl from #228 stay — that part of the bundle was correct. Inert when the env var is unset; inert when the convention path doesn't exist (tests, dev environments, non-CS callers). The hardlink_dir_contents call is the same one already exercised immediately above for the synth-dir copy. Validated end-to-end in the minimalmertic deployment over 2026-05-25/26: 260+ packages signed under the new bundle (C, Go, Cargo, npm, pnpm, pypi), and `bash-bootstrap@5.3` reached a self-reproducing fixed point across three build cycles using the mechanism (CS-attested artifact → trust anchor for cycle 2 → same sha out, no Mac/CI prebuilt left in the chain). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Three independent additions to minimal that unblock running
minimal packageinside Google Cloud Confidential Space as a nested sandbox (a top-level minimal invocation that itself shells out tominimal packageinside hermetic-builder workloads).All three are additive — no existing-caller behavior changes.
Changes
sandbox2: opt-in bind-mount /proc (52996b5)The default sandbox masks
/procto prevent introspection. Nested sandbox use (oneminimalinvocation spawning anotherminimal packageunderneath) needs the inner sandbox to read/proc/self/*for its own bookkeeping. Adds aproc_bind_mount: boolfield onSandbox. Default off (preserves existing behavior). When true, sandbox2 bind-mounts the host/procinto the sandbox.op+orchestrator: caller-supplied env_vars + rootfs for build sandboxes (a3d35c1)The build-spec sandbox config is currently fixed: caller can't add env vars or override the rootfs without recompiling minimal. Adds an
extra_env_vars: Option<...>+extra_rootfs: Option<PathBuf>toBackendShared. Plumbed throughLocalBackend::build_in_sandboxso existing callers see no change; the new fields defaultNone.Motivating use: hermetic-builder-rs runs in CS with a pre-staged rootfs (toolchain + libs) different from the minimal default, and needs to inject
BUILD_PKG,OUTPUT_DIR,STATE_DIRenv vars based on the queue task it's processing.sandbox2: capture stdout tail in InvocationFailed (6c62ce9)Build failures that print diagnostics to stdout (autotools
configurefor example) currently lose those bytes —InvocationFailedcarries only stderr. Adds parallel stdout capture (same 4KiB rolling tail as stderr). Operators triaging hermetic build failures gain ~2x diagnostic surface for free.Test plan
cargo fmt --checkclean (CI; macOS host can't run sandbox2 clippy — Linux-only deps)Relationship to #202
Lifted these out of #202's branch where they were stacked. PR #202 keeps the
--offlineflag theme; this PR is purely about CS-compatibility sandbox plumbing. Reviewed separately on purpose — the changes are orthogonal.Summary by CodeRabbit