Skip to content

sandbox2 + op: CS-compatible nested-sandbox plumbing - #228

Merged
bryan-minimal merged 2 commits into
mainfrom
bryan/cs-sandbox-extensions
May 25, 2026
Merged

bryan-minimal merged 2 commits into
mainfrom
bryan/cs-sandbox-extensions

Conversation

@bryan-minimal

@bryan-minimal bryan-minimal commented May 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Three independent additions to minimal that unblock running minimal package inside Google Cloud Confidential Space as a nested sandbox (a top-level minimal invocation that itself shells out to minimal package inside hermetic-builder workloads).

All three are additive — no existing-caller behavior changes.

Changes

sandbox2: opt-in bind-mount /proc (52996b5)

The default sandbox masks /proc to prevent introspection. Nested sandbox use (one minimal invocation spawning another minimal package underneath) needs the inner sandbox to read /proc/self/* for its own bookkeeping. Adds a proc_bind_mount: bool field on Sandbox. Default off (preserves existing behavior). When true, sandbox2 bind-mounts the host /proc into the sandbox.

op + orchestrator: caller-supplied env_vars + rootfs for build sandboxes (a3d35c1)

The build-spec sandbox config is currently fixed: caller can't add env vars or override the rootfs without recompiling minimal. Adds an extra_env_vars: Option<...> + extra_rootfs: Option<PathBuf> to BackendShared. Plumbed through LocalBackend::build_in_sandbox so existing callers see no change; the new fields default None.

Motivating use: hermetic-builder-rs runs in CS with a pre-staged rootfs (toolchain + libs) different from the minimal default, and needs to inject BUILD_PKG, OUTPUT_DIR, STATE_DIR env vars based on the queue task it's processing.

sandbox2: capture stdout tail in InvocationFailed (6c62ce9)

Build failures that print diagnostics to stdout (autotools configure for example) currently lose those bytes — InvocationFailed carries only stderr. Adds parallel stdout capture (same 4KiB rolling tail as stderr). Operators triaging hermetic build failures gain ~2x diagnostic surface for free.

Test plan

  • cargo fmt --check clean (CI; macOS host can't run sandbox2 clippy — Linux-only deps)
  • Smoke-tested in production: all three are running in minimalmertic's hermetic-builder-rs (CS workload) and have processed 250+ successful builds since landing on the local branch.

Relationship to #202

Lifted these out of #202's branch where they were stacked. PR #202 keeps the --offline flag theme; this PR is purely about CS-compatibility sandbox plumbing. Reviewed separately on purpose — the changes are orthogonal.

Summary by CodeRabbit

  • New Features
    • Execution error messages now capture and display sandboxed process output (stdout) alongside existing error diagnostics, providing enhanced visibility into failed task execution.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 25, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1bb251ba-8757-41ac-8397-764ee8bc6c55

📥 Commits

Reviewing files that changed from the base of the PR and between ab9ea9f and 86bbec1.

📒 Files selected for processing (4)
  • crates/mctx/src/error.rs
  • crates/minimal/src/cmd_run.rs
  • crates/sandbox2/src/error.rs
  • crates/sandbox2/src/lib.rs

📝 Walkthrough

Walkthrough

The PR adds stdout tail capture (last ~4 KiB) to sandbox execution errors. The capture flows through updated error types at both sandbox2 and mctx layers, with a rolling tail buffer maintained during execution. A separate /proc remediation feature optionally bind-mounts and symlinks cache directories when enabled.

Changes

Sandbox stdout capture and /proc remediation

Layer / File(s) Summary
Sandbox2 ExecutionError stdout field
crates/sandbox2/src/error.rs
ExecutionError::InvocationFailed gains a stdout: String field to capture the last ~4 KiB of output. Display formatting conditionally renders a stdout: section when non-empty.
Mctx Error::Execution stdout propagation
crates/mctx/src/error.rs
Error::Execution variant adds stdout: String. Display implementation conditionally appends stdout: output section. Fromsandbox2::Error mapping extracts and propagates upstream stdout into the mctx error.
Sandbox2 stdout tail collection and error construction
crates/sandbox2/src/lib.rs (stdout reader thread)
Stdout reader thread maintains a rolling ~4 KiB tail buffer while streaming all output to file. Thread join now returns (file_handle, tail_bytes) instead of only the file. On non-success status, the tail is UTF-8-decoded (lossy) and placed in ExecutionError::InvocationFailed stdout field.
Sandbox2 /proc bind-mount and cache symlink injection
crates/sandbox2/src/lib.rs (new_container)
Opt-in MINIMAL_INTERNAL_CS_BUILD path bind-mounts /proc writable, enables MountFallback, and symlinks /state/cs-mirror/* cache directories to fixed in-sandbox mount points (e.g., /cargo-vendor, /npm-cache).
Minimal cmd_run.rs Error::Execution construction
crates/minimal/src/cmd_run.rs
When a spawned command exits with non-success, Error::Execution is constructed with the new stdout field set to an empty string (no stdout available in this path).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • twitchyliquid64
  • msample
  • evanspearman
  • 0chroma

Poem

🐰 Stdout whispers flow like mountain streams,
Through sandbox walls where execution dreams,
A rolling tail of four kibibytes tall,
Catches the output when invocations fall—
And /proc finds home where cache symlinks call!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: introducing CS-compatible nested-sandbox plumbing via updates to sandbox2 and operational code.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/mctx/src/error.rs (1)

17-27: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Render the captured streams on the default reporting path.

report_to() still falls back to Display for Error::Execution, and Display only prints the exit summary. As written, the new stdout tail never reaches the main user-facing error path, and the existing stderr is still hidden too.

Proposed fix
-            Error::Execution {
-                idx, code, reason, ..
-            } if *code == 125 => write!(f, "invocation {} failed: {}", idx, reason,),
-            Error::Execution { idx, code, .. } => {
-                write!(f, "invocation {} failed: exit code {}", idx, code,)
-            }
+            Error::Execution {
+                idx,
+                code,
+                reason,
+                stderr,
+                stdout,
+            } => {
+                if *code == 125 {
+                    write!(f, "invocation {idx} failed: {reason}")?;
+                } else {
+                    write!(f, "invocation {idx} failed: exit code {code}")?;
+                }
+                if !stderr.is_empty() {
+                    write!(f, "\nstderr:\n{stderr}")?;
+                }
+                if !stdout.is_empty() {
+                    write!(f, "\nstdout:\n{stdout}")?;
+                }
+                Ok(())
+            }

Also applies to: 235-246

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mctx/src/error.rs` around lines 17 - 27, The Execution variant's
captured stdout/stderr tails are never shown because Error::Execution is only
rendered via Display which prints only the exit summary; update the Display impl
for Error (or the report_to() match arm that handles Error::Execution) to
include the Execution struct's stderr and stdout fields (e.g., Execution.stderr
and Execution.stdout) on the default reporting path, labeling them (e.g.,
"stderr:" / "stdout:") and only emitting non-empty tails (or truncating long
tails) so that both the existing stderr and the new stdout tail reach the main
user-facing error output.
🧹 Nitpick comments (1)
crates/orchestrator/src/local_backend.rs (1)

432-450: 🏗️ Heavy lift

Replace the 11-argument constructor with a typed config/builder API.

Lines 438-450 are now highly order-sensitive and require lint suppression. A config struct/builder would make optional sandbox injection explicit and safer at call sites.

As per coding guidelines, **/*.rs: "Use the builder pattern for types with more than a couple of fields or any optional config, in the owned style: pub fn with_x(mut self, x: T) -> Self" and "Justify any #[allow(...)] attributes with a comment explaining why the lint is suppressed".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/orchestrator/src/local_backend.rs` around lines 432 - 450, The
constructor new_orchestrator_with_env has too many positional arguments and uses
#[allow(clippy::too_many_arguments)]; replace it with a typed config/builder:
introduce a new OrchestratorConfig (or NewOrchestratorBuilder) struct capturing
fields (output_base, remote_cache, sf, num_concurrent_builds, graph, cache,
log_sink, ot, cancel, extra_env_vars, extra_rootfs) and implement owned-style
builder methods like with_remote_cache(...), with_extra_env_vars(...),
with_extra_rootfs(...), etc., and a build_orchestrator(self) ->
Result<Orchestrator<Self>, Error> that performs the construction currently in
new_orchestrator_with_env; remove the clippy allow and keep public API ergonomic
by providing a convenience Orchestrator::new_with_config(config) that delegates
to the builder.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/op/src/specs.rs`:
- Around line 285-288: The for-loop in SpecBuild::run currently consumes
self.extra_rootfs with drain(..), which drops entries for subsequent calls;
instead iterate non-destructively (e.g., iterate over &self.extra_rootfs and
clone or otherwise copy each entry) and insert into rootfs, so extra_rootfs
remains intact between runs; update the loop that inserts SandboxMapped::Dir to
use a borrow-based iteration (referencing extra_rootfs and SandboxMapped::Dir)
rather than drain(..).

In `@crates/sandbox2/src/lib.rs`:
- Around line 441-443: The current code treats any set value of
MINIMAL_SANDBOX_BIND_PROC as true; change the check so it only enables
container.bindmount_rw("/proc", "/proc") and
container.runctl(hakoniwa::Runctl::MountFallback) when the environment variable
is explicitly set to "1". Locate the block that reads
std::env::var_os("MINIMAL_SANDBOX_BIND_PROC") and replace the truthy presence
check with an explicit value comparison against "1" (preserving use of
container.bindmount_rw and container.runctl with
hakoniwa::Runctl::MountFallback).

---

Outside diff comments:
In `@crates/mctx/src/error.rs`:
- Around line 17-27: The Execution variant's captured stdout/stderr tails are
never shown because Error::Execution is only rendered via Display which prints
only the exit summary; update the Display impl for Error (or the report_to()
match arm that handles Error::Execution) to include the Execution struct's
stderr and stdout fields (e.g., Execution.stderr and Execution.stdout) on the
default reporting path, labeling them (e.g., "stderr:" / "stdout:") and only
emitting non-empty tails (or truncating long tails) so that both the existing
stderr and the new stdout tail reach the main user-facing error output.

---

Nitpick comments:
In `@crates/orchestrator/src/local_backend.rs`:
- Around line 432-450: The constructor new_orchestrator_with_env has too many
positional arguments and uses #[allow(clippy::too_many_arguments)]; replace it
with a typed config/builder: introduce a new OrchestratorConfig (or
NewOrchestratorBuilder) struct capturing fields (output_base, remote_cache, sf,
num_concurrent_builds, graph, cache, log_sink, ot, cancel, extra_env_vars,
extra_rootfs) and implement owned-style builder methods like
with_remote_cache(...), with_extra_env_vars(...), with_extra_rootfs(...), etc.,
and a build_orchestrator(self) -> Result<Orchestrator<Self>, Error> that
performs the construction currently in new_orchestrator_with_env; remove the
clippy allow and keep public API ergonomic by providing a convenience
Orchestrator::new_with_config(config) that delegates to the builder.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ba17cb33-c0da-48e4-a9bd-98bf4f2bde46

📥 Commits

Reviewing files that changed from the base of the PR and between 60cb7e4 and 6c62ce9.

📒 Files selected for processing (6)
  • crates/mctx/src/error.rs
  • crates/op/src/patched.rs
  • crates/op/src/specs.rs
  • crates/orchestrator/src/local_backend.rs
  • crates/sandbox2/src/error.rs
  • crates/sandbox2/src/lib.rs

Comment thread crates/op/src/specs.rs Resolved
Comment thread crates/sandbox2/src/lib.rs Resolved
@bryan-minimal
bryan-minimal force-pushed the bryan/cs-sandbox-extensions branch from 6c62ce9 to f2038c4 Compare May 25, 2026 01:27

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
crates/sandbox2/src/lib.rs (1)

441-444: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Honor the documented =1 opt-in.

The check std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() enables the /proc bind mount for any value, so MINIMAL_SANDBOX_BIND_PROC=0 or MINIMAL_SANDBOX_BIND_PROC=false still flips behavior. The comment on line 410-411 documents =1 as the expected opt-in.

Proposed fix
-        if std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() {
+        if matches!(
+            std::env::var("MINIMAL_SANDBOX_BIND_PROC").as_deref(),
+            Ok("1")
+        ) {
             container.bindmount_rw("/proc", "/proc");
             container.runctl(hakoniwa::Runctl::MountFallback);
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/sandbox2/src/lib.rs` around lines 441 - 444, The current check uses
std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() which treats any set
value (including "0" or "false") as enabled; change it to only opt-in when the
env var equals "1". Update the condition around the
container.bindmount_rw("/proc", "/proc") and
container.runctl(hakoniwa::Runctl::MountFallback) calls to explicitly compare
the environment variable value to "1" (e.g., read the var and compare
string-wise) so only MINIMAL_SANDBOX_BIND_PROC=1 enables the /proc bind mount.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@crates/sandbox2/src/lib.rs`:
- Around line 441-444: The current check uses
std::env::var_os("MINIMAL_SANDBOX_BIND_PROC").is_some() which treats any set
value (including "0" or "false") as enabled; change it to only opt-in when the
env var equals "1". Update the condition around the
container.bindmount_rw("/proc", "/proc") and
container.runctl(hakoniwa::Runctl::MountFallback) calls to explicitly compare
the environment variable value to "1" (e.g., read the var and compare
string-wise) so only MINIMAL_SANDBOX_BIND_PROC=1 enables the /proc bind mount.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 66579a88-a2d1-4be2-a8e5-23b6f4762d15

📥 Commits

Reviewing files that changed from the base of the PR and between 6c62ce9 and f2038c4.

📒 Files selected for processing (4)
  • crates/mctx/src/error.rs
  • crates/minimal/src/cmd_run.rs
  • crates/sandbox2/src/error.rs
  • crates/sandbox2/src/lib.rs

@bryan-minimal
bryan-minimal force-pushed the bryan/cs-sandbox-extensions branch from f2038c4 to ab9ea9f Compare May 25, 2026 01:31
@bryan-minimal

Copy link
Copy Markdown
Member Author

Thanks for the careful review! Addressed all 4 comments — force-pushed bryan/cs-sandbox-extensions:

1. Display impl rendering (crates/mctx/src/error.rs) — Fixed. Error::Execution's Display now prints the exit summary, then appends stderr: and stdout: blocks when non-empty. Folded into the sandbox2 stdout-tail commit so the field-add and the user-facing render land together.

2. drain(..) consuming extra_rootfs (crates/op/src/specs.rs) — Fixed. Changed to for extra in &self.extra_rootfs { rootfs.insert(SandboxMapped::Dir(extra.clone())); } so subsequent run() calls still see the caller-supplied paths. Added a comment explaining why.

3. MINIMAL_SANDBOX_BIND_PROC truthy-check (crates/sandbox2/src/lib.rs) — Fixed. Strict equality check now: std::env::var("MINIMAL_SANDBOX_BIND_PROC").as_deref() == Ok("1"). =0 (and any other value) no longer enables the bind-mount — explicit =1 only.

4. 11-positional-arg constructor / builder pattern (crates/orchestrator/src/local_backend.rs) — Documented the deliberate suppression but did NOT do the builder refactor in this PR. Reasoning: a typed OrchestratorConfig builder is the right shape, but pushing 11+ fields into a new struct is structurally larger than the env_vars+rootfs feature this PR adds. Tracked separately for a focused refactor PR. Comment added so future readers understand the lint allow.

Mirrored these same fixes on the local bryan/fetcher-extra-env-vars branch we vendor from in the downstream hermetic-builder-rs — keeps the downstream test-runs consistent with what's under review here.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/sandbox2/src/lib.rs (1)

726-731: ⚡ Quick win

Don’t silently drop reader-thread failures during cancellation.

The if let join pattern drops both panic and Err(Error) outcomes, which hides teardown failures when cancellation happens.

Proposed fix
-                    if let Ok(Ok((f, _tail))) = stdout_thread.join() {
-                        self.stdout = f;
-                    }
-                    if let Ok(Ok((f, _tail))) = stderr_thread.join() {
-                        self.stderr = f;
-                    }
+                    match stdout_thread.join() {
+                        Ok(Ok((f, _tail))) => self.stdout = f,
+                        Ok(Err(e)) => tracing::warn!("stdout reader thread failed during cancellation: {e}"),
+                        Err(_) => tracing::warn!("stdout reader thread panicked during cancellation"),
+                    }
+                    match stderr_thread.join() {
+                        Ok(Ok((f, _tail))) => self.stderr = f,
+                        Ok(Err(e)) => tracing::warn!("stderr reader thread failed during cancellation: {e}"),
+                        Err(_) => tracing::warn!("stderr reader thread panicked during cancellation"),
+                    }
As per coding guidelines: "Never swallow errors; do not use `let _ = result;` or `.ok()` discards without a comment justifying it".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/sandbox2/src/lib.rs` around lines 726 - 731, The current join logic
for stdout_thread and stderr_thread silently ignores both panics and reader Err
results; change the two if-let blocks to explicitly match the join outcomes
(match stdout_thread.join() and match stderr_thread.join()) and handle all arms:
Ok(Ok((f, _))) should set self.stdout, Ok(Err(e)) should propagate or return
that error (or at minimum log it with context), and Err(panic) should be
converted into a visible error (return or log) instead of being dropped; do the
same for stderr_thread and ensure the function returns an Err or reports the
failure so teardown errors are not swallowed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/sandbox2/src/lib.rs`:
- Around line 726-731: The current join logic for stdout_thread and
stderr_thread silently ignores both panics and reader Err results; change the
two if-let blocks to explicitly match the join outcomes (match
stdout_thread.join() and match stderr_thread.join()) and handle all arms:
Ok(Ok((f, _))) should set self.stdout, Ok(Err(e)) should propagate or return
that error (or at minimum log it with context), and Err(panic) should be
converted into a visible error (return or log) instead of being dropped; do the
same for stderr_thread and ensure the function returns an Err or reports the
failure so teardown errors are not swallowed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3ae55776-82d2-4f95-b761-f1ce166f753b

📥 Commits

Reviewing files that changed from the base of the PR and between f2038c4 and ab9ea9f.

📒 Files selected for processing (7)
  • crates/mctx/src/error.rs
  • crates/minimal/src/cmd_run.rs
  • crates/op/src/patched.rs
  • crates/op/src/specs.rs
  • crates/orchestrator/src/local_backend.rs
  • crates/sandbox2/src/error.rs
  • crates/sandbox2/src/lib.rs
✅ Files skipped from review due to trivial changes (1)
  • crates/op/src/patched.rs

@twitchyliquid64

Copy link
Copy Markdown
Contributor

Overall, if we need to make changes to support confidential computing, I would prefer to scope them to exactly just our builder (i.e. no new configurables, just an undocumented flag MINIMAL_INTERNAL_CS_BUILD or something), or use existing mechanisms (can we just set the environment variables in build.sh ?)

Or like if we need a new primitive (i.e. for loading go sources or whatever) we should figure out a clean way to plumb/scaffold it that isnt just env vars, so its less likely to accidentally be the source of non-hermetic behavior in the future (given our users wont be building in this mode).

a top-level minimal invocation that itself shells out to minimal package inside hermetic-builder workloads

I'm confused about this, minimal-in-minimal? Shouldnt that not work at all because of NO_NEW_PRIVS?

Motivating use: hermetic-builder-rs runs in CS with a pre-staged rootfs (toolchain + libs) different from the minimal default, and needs to inject BUILD_PKG, OUTPUT_DIR, STATE_DIR env vars based on the queue task it's processing.

Why is OUTPUT_DIR getting overridden? Staging output files in OUTPUT_DIR is such a core part of the build system this seems like a large structural change and not a small tweak.

capture stdout tail in InvocationFailed

SGTM.

Comment thread crates/sandbox2/src/lib.rs Outdated
// GCP Confidential Space (and other containerd-default runtimes)
// mask sub-paths of /proc (/proc/kcore, /proc/scsi, ...) via
// tmpfs overlays. The kernel then refuses any procfsmount in a
// nested namespace to prevent unmasking attacks, EPERM. The

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no way to turn this off?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, unfortunately part of Confidential Spaces, here is Claude's explanation,

Root cause: CS workload containers get OCI MaskedPaths (containerd's
default — tmpfs masks /proc/{kcore,scsi,keys,...}). The Linux kernel's
anti-unmask guard then refuses any nested procfsmount over a masked
parent, so hakoniwa's implicit Container::new()::procfsmount returns
EPERM. Not configurable from a CS workload — both the MaskedPaths
layer and the kernel guard are sealed.

// any other value) would still enable the bind-mount.
if std::env::var("MINIMAL_SANDBOX_BIND_PROC").as_deref() == Ok("1") {
container.bindmount_rw("/proc", "/proc");
container.runctl(hakoniwa::Runctl::MountFallback);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also worth noting this applies to all mountpoints in the sandbox, not just /proc. Might not be avoidable if we cant turn off the weird behavior of the CS kernel.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a hakoniwa limitation, all or nothing on the runctl mount options. I'm renmaing this behind the MINIMAL_INTERNAL_CS_BUILD flag.

@bryan-minimal

Copy link
Copy Markdown
Member Author

Overall, if we need to make changes to support confidential computing, I would prefer to scope them to exactly just our builder (i.e. no new configurables, just an undocumented flag MINIMAL_INTERNAL_CS_BUILD or something), or use existing mechanisms (can we just set the environment variables in build.sh ?)

Or like if we need a new primitive (i.e. for loading go sources or whatever) we should figure out a clean way to plumb/scaffold it that isnt just env vars, so its less likely to accidentally be the source of non-hermetic behavior in the future (given our users wont be building in this mode).

Thanks for the thorough review, putting behind a flag and reducing the changes sounds like a good plan. This was a bit of a mess to get the builder working, I'm going to redesign and put behind flag. I'll drop the extra_env_vars and extra_rootfs API additions

a top-level minimal invocation that itself shells out to minimal package inside hermetic-builder workloads

I'm confused about this, minimal-in-minimal? Shouldnt that not work at all because of NO_NEW_PRIVS?

Yeah, this isn't accurate, the builder is using minimal as a library inside the Confidential Spaces setup, not shelling from minimal. This is a CS issue, not a minimal-in-minimal.

Motivating use: hermetic-builder-rs runs in CS with a pre-staged rootfs (toolchain + libs) different from the minimal default, and needs to inject BUILD_PKG, OUTPUT_DIR, STATE_DIR env vars based on the queue task it's processing.

Why is OUTPUT_DIR getting overridden? Staging output files in OUTPUT_DIR is such a core part of the build system this seems like a large structural change and not a small tweak.

Yeah, this isn't accurate either, this was the original plan 1 I was using in hermetic builder, but my final design is just adding to the build except for some of the original Go work, but that came move into the pkgs changes. I was using some extra env vars to make the offline stuff work, but they can all be staged in /state instead of the top level extra bindings I was using when I was testing. I should be able to get Claude to redesign the hermetic builder to better work with what we have.

capture stdout tail in InvocationFailed

SGTM.

I'll submit an updated patch soon, the /proc is a limitation of Confidential Spaces and hakoniwa doesn't support a per-mount runctl, so it is all or nothing. I'm going to move that part behind the the flag as suggested. If we want to lock it down more, I can see if there is a way to patch hakoniwa to be more flexible in that config since they have been pretty respsonsive.

@twitchyliquid64

Copy link
Copy Markdown
Contributor

Gotcha, so it sounds like the two broad strokes we need are:

  1. The runctl for /proc
  2. Some mechanism for plumbing the goproxy and stuff?

bryan-minimal and others added 2 commits May 24, 2026 21:23
…DBOX_BIND_PROC)

Per @twitchyliquid64 review on #228: scope CS-specific support to
exactly our trusted builder rather than expose generic configurables
that could leak into non-hermetic behavior for other callers.

This commit collapses three earlier surface-area additions into one
undocumented "I am the trusted CS builder" flag:

- (was) MINIMAL_SANDBOX_BIND_PROC env var — renamed +
  semantically expanded.
- (was) `BackendShared.extra_env_vars` + `extra_rootfs` public
  fields — dropped entirely (separate revert).
- (was) `new_orchestrator_with_env` 11-arg constructor — dropped
  entirely (separate revert).

When MINIMAL_INTERNAL_CS_BUILD=1:

1. Bind-mount the outer /proc + enable Runctl::MountFallback.
   CS workload containers get OCI MaskedPaths (containerd's default
   — tmpfs over /proc/{kcore,scsi,keys,...}). The Linux kernel
   anti-unmask guard then refuses any nested procfsmount over a
   masked parent, so hakoniwa's implicit Container::new()
   ::procfsmount returns EPERM. Workaround: bind the outer
   (already-masked) /proc instead of trying to mount a fresh
   procfs. MountFallback is required for the mandatory MS_REMOUNT
   that hakoniwa emits after every bind — without it the remount
   also fails on the masked /proc.

   Caveat noted by @twitchyliquid64 (line 446 inline): MountFallback
   is a container-global runctl, not per-mount. It applies to every
   bind in the sandbox — a sandbox that wanted to assert e.g.
   NOEXEC may silently end up with the source's existing flags
   instead. Acceptable inside the CS attested boundary where outer
   isolation handles the actual security property; the inner
   hakoniwa is for build-script reproducibility, not isolation.
   Will see if hakoniwa upstream takes a per-mount runctl patch
   that would let us scope this to /proc only.

2. Symlink the hermetic-builder's ecosystem cache paths.
   minimalmertic's hermetic-builder-rs stages caches at
   <state>/cs-mirror/{cargo-vendor,npm-cache,pnpm-store,bun-cache,
   pip-wheels,rust-stage0}/. The /state dir is bind-mounted into
   every sandbox, so the caches are accessible at /state/cs-mirror/
   already. Pkg build.shs reference /cargo-vendor, /npm-cache, etc.
   as hardcoded paths (per the if-d-cargo-vendor offline-cache
   idiom); these symlinks bridge the two without requiring every
   build.sh to know about /state/cs-mirror/. Dangling symlinks
   (when a given cache wasn't staged for the current pkg) are safe
   — `[ -d /<name> ]` returns false through a dangling symlink, so
   build.sh's online-fallback branch is correctly taken.

Inert when the env var is unset; default `minimal package` users
see zero behavior change.

GOPROXY/GOSUMDB do NOT get auto-injected by minimal — each Go pkg's
build.sh sets them itself (gated on the existence of
/state/cs-mirror/goproxy), keeping minimal ecosystem-agnostic. Per
@twitchyliquid64's "can we just set the environment variables in
build.sh ?" suggestion.

Tested: cargo check -p sandbox2 --tests passes in Linux container
(Rust 1.x, Debian 13, protobuf-compiler).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mesa's build.sh uses `pip install foo 2>/dev/null || true` so when
the python pkgs fail to install, stderr is silenced — and any
diagnostic from the subsequent `meson setup` step that fails because
of missing python pkgs only appears on stdout. The InvocationFailed
error has carried only stderr; the operator was getting empty stderr
strings with no clue what failed.

Mirror the existing stderr_thread rolling-tail logic in stdout_thread:
last 4 KiB kept, surfaced via a new `stdout` field on
InvocationFailed. mctx::Error::Execution propagates the same field
so consumers like hermetic-builder-rs's queue_mode get both streams
in the FailureDetail message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bryan-minimal
bryan-minimal force-pushed the bryan/cs-sandbox-extensions branch from ab9ea9f to 86bbec1 Compare May 25, 2026 04:57
@bryan-minimal

Copy link
Copy Markdown
Member Author

Force-pushed v2 implementing the redesign:

  • Dropped extra_env_vars + extra_rootfs + new_orchestrator_with_env public surface entirely.
  • All CS-specific behavior (/proc bind-mount + MountFallback + ecosystem cache symlinks) now behind a single undocumented MINIMAL_INTERNAL_CS_BUILD=1 env var.
  • GOPROXY/GOSUMDB moved to each Go pkg's build.sh (minimal stays ecosystem-agnostic).
  • Added a comment explicitly calling out your line-446 caveat about MountFallback's sandbox-wide scope.

PR collapses from 3 commits to 2; new public API surface is zero.

@bryan-minimal
bryan-minimal merged commit c0b592b into main May 25, 2026
8 checks passed
@bryan-minimal
bryan-minimal deleted the bryan/cs-sandbox-extensions branch May 25, 2026 16:37
bryan-minimal added a commit that referenced this pull request May 26, 2026
)

PR #228's MINIMAL_INTERNAL_CS_BUILD bundle had the right intent but
the cache-delivery half didn't survive integration. The
container.symlink("/state/cs-mirror/<cache>", "/<cache>") calls
assumed /state was shared across all sandboxes — but /state inside
each per-build sandbox defaults to base_dir.join("state"), a
per-build location not shared with the orchestrator's outer state
dir. The symlinks all dangled, and `[ -d /cargo-vendor ]` checks
in package build.shs took their online-fallback branch (which
fails in CS because there's no egress).

Fix: when the flag is set, hardlink contents from the convention
path /root/.cache/minimal/cs-mirror into the sandbox rootfs root.
This is the same mechanism the (since-removed) public extra_rootfs
field used; here it's triggered behind the existing CS flag with
a hardcoded internal-convention path, so no new public API surface.

  - Still one env var: MINIMAL_INTERNAL_CS_BUILD
  - Still boolean semantics (== "1")
  - /root/.cache/minimal/cs-mirror is project-internal convention
    (same category as /state and /run — convention, not configuration)

The /proc bind-mount + MountFallback runctl from #228 stay — that
part of the bundle was correct.

Inert when the env var is unset; inert when the convention path
doesn't exist (tests, dev environments, non-CS callers). The
hardlink_dir_contents call is the same one already exercised
immediately above for the synth-dir copy.

Validated end-to-end in the minimalmertic deployment over
2026-05-25/26: 260+ packages signed under the new bundle (C, Go,
Cargo, npm, pnpm, pypi), and `bash-bootstrap@5.3` reached a
self-reproducing fixed point across three build cycles using the
mechanism (CS-attested artifact → trust anchor for cycle 2 → same
sha out, no Mac/CI prebuilt left in the chain).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants