Repository navigation
Add --offline flag covering source-tarball + git fetches - #202
Conversation
…ct cache
Today `--no-fetch` only suppresses the remote build-artifact cache
(`gs://minimal-staging-cache`). Source-tarball downloads (op::SourceLoad)
and git clones / fetches (checkouts::Manager) ignore the flag — they
silently hit the network on cache miss.
This is a correctness gap for any consumer running `minimal package`
in a network-isolated environment (e.g., a Confidential Space VM
that pre-stages all inputs in a sha-pinned mirror, then expects
`--no-fetch` to mean "I'm offline; fail clean if anything's missing").
Today such a consumer can't tell whether minimal is using the cached
bytes or attempting a doomed network call.
Changes:
crates/common/src/file_cache.rs:
- FileCache gains an `offline: bool` field + `new_with_offline`
ctor + `is_offline()` accessor. Existing `new(dir)` preserved
for the default (online) path.
- CachingDownloader::download checks `cache.offline` after
read_cached miss; returns FileCacheError::OfflineCacheMiss
instead of falling through to network.
- New error variant carries (sha256, filename) for diagnostics.
crates/common/src/remote_storage.rs:
- RemoteStorage gains `new_with_offline(dir, auth, offline)`
that forwards `offline` into FileCache. Default `new` unchanged.
crates/checkouts/src/lib.rs:
- Manager gains `offline: bool` field + `new_in_dir_with_offline`.
Existing `new_in_dir` preserved.
- update(): no-op when offline (caller intent is "use what's
cached"; refresh-of-known-remote is non-destructive to skip).
- checkout_of() known-remote path: skip `repo.fetch()` when offline.
- checkout_of() unknown-remote path: return Error::OfflineCacheMiss
(cannot clone without network).
crates/checkouts/src/error.rs:
- New OfflineCacheMiss { remote } variant + Display.
crates/mctx/src/lib.rs:
- VcsManager init: pass `!use_remote_cache()` through to
`new_in_dir_with_offline`.
- RemoteStorage init: same shape via `new_with_offline`.
crates/mctx/src/error.rs:
- Map checkouts::Error::OfflineCacheMiss to a clear message.
After the patch, `--no-fetch` is a true "offline mode" flag: cache
miss = fail with a helpful error (pointing operators at cache
pre-population) rather than a silent-then-failing network call.
Smoke test (manual, on the dev branch): `minimal --no-fetch package
sed --rebuild` with a wiped cache surfaces:
Error: failed to fetch upstream layer: offline cache miss for
git remote https://github.com/gominimal/pkgs — --no-fetch is set;
pre-populate the vcs/ cache or remove the flag
Existing test suite (cargo test -p common -p checkouts) passes
without modification — change is purely additive on the cache path.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an offline mode flag propagated through Manager, FileCache, RemoteStorage, Config, Context, and CLI. In offline mode, cache or VCS misses return OfflineCacheMiss errors and network fetches are skipped; constructors, error mappings, and tests were updated. ChangesOffline Mode Support for VCS and File Caching
Sequence Diagram(s)sequenceDiagram
participant CLI
participant Context
participant VcsManager
participant RemoteStorage
participant FileCache
CLI->>Context: run with --offline
Context->>VcsManager: new_in_dir_with_offline(true)
Context->>RemoteStorage: new_with_offline(..., true)
VcsManager->>FileCache: read existing bare repo or return OfflineCacheMiss
RemoteStorage->>FileCache: request cached artifact -> hit or OfflineCacheMiss
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
|
This change seem like it's still allowlisting source downloads from our internal GCS? So it feels like this should be some form of --trusted-source-domains [list,..] style of option. The original --no-fetch behaviour was nice for rebuilding the world locally on your laptop in a package repo (whether they came from GCS, github, or some mirror). It's useful for any pkg repo (well, if/when the bin cache source can be overridden) |
|
For rebuilds, we now have the Can discuss separate to this PR, but ive been thinking about turning down the --no-cache flag, its too confusing + --rebuild should cover its intended use. |
| impl Manager { | ||
| /// Initializes the checkouts manager in the given directory, returning a | ||
| /// thread-safe, cloneable handle to the manager instance. | ||
| pub fn new_in_dir<P: Into<PathBuf>>(base_dir: P) -> Result<ManagerHandle, Error> { |
There was a problem hiding this comment.
Future nit: when this grows more configurations we should create an Options struct and take that for a new method, ala Builder pattern.
There was a problem hiding this comment.
Acknowledged — agreed, will reach for an Options struct + builder when this constructor accumulates more flags. Tracking as future work; left the current shape as-is since it's only one new param.
| /// In offline mode, returns [Error::OfflineCacheMiss] for the first remote we'd | ||
| /// have to fetch. | ||
| pub fn update(&mut self) -> Result<(), Error> { | ||
| if self.offline { |
There was a problem hiding this comment.
This is called from things that really want to be fresh like minimal update - so prefer returning an error rather than silently lameducking; noop'ing here could lead to some hard-to-debug bugs.
Fwiw, update() shouldnt get called willy-nilly.
There was a problem hiding this comment.
Good call — fixed in 048c6b1. update() now returns Error::OfflineCacheMiss in offline mode (using the first known remote, or "<no remotes>" placeholder if state is empty). Test updated to assert the new behavior.
|
it seems cleaner to have --no-fetch-srcs for this. The srcs are not diff levels of the binary/outputs cache or a caches at all - misses aren't tolerated. |
|
Thanks both — agreed on the separation. The conflation here was wrong: "rebuild from source locally" and "I'm fully offline" are distinct semantics that shouldn't share a flag, especially with Refactor plan:
@msample re: the GCS-allowlist question — for the offline use case the URL doesn't matter because the cache is sha-keyed: pre-populate Will push the updated branch shortly. |
Per @msample / @twitchyliquid64 feedback on #202: extending --no-fetch conflated two distinct semantics ("rebuild from source locally" vs "I'm fully offline"). With --rebuild now covering the laptop-rebuild use case independently, --no-fetch's existing meaning is restored exactly and a new --offline flag carries the offline-mode semantics. Behavior changes: - New `--offline` global CLI flag (`crates/minimal/src/main.rs`). - `Config` gains `offline: bool` field, `with_offline()` builder, `is_offline()` accessor (`crates/mctx/src/config.rs`). - `use_remote_cache()` returns `!no_fetch && !is_offline()` — the artifact-cache-skip half of --no-fetch is implied by --offline, so callers who say --offline don't also need --no-fetch. - mctx `RemoteStorage` + `VcsManager` initializers now pass `config.is_offline()` into the underlying offline mode (was `!use_remote_cache()` in the previous patch revision). With this, --no-fetch's existing semantic is unchanged: artifact-cache off, source/VCS fetches still happen. Tests: - 3 new tests in `crates/common/src/file_cache.rs`: - offline_cache_miss_errors_without_fetching - offline_cache_hit_returns_path_without_fetching - online_cache_miss_invokes_backend All three use a `PanickingBackend` for offline cases that asserts the network code path is never reached (the property that matters for hermeticity). - 3 new tests in `crates/checkouts/src/lib.rs`: - offline_unknown_remote_errors - offline_update_is_noop - default_manager_is_online CI: cargo fmt --check + cargo clippy --all-targets -- -D warnings + cargo test all clean. Future-work suggestions noted on the PR (proptest infrastructure, fuzz harness for parser surfaces) — both are valuable but structurally separate from this change and out of scope here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
crates/mctx/src/config.rs (1)
222-228: ⚡ Quick winAdd explicit tests for the
offline×no_fetchbehavior matrix.This method now defines core cache semantics; a small matrix test prevents subtle regressions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mctx/src/config.rs` around lines 222 - 228, Add unit tests covering all four combinations of the Config flags to lock in cache semantics: instantiate Config (or builder) with (offline, no_fetch) = (false,false), (false,true), (true,false), (true,true) and assert the expected outcomes for both the derived network-fetch predicate (!self.no_fetch && !self.offline) and the is_offline() method; reference the is_offline() method and the no_fetch field (and the function or method that computes the fetch-allowed boolean if present) when adding these tests in the config test module so any future changes to is_offline or no_fetch behavior will be caught.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mctx/src/lib.rs`:
- Around line 412-418: The call to common::RemoteStorage::new_with_offline(...)
uses .unwrap(), which can panic; replace the unwrap with proper error
propagation (e.g., use ? or map_err to convert the error into your crate's Error
type) and return a contextual Error from the surrounding function (include
context such as "failed to initialize RemoteStorage with downloads_dir:
{self.config.downloads_dir()}"). Update the surrounding function signature to
return Result if needed and ensure the initialization uses the returned Err
instead of crashing.
---
Nitpick comments:
In `@crates/mctx/src/config.rs`:
- Around line 222-228: Add unit tests covering all four combinations of the
Config flags to lock in cache semantics: instantiate Config (or builder) with
(offline, no_fetch) = (false,false), (false,true), (true,false), (true,true) and
assert the expected outcomes for both the derived network-fetch predicate
(!self.no_fetch && !self.offline) and the is_offline() method; reference the
is_offline() method and the no_fetch field (and the function or method that
computes the fetch-allowed boolean if present) when adding these tests in the
config test module so any future changes to is_offline or no_fetch behavior will
be caught.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c94a0ae4-7af4-4344-b941-f772a3133a93
📒 Files selected for processing (5)
crates/checkouts/src/lib.rscrates/common/src/file_cache.rscrates/mctx/src/config.rscrates/mctx/src/lib.rscrates/minimal/src/main.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- crates/common/src/file_cache.rs
- crates/checkouts/src/lib.rs
…rage init @twitchyliquid64 (#202 review): `Manager::update` should not silently no-op in offline mode. It's called by callers like `minimal update` that explicitly want fresh state, so silently lameducking would mask hard-to-debug bugs. Now returns Error::OfflineCacheMiss with the first known remote (or "<no remotes>" placeholder if state is empty). Test updated to assert the new behavior. @coderabbitai (#202 review): mctx::Context::remote_storage() unwrap'd on RemoteStorage init failure, which can panic on recoverable startup errors. Replaced with .map_err(|e| Error::Other(anyhow!(...)))? to propagate via the existing error type. Drive-by — was pre-existing in the line we already touched, fixing while close. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 423a39a5-2e68-4458-88d3-7ed0b6457535
📒 Files selected for processing (2)
crates/checkouts/src/lib.rscrates/mctx/src/lib.rs
| // Offline: skip the fetch and try the worktree checkout against | ||
| // what's already in the bare repo. If the requested ref isn't | ||
| // there, `new_worktree` will surface a git error from below | ||
| // — which is the right outcome (caller gets to know the ref | ||
| // wasn't pre-populated). | ||
| if !self.offline { | ||
| repo.fetch()?; | ||
| } | ||
| let checkout = repo.new_worktree(checkout_dir.clone(), at.clone())?; |
There was a problem hiding this comment.
Return OfflineCacheMiss for known-remote ref misses in offline mode.
In offline mode, Line 334 can return a raw git error when the requested ref is not already present locally. That is still an offline cache miss and should surface as Error::OfflineCacheMiss { remote } for consistent UX/error mapping.
This currently creates two different error shapes for the same offline-miss class (unknown remote → OfflineCacheMiss, known remote but missing ref → git error).
The patch introduced --offline as a separate flag (per review), but the OfflineCacheMiss error messages still pointed at --no-fetch. Found during a local smoke test where the error pointed operators at the wrong flag. Fixes three sites: - crates/common/src/file_cache.rs:FileCacheError::OfflineCacheMiss - crates/checkouts/src/error.rs:Error::OfflineCacheMiss - crates/mctx/src/error.rs (the From impl for checkouts::Error) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Stacked one more commit (9e6668b): fixes a wording bug in the OfflineCacheMiss error messages. They were still saying Also independently smoke-tested the end-to-end offline flow: |
|
This PR has been waiting for review for 11 days. A maintainer probably If the PR is intentionally on hold, consider converting it to a draft
|
…line mode) The remote artifact cache is sha-verified on hydrate, so cached pulls are hermetically clean even when offline mode forbids source-tarball or VCS fetches. Drop the offline check from use_remote_cache (now !no_fetch only). Motivating use case: minimalmertic hermetic-builder-rs sets with_offline(true) to surface mirror-source gaps as clear errors. It also wants to read from a signer-attested build cache for expensive deps like llvm. Previously these were mutually exclusive; now they compose. Set with_no_fetch explicitly to disable the artifact cache in fully air-gapped builds. Non-breaking for callers that rely on offline implying no_fetch — those callers should now set both flags explicitly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/mctx/src/config.rs (1)
189-191:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winFix stale offline/no_fetch docs and rustdoc intra-link targets
Config::offlinedocs still say it “Implies”[Self::no_fetch], butofflineis independent ofno_fetch(anduse_remote_cacheis gated only onno_fetch).Config::use_remote_cachedocs reference an invalid rustdoc target ([Self::with_no_fetch]);with_no_fetchis onConfigBuilder.Suggested doc fixes
- /// any network call. Implies the artifact-cache-skip half of [Self::no_fetch]. + /// any network call. Independent from [Self::no_fetch]. @@ - /// allowed in offline mode. Set [Self::with_no_fetch] explicitly to + /// allowed in offline mode. Set [ConfigBuilder::with_no_fetch] explicitly to /// disable the artifact cache in fully air-gapped configurations.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mctx/src/config.rs` around lines 189 - 191, Update the field docs so they are accurate: for Config::offline, remove the phrase that says it "Implies the artifact-cache-skip half of [Self::no_fetch]" and state that offline is independent from no_fetch (it errors on cache/VCS misses without changing no_fetch/use_remote_cache behavior); for the Config::use_remote_cache docs fix the broken rustdoc intra-link by referencing the builder method correctly (ConfigBuilder::with_no_fetch or with_no_fetch on ConfigBuilder) instead of the invalid [Self::with_no_fetch] target so the link points to the actual ConfigBuilder method.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/mctx/src/config.rs`:
- Around line 189-191: Update the field docs so they are accurate: for
Config::offline, remove the phrase that says it "Implies the artifact-cache-skip
half of [Self::no_fetch]" and state that offline is independent from no_fetch
(it errors on cache/VCS misses without changing no_fetch/use_remote_cache
behavior); for the Config::use_remote_cache docs fix the broken rustdoc
intra-link by referencing the builder method correctly
(ConfigBuilder::with_no_fetch or with_no_fetch on ConfigBuilder) instead of the
invalid [Self::with_no_fetch] target so the link points to the actual
ConfigBuilder method.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f703b964-b444-42d8-b91a-72df12e3198e
📒 Files selected for processing (1)
crates/mctx/src/config.rs
|
Pushed one more commit on top: This refines the design from earlier review: the remote artifact cache is sha-verified on hydrate (
Motivating use case (concrete): the minimalmertic hermetic-builder-rs sets Non-breaking for callers that relied on the old coupling — they should set both flags explicitly. Default behavior preserved for Diff: Sibling PR #228 carries the related sandbox2 + op changes that the hermetic-builder-rs use case needs alongside this — kept separate because that's CS-sandbox plumbing rather than offline-mode work. |
Summary
Adds a new
--offlineglobal flag for running minimal in network-isolated environments where every input is pre-staged in the local cache. On cache miss, surfaces a clear error rather than attempting any network call.--no-fetchsemantics unchanged — keeps its existing meaning of "skip the remote artifact cache, build from source instead", which composes naturally with--rebuildfor the laptop-rebuild-the-world use case.Motivation
Running
minimal package <pkg>inside a network-isolated build VM (Confidential Space, no internet egress) where source bytes are pre-staged in a sha-pinned mirror. There was no flag combination that meant "use cache only — if anything's missing, fail loud" —--no-fetchonly affected the remote artifact cache, while source URL fetches and git clones quietly hit the network on cache miss.After this patch, the same invocation surfaces:
Design
Per review feedback (thanks @msample, @twitchyliquid64) — extending
--no-fetchwould have conflated two distinct semantics. Cleaner shape:--no-fetch(existing): skip the remote artifact cache. Source/VCS fetches still happen on cache miss. Useful for laptop rebuild-the-world flows alongside--rebuild.--offline(new): treat every cache as authoritative. Source / VCS / artifact-cache misses all error rather than fetch. Implies the artifact-cache-skip half of--no-fetch(you can't reach the artifact cache when offline anyway), so--offlinealone is sufficient.--no-cacheand--rebuild:--offlinedoesn't force a rebuild, it just gates the network.Changes
crates/common/src/file_cache.rsFileCachegainsoffline: bool+new_with_offline(dir, offline)ctor +is_offline()accessor. Existingnew(dir)preserved (defaults tooffline=false).CachingDownloader::downloadcheckscache.offlineafterread_cachedreturnsNone; returnsFileCacheError::OfflineCacheMiss { sha256, filename }instead of falling through to the network fetcher.crates/common/src/remote_storage.rsRemoteStorage::new_with_offline(dir, with_gcloud_auth, offline)forwardsofflineintoFileCache. Defaultnewunchanged.crates/checkouts/src/lib.rsManagergainsoffline: bool+new_in_dir_with_offline. Existingnew_in_dirpreserved.update(): no-op when offline (caller intent is "use what's there"; refresh-of-known-remote is non-destructive to skip).checkout_of()known-remote path: skiprepo.fetch()when offline.checkout_of()unknown-remote path: returnError::OfflineCacheMiss { remote }.crates/checkouts/src/error.rs: newOfflineCacheMissvariant +Display.crates/mctx/src/config.rsConfiggainsoffline: boolfield. Newwith_offline()builder +is_offline()accessor.use_remote_cache()becomes!no_fetch && !is_offline()so--offlinestrictly implies the artifact-cache-skip behavior.crates/mctx/src/lib.rs: VcsManager + RemoteStorage initializers passconfig.is_offline()through to the new_with_offlineconstructors.crates/mctx/src/error.rs: mapscheckouts::Error::OfflineCacheMissto a clear top-level message.crates/minimal/src/main.rs: new--offlineglobal CLI flag wired toConfigBuilder::with_offline.What this is not
~/.cache/minimal/downloads/<sha256>/<filename>and thevcs/directory themselves.--trusted-source-domains-style allowlist (per @msample's prompt). For the offline use case the URL doesn't matter — minimal looks up bytes by sha — but a trusted-domains flag for semi-online flows would be a reasonable separate addition.Test plan
cargo fmt --checkcleancargo clippy --all-targets -- -D warningscleancargo test— full test suite passesfile_cache::tests::offline_cache_miss_errors_without_fetching—PanickingBackendasserts the network path is never invokedfile_cache::tests::offline_cache_hit_returns_path_without_fetching— same assertion under cache-hitfile_cache::tests::online_cache_miss_invokes_backend— sanity check that default behavior is unchangedcheckouts::tests::offline_unknown_remote_errorscheckouts::tests::offline_update_is_noopcheckouts::tests::default_manager_is_onlineminimal --offline package -v --rebuild sederrors withOfflineCacheMissinstead of silently re-fetching.Future work (out of scope here)
Summary by CodeRabbit