diff --git a/docs/internal/release-pipeline.md b/docs/internal/release-pipeline.md new file mode 100644 index 0000000000..8c2da791c8 --- /dev/null +++ b/docs/internal/release-pipeline.md @@ -0,0 +1,179 @@ +--- +title: Release pipeline +description: How releases are built, staged, smoke-tested, promoted, and installed: release.yml, nightly.yml, promote.yml, prune-releases.yml, and the curl|sh installer. +--- + +> This is internal documentation. It is not published to the docs site. + +# Release pipeline + +Distribution is bucket-centric. A release run stages an **immutable** +`gs://minimal-one/versions//` folder (artifacts + a `components` +manifest); which version users actually receive is decided separately by +mutable **channel pointer files** (`stable`, `unstable`, `nightly`) at the +bucket root, written by [`scripts/set-channel.sh`](../../scripts/set-channel.sh). +Staging is inert; flipping a pointer is the single, cheap, reversible action +that ships (or rolls back) a version. All GCS writes authenticate via GitHub +OIDC / Workload Identity Federation. + +## release.yml: build, sign, stage + +[`.github/workflows/release.yml`](../../.github/workflows/release.yml) runs on +manual `workflow_dispatch` (inputs: `dry_run`, `skip_ci_verify`), via +`workflow_call` from nightly.yml, or on a `vMAJOR.MINOR.PATCH[-suffix]` tag +push. + +**verify-ci gate.** The `verify-ci` job requires the five lane aggregators, +`ci-success`, `ci-linux-native-success`, `ci-linux-kvm-success`, +`ci-macos-success`, `ci-shell-installer-success` (the required checks on +`main`; see [docs/ci-strategy.md](../ci-strategy.md)), to have reported +success on the exact commit being released. `skip_ci_verify` is an admin +override for green-but-unreported commits. + +**Build jobs.** + +- `build-release-linux-{amd64,arm64}`: static musl builds of `mip`, `min` + (package `minimal`), and `minimald`, one cargo invocation per package so the + fat-LTO links serialize. The amd64 job additionally builds a native-glibc + `minvmd` against a materialized libkrun prefix (there is currently no + `minvmd-linux-arm64` release artifact). +- `build-release-macos-arm64` (self-hosted Apple Silicon, gated on the + `RUN_MACOS_CI` kill-switch): builds `minvmd` (libkrun / + Hypervisor.framework) and `min`, rewrites minvmd's libkrun linkage to + `@rpath` ([`scripts/rewrite-macos-linkage.sh`](../../scripts/rewrite-macos-linkage.sh)), + verifies `min` links only system libraries, and Developer-ID-signs both + (hardened runtime + timestamp, notarization-ready). +- `build-libkrun-macos-arm64`: builds the trimmed `libkrun.1.dylib` shipped to + macOS users (same pinned build every macOS CI lane tests against) on a + hosted runner; `sign-macos-artifacts` re-signs it and the darwin `gvproxy` + with the Developer ID identity on the self-hosted runner. +- `fetch-release-guest-artifacts`: guest kernel Image and ext4 rootfs (cache + pulls keyed by the pinned upstream commit) plus pin-verified `gvproxy` + binaries, for both guest arches. +- `build-release-initramfs`: packs the guest initramfs (minimald as pid-1) + from the same shipped musl `minimald` binaries. +- `compute-version-string`: reads the version out of a built binary so the + GitHub Release name matches `minimald -V` exactly. + +**release job.** Downloads everything, generates shell completions for `mip`, +`min`, and `minimald`, then: + +- uploads a legacy `minimalone-.tar.zst` bundle to + `gs://minimal-shim/archives/`, retained for backward compatibility with + the legacy `minimal-shim` archive path; +- creates the GitHub Release (tag `release-`, or the pushed `v*` tag; a + suffixed tag becomes a prerelease) with all binaries, guest artifacts, and + `completions.tar.gz`. + +**stage-installer job** (skipped entirely on `dry_run`): + +- [`scripts/stage-release.sh`](../../scripts/stage-release.sh) uploads the + artifacts and a `components` manifest (one row per component/os/arch with + its SHA-256, kind, and install destination: the authoritative per-platform + component list) to immutable `gs://minimal-one/versions//`; +- a version-pinned copy of [`scripts/install.sh`](../../scripts/install.sh) + is uploaded next to them, so each version's install path is self-contained; +- [`scripts/set-channel.sh`](../../scripts/set-channel.sh) points the + `unstable` channel at the new version. `unstable` auto-advances on every + release: no gate. + +## nightly.yml: daily cut + smoke + nightly channel + +[`.github/workflows/nightly.yml`](../../.github/workflows/nightly.yml) runs at +10:00 UTC. A `check` job skips the rebuild when HEAD is already staged (public +HEAD request on the version's `components` manifest), then invokes release.yml +via `workflow_call`. Three smoke jobs run the shared session e2e +([`scripts/session-e2e.sh`](../../scripts/session-e2e.sh)) against the +**shipped** artifacts (native Linux daemon, Linux + KVM microVM, and the +signed macOS binaries assembled in the installer layout), feeding a +skip-tolerant `smoke-success` aggregator. Only then does `promote-nightly` +flip the `nightly` pointer via set-channel.sh. `nightly-tests.yml` is the +separate 06:00 UTC nightly *test* tier, unrelated to releasing; see +[docs/ci-strategy.md](../ci-strategy.md). + +## promote.yml: gated promotion to stable + +[`.github/workflows/promote.yml`](../../.github/workflows/promote.yml) +(workflow name `promote-cli`) is the manual path that moves the `stable` (or +`unstable`) pointer. Inputs: `sha` (defaults to the latest staged version in +the bucket), `target`, `dry_run`, `override_provenance`. The `gate` job opens a +promotion-approval GitHub issue, notifies Slack, and polls until someone on the +workflow's approver allowlist (the initiating user is excluded) comments +`approved` / `denied`; closing the issue without approval counts as denial. + +**nightly-provenance gate.** Before flipping the pointer, the `promote` job +runs [`scripts/verify-nightly-provenance.sh --sha "$SHA"`](../../scripts/verify-nightly-provenance.sh), +which queries GitHub Actions to confirm the version was built by a successful +`nightly.yml` run whose Linux smoke jobs actually ran and passed (the +skip-tolerant `smoke-success` aggregator alone is not trusted, so a no-op +nightly run that skipped its smokes cannot vouch for a SHA; `smoke-macos` may +be skipped by the `RUN_MACOS_CI` kill-switch). This rejects manually +dispatched release.yml runs and tag-push builds. Staging a version does not +make it promotable. The `override_provenance` +emergency input bypasses this gate when checked; use it only for documented +emergency releases that skip the nightly path. + +The `promote` job then runs set-channel.sh, which verifies the version is +actually staged before flipping the pointer. Finally it dispatches a +`reference-docs-promoted` repository event that triggers a rebuild of the +published reference docs at docs.minimal.dev. + +## prune-releases.yml: GitHub Release housekeeping + +[`.github/workflows/prune-releases.yml`](../../.github/workflows/prune-releases.yml) +runs at 04:17 UTC on Mondays and Fridays (plus manual dispatch, which defaults +to a dry-run preview) and deletes auto-cut `release-` GitHub Releases and +their tags once they age out (default: older than 2 months, always keeping the +10 newest). The logic lives in +[`scripts/prune-releases.sh`](../../scripts/prune-releases.sh); exact +`vMAJOR.MINOR.PATCH` releases are never deleted (hard guard), and the +installer serves from GCS, so pruning removes redundant copies, not live +install targets. + +## install.sh: channel → version → components + +[`scripts/install.sh`](../../scripts/install.sh) is the strict-POSIX `curl | +sh` installer. Against `https://storage.googleapis.com/minimal-one` it fetches +the channel pointer (default `stable`; the per-channel endpoints under +`go.minimal.dev/` serve this same script with a +`MINIMAL_INSTALL_TARGET_OVERRIDE=` line injected to pin the target), +resolves the version, fetches the +immutable `versions//components` manifest (refusing an unknown +`# format:` header), and for each row matching the host os/arch downloads, +SHA-256-verifies, and atomically installs the file. The on-disk hash is the +skip oracle, so reruns only touch changed components, and a running daemon is +stopped before an executable is swapped. Per-platform sets (from +stage-release.sh's `COMPONENTS` table): Linux amd64/arm64 get `bin/min`, +`bin/mip`, `bin/minimald`, a `git-remote-min` symlink, and the AppArmor +profile/tunable/loader under `data/`; macOS arm64 gets `bin/min`, +`bin/minvmd`, `bin/gvproxy`, `lib/libkrun.1.dylib`, the `git-remote-min` +symlink, and the guest payload (`data/{vmlinuz,rootfs.img,initramfs.cpio}`). +It also wires shell integration (PATH init files, `min` completions, one +marker-fenced rc block). `--uninstall` reverses all of it offline from the +local install record, keeping user-modified files unless `--force`; +`--purge` also removes the data/state/cache trees. + +## Runbook + +**Cut a release.** + +1. Pick a `main` commit whose five lane aggregators are green. +2. Actions → *Release* → Run workflow (use `dry_run` to rehearse), or push a + `vX.Y.Z` tag for a versioned GitHub Release. +3. The run stages `gs://minimal-one/versions//` and auto-points + `unstable` at it. Nightly runs do the same for `nightly` after smoke. + +**Promote to stable.** + +1. **Verify the SHA was nightly-built.** The promote workflow rejects versions + not built and smoke-tested by `nightly.yml`: manually dispatched or + tag-staged SHAs fail the provenance gate after the multi-hour approval + completes, even if a later no-op nightly run carries the same SHA. If you + need to promote a non-nightly version in an emergency, check + `override_provenance` when dispatching. +2. Actions → *promote-cli* → Run workflow with `target: stable` (optionally a + specific staged `sha`; empty means the latest staged). +3. A second person from the approver allowlist comments `approved` on the + auto-opened issue (the initiator cannot self-approve). +4. The provenance check runs, then set-channel.sh flips the `stable` pointer. + Rollback is the same workflow pointed at a previously staged version. diff --git a/scripts/verify-nightly-provenance.sh b/scripts/verify-nightly-provenance.sh index c77105ef59..827a315c8f 100755 --- a/scripts/verify-nightly-provenance.sh +++ b/scripts/verify-nightly-provenance.sh @@ -5,12 +5,17 @@ # Promotion (promote.yml) only lets a channel point at a version the nightly # release workflow actually produced. A version passes when the GitHub Actions # API shows a successful run of the nightly workflow whose head commit matches -# the staged version's short SHA; otherwise this script exits non-zero and the -# promotion fails before the channel pointer is touched. +# the staged version's short SHA AND whose smoke-success aggregator job actually +# ran (conclusion "success", not "skipped"); otherwise this script exits +# non-zero and the promotion fails before the channel pointer is touched. # -# A nightly run that skipped its build because the SHA was already staged still -# counts: the run pointed the `nightly` channel at that SHA, so the version went -# through the nightly path either way. +# The smoke-success aggregator is skip-tolerant by design (a no-op night or a +# kill-switch skip still concludes "success"), so this script additionally +# requires the Linux smoke jobs themselves to have run and passed. A no-op +# nightly run that skipped its build (the SHA was already staged) skips those +# jobs and does NOT pass verification — the version must have been smoked by a +# run that actually built and tested it. smoke-macos alone may be skipped (the +# RUN_MACOS_CI kill-switch). # # Usage: # scripts/verify-nightly-provenance.sh --sha SHORTSHA [options] @@ -72,18 +77,59 @@ command -v gh >/dev/null 2>&1 || die "gh not found" sha_lc=$(printf '%s' "$SHA" | tr '[:upper:]' '[:lower:]') -# One "head_sha html_url" line per successful run. Capture before grepping so a -# grep -m1 early exit can't SIGPIPE gh under pipefail. +# One "head_sha run_id html_url" line per successful run. Capture before +# grepping so a grep -m1 early exit can't SIGPIPE gh under pipefail. runs=$(gh api --paginate \ "repos/${REPO}/actions/workflows/${WORKFLOW_FILE}/runs?status=success&per_page=100" \ - --jq '.workflow_runs[] | "\(.head_sha) \(.html_url)"') + --jq '.workflow_runs[] | "\(.head_sha) \(.id) \(.html_url)"') # Anchor the prefix match to the head_sha field (hex run, then the space -# before the URL) so the input can only ever match a prefix of the commit SHA. +# before the run_id) so the input can only ever match a prefix of the commit SHA. match=$(printf '%s\n' "$runs" | grep -m1 -E "^${sha_lc}[0-9a-f]* " || true) [ -n "$match" ] || die "version '$SHA' was not built by a successful ${WORKFLOW_FILE} run in ${REPO}; \ promote a nightly-built version, or re-dispatch with the emergency provenance override" -printf 'verify-nightly-provenance: %s was built by %s run %s\n' \ - "$SHA" "$WORKFLOW_FILE" "${match#* }" >&2 +# Extract run_id (second field) and html_url (third field onwards). +run_id="${match#* }" # drop head_sha +run_id="${run_id%% *}" # keep only run_id +run_url="${match#* }" # drop head_sha +run_url="${run_url#* }" # drop run_id, leaving html_url + +# Verify that smoke tests actually ran and passed. The smoke-success aggregator +# is `if: always()` and skip-tolerant: on a no-op nightly run (the SHA was +# already staged, so build and smokes were all skipped) it still concludes +# "success". Checking it alone would let an unsmoked version through, so the +# Linux smoke jobs are checked individually below; smoke-success is still +# required to catch failures and cancellations the aggregator does red on. +smoke_jobs=$(gh api \ + "repos/${REPO}/actions/runs/${run_id}/jobs?per_page=100" \ + --jq '.jobs[] | select(.name == "smoke-success" or .name == "smoke-linux-amd64" or .name == "smoke-linux-kvm") | "\(.name) \(.conclusion)"' 2>/dev/null || true) + +smoke_conclusion=$(printf '%s\n' "$smoke_jobs" | awk '$1 == "smoke-success" { print $2; exit }') + +if [ "$smoke_conclusion" != "success" ]; then + if [ -z "$smoke_conclusion" ]; then + die "version '$SHA' in ${WORKFLOW_FILE} run ${run_url} has no smoke-success job; \ +smoke tests must complete before promotion" + else + die "version '$SHA' in ${WORKFLOW_FILE} run ${run_url} has smoke-success with conclusion '$smoke_conclusion' (not 'success'); \ +smoke tests must pass before promotion" + fi +fi + +# The aggregator passed; now require the Linux smoke jobs to have actually run +# and succeeded, so a no-op night (all smokes skipped) cannot wrap an unsmoked +# SHA into a promotable run. smoke-macos is deliberately not required: the +# RUN_MACOS_CI kill-switch may skip it, and smoke-success already reds on its +# genuine failures. +for job in smoke-linux-amd64 smoke-linux-kvm; do + conclusion=$(printf '%s\n' "$smoke_jobs" | awk -v j="$job" '$1 == j { print $2; exit }') + if [ "$conclusion" != "success" ]; then + die "version '$SHA' in ${WORKFLOW_FILE} run ${run_url} has ${job} with conclusion '${conclusion:-absent}' (not 'success'); \ +the smoke jobs must actually run and pass before promotion (a no-op nightly run does not count)" + fi +done + +printf 'verify-nightly-provenance: %s was built by %s run %s (smoke-success: %s, linux smokes: success)\n' \ + "$SHA" "$WORKFLOW_FILE" "$run_url" "$smoke_conclusion" >&2 diff --git a/scripts/verify-nightly-provenance_test.sh b/scripts/verify-nightly-provenance_test.sh index 7e0157a735..39869c82fe 100755 --- a/scripts/verify-nightly-provenance_test.sh +++ b/scripts/verify-nightly-provenance_test.sh @@ -23,12 +23,25 @@ cat >"$root/bin/gh" <<'EOF' printf '%s\n' "$@" >>"${GH_STUB_ARGS:?}" code="${GH_STUB_EXIT:-0}" [ "$code" -eq 0 ] || exit "$code" -cat "${GH_STUB_RUNS:?}" + +# Route requests to the appropriate canned response based on the API path. +# $2 is the API path (api [opts...]). +case "$2" in + *"/actions/runs/"*"/jobs"*) + # Jobs endpoint: return canned jobs response. + cat "${GH_STUB_JOBS:?}" + ;; + *) + # Workflow runs endpoint: return canned runs list. + cat "${GH_STUB_RUNS:?}" + ;; +esac EOF chmod +x "$root/bin/gh" export PATH="$root/bin:$PATH" export GH_STUB_ARGS="$root/gh-args" export GH_STUB_RUNS="$root/runs" +export GH_STUB_JOBS="$root/jobs" unset GH_STUB_EXIT pass=0 fail=0 @@ -52,12 +65,19 @@ expect() { fi } -# One successful nightly run per line: " ". +# One successful nightly run per line: " ". sha_a="dd20ee67e6349ca8573df43e16abd407a617a0df" sha_b="0badf00d5eed5eed5eed5eed5eed5eed5eed5eed" cat >"$GH_STUB_RUNS" <"$GH_STUB_JOBS" <"$GH_STUB_ARGS" @@ -95,6 +115,8 @@ expect 0 "https://example.test/runs/2" "match on a later line is found" \ -- "$script" --sha 0badf00d --repo acme/widgets expect 0 "https://example.test/runs/1" "uppercase input is normalized" \ -- "$script" --sha DD20EE67 --repo acme/widgets +expect 0 "smoke-success: success" "success message includes smoke-success status" \ + -- "$script" --sha dd20ee67 --repo acme/widgets expect 1 "was not built by a successful" "unknown sha fails and mentions the override" \ -- "$script" --sha deadbeef --repo acme/widgets expect 1 "provenance override" "failure message points at the emergency override" \ @@ -110,6 +132,73 @@ if grep -q "repos/acme/widgets/actions/workflows/custom.yml/runs?status=success" else bad "queries the requested repo and workflow (args: $(cat "$GH_STUB_ARGS"))" fi +# Verify jobs endpoint is queried for the matched run. +if grep -q "repos/acme/widgets/actions/runs/1001/jobs" "$GH_STUB_ARGS"; then + ok "queries jobs endpoint for matched run" +else + bad "queries jobs endpoint for matched run (args: $(cat "$GH_STUB_ARGS"))" +fi + +# ── Smoke-success job verification ─────────────────────────────────────────── +# Reset to valid runs data for smoke tests. +cat >"$GH_STUB_RUNS" <"$GH_STUB_JOBS" +expect 1 "conclusion 'skipped'" "skipped smoke-success fails with clear message" \ + -- "$script" --sha dd20ee67 --repo acme/widgets +expect 1 "smoke tests must pass" "skipped smoke-success mentions smoke tests must pass" \ + -- "$script" --sha dd20ee67 --repo acme/widgets + +# The real no-op nightly shape: smoke-success is if: always() and concludes +# "success" while every smoke job was skipped. Must NOT pass verification. +cat >"$GH_STUB_JOBS" <"$GH_STUB_JOBS" <"$GH_STUB_JOBS" +expect 1 "no smoke-success job" "missing smoke-success job fails with clear message" \ + -- "$script" --sha dd20ee67 --repo acme/widgets +expect 1 "smoke tests must complete" "missing smoke-success mentions tests must complete" \ + -- "$script" --sha dd20ee67 --repo acme/widgets + +# Failed smoke-success: smoke tests ran but failed. +echo "smoke-success failure" >"$GH_STUB_JOBS" +expect 1 "conclusion 'failure'" "failed smoke-success fails with clear message" \ + -- "$script" --sha dd20ee67 --repo acme/widgets + +# Cancelled smoke-success: run was cancelled mid-flight. +echo "smoke-success cancelled" >"$GH_STUB_JOBS" +expect 1 "conclusion 'cancelled'" "cancelled smoke-success fails with clear message" \ + -- "$script" --sha dd20ee67 --repo acme/widgets + +# Restore success for remaining tests. +cat >"$GH_STUB_JOBS" <