Repository navigation
docs(concepts): lift the supply-chain, sandboxing, packages, and stacks guides #938
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
4b73824
docs(concepts): lift the supply-chain, sandboxing, packages, and stac…
norrietaylor 3d8b938
docs(concepts): fix cache order, restore the profiles guide, and poli…
norrietaylor 30adac9
docs(concepts): frame mip as the build plane, not the primary surface
norrietaylor ff5a61c
docs(concepts): accuracy fixes and session-sandbox coverage from review
norrietaylor File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| --- | ||
| description: Packages are reproducible, dependency-aware units of software in Minimal. Covers usage, definition, caching, and hermetic builds. | ||
| --- | ||
|
|
||
| # Minimal packages | ||
|
|
||
| Packages are the core of software production in Minimal. Packages are reproducible, maintainable, and dependency-aware building blocks designed for environment parity across the software development lifecycle (SDLC). | ||
|
|
||
| Structurally, a package is the union of its declarative build specification and the resulting binary artifacts. A metaphor for Minimal's packages would be a recipe (the build specification) and the meal that's cooked from it (the resulting artifacts). This relationship is cemented by a [layer of metadata](#definition) that defines the package's dependencies, architecture, and provenance. In our metaphor, these would be the optional modifications made to tailor the dish so it's more to your liking. By binding the "recipe" to the "result," Minimal ensures that you have complete control over every dish you make, verifying that every component of your environment is transparent, verifiable, and maintainable at scale. | ||
|
|
||
| ## Using packages {#using} | ||
|
|
||
| A Minimal package is available anywhere it is called for: | ||
|
|
||
| - A package declared on a [task](../reference/tasks.md) will always be installed in the environment the task runs. | ||
| - A package declared as an input to a [build spec](../reference/build-specs.md) will always be present in the build sandbox. | ||
| - A package declared on a [stack](./stacks.md) will always be present in the tasks and builds that use the stack. | ||
| - A package declared on a [profile](./profiles.md) will always be present in the tasks that use the profile. | ||
|
|
||
|
|
||
| ## Defining packages {#definition} | ||
|
|
||
| Packages are defined in a [build spec](../reference/build-specs.md) file located at `packages/<package name>/build.ncl`. The `packages/` directory | ||
| is always adjacent to the [`minimal.toml`](../reference/minimal-dot-toml.md) file at the base of the layer, but can be omitted if the layer does not | ||
| define any packages. | ||
|
|
||
| The package definition captures a variety of useful information: | ||
|
|
||
| - The build-time and runtime dependencies of the package, defined in terms of other packages | ||
| - How to build the package from source-code: including build commands, where to fetch the source, and what outputs compose the package | ||
| - Runtime semantics, such as state directories or environment variables that need to be configured at runtime | ||
| - Metadata, such as the current version of the software, the software license, and the authoritative source of releases. | ||
|
|
||
| ## Caching {#caching} | ||
|
|
||
| Built packages are cached against their _spec hash_: a cryptographic hash of the metadata of the package, including the recipe for | ||
| building the package (and the recipes for building all its transitive dependencies). Using these hashes as the storage key ensures | ||
| that the exact package described in your repository or [software supply chain](./software-supply-chain.md) is used anywhere it is | ||
| requested, as well as providing a reliable key for fetching packages remotely. | ||
|
|
||
| When Minimal needs a package, it first checks the local cache. If the package is not present locally, Minimal fetches it from the remote binary cache. If the remote fetch is disabled or the package is unavailable there, Minimal builds it locally. | ||
|
|
||
| ## Package builds | ||
|
|
||
| All Minimal package builds take place in a hermetic, isolated environment to prevent interference or non-reproducibility from the machine running the build. | ||
|
|
||
| ## Sideloading additional packages | ||
|
|
||
| Sometimes you may want to use packages defined in a separate repository, but still use the toolchains and software versions defined by your [`[upstream]`](../reference/minimal-dot-toml.md#upstream). | ||
|
|
||
| Sideloads allow you to do this: each sideload specified in your `minimal.toml` brings in additional packages from other repositories, while building those packages | ||
| using the version of packages defined by your [`[upstream]`](../reference/minimal-dot-toml.md#upstream). See the [`sideloading schema`](../reference/minimal-dot-toml.md#sideload) for additional information. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| --- | ||
| description: Profiles declare reusable customizations (packages, env vars, and filesystem mappings) that can be applied to tasks. | ||
| --- | ||
|
|
||
| # Minimal Profiles | ||
|
|
||
| In Minimal, profiles are used to declare customizations such as additional packages, environment variables, or filesystem mappings. These customizations | ||
| can then be easily applied to an environment, such as a [task](../reference/tasks.md). | ||
|
|
||
| ## Defining a profile | ||
|
|
||
| Like packages, profiles are defined in a configuration file located at `profiles/<profile name>/profile.ncl`, either in your | ||
| codebase or any layer above it in the [software supply chain](./software-supply-chain.md). The `profiles/` directory is | ||
| always adjacent to the [`minimal.toml`](../reference/minimal-dot-toml.md) file at the base of the layer, but can be omitted if | ||
| the repository does not define any profiles. | ||
|
|
||
| Like packages, profiles are defined using [Nickel](https://nickel-lang.org/): | ||
|
|
||
| ```ncl | ||
| let { profile, .. } = import "minimal.ncl" in | ||
| profile { | ||
| name = "my-terraform-debug", | ||
|
|
||
| env_vars.TF_LOG = "trace", | ||
| packages = [ | ||
| # Make terraform package always present | ||
| "terraform", | ||
| ], | ||
| } | ||
| ``` | ||
|
|
||
| ## Using a profile | ||
|
|
||
| ### In all tasks | ||
|
|
||
| Setting `defaults.profile` within a codebase [`minimal.toml`](../reference/minimal-dot-toml.md) will apply the profile to all tasks which do not have a profile explicitly set. | ||
|
|
||
| ```toml | ||
| [defaults] | ||
| profile = "my-profile" # my-profile is set on all tasks | ||
| ``` | ||
|
|
||
| ### On specific tasks | ||
|
|
||
| Profiles can be applied individually on tasks as well: | ||
|
|
||
| ```toml | ||
| [tasks.shell] | ||
| profile = "my-profile" | ||
| ``` | ||
|
|
||
| Tasks which explicitly set a profile do not inherit any profile defined in `[defaults]`. Thus, a task can be configured to not use the default profile by setting `profile = ""`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| --- | ||
| description: How Minimal isolates all builds and tasks from the host system using hermetic sandboxes. | ||
| --- | ||
|
|
||
| # Sandboxing | ||
|
|
||
| Sandboxing is essential to ensure all builds and tasks are insulated from the machine they run in. **In Minimal, all tasks and builds run in a sandbox**. | ||
|
|
||
|
|
||
| ## Package builds | ||
|
|
||
| Minimal packages encapsulate all tooling and software, so it's essential they are compiled in a hermetically-sealed environment to provide a strong | ||
| foundation to the rest of the ecosystem. As such, package builds take place in a cleanroom sandbox that shares nothing with the host machine, aside from network access when a dependency calls for it. | ||
|
|
||
| Specifically, the cleanroom sandbox wires: | ||
|
|
||
| - Files representing the build inputs and runtime dependencies of the package | ||
| - Working directories `/build`, `/tmp`, and an empty `/state`. | ||
| - The source of the package being built | ||
| - Network connectivity (when called-for by a dependency) | ||
|
|
||
| At the completion of the build, artifacts are gathered based on the outputs specified in the packages' build-spec, and | ||
| are cached for later consumption when needed by a task or another package build. | ||
|
|
||
| By default, Minimal is configured to fetch completed builds from our binary cache, to avoid a slow process building everything locally the first time it is needed. | ||
| You can force builds to run locally with the build CLI's `--no-fetch` and `--no-cache` flags. | ||
|
|
||
|
|
||
| ## The task sandbox | ||
|
|
||
| When a task is invoked, its configuration is used to setup and launch a task sandbox. This sandbox wires: | ||
|
|
||
| - Files representing the packages requested and their runtime dependencies. The packages requested for a task includes | ||
| any that are explicitly defined on the task, those defined on the task's [profile](./profiles.md) (or the default profile if set), and those defined by the repository stack (if set). | ||
| - The repository's files and directories, from the repository root downward, but not above it. | ||
| - A `/state` directory, which can be shared between tasks and task invocations by specifying a task `state_key`. Package managers | ||
| are typically wired to cache source downloads and intermediate build artifacts in this directory. | ||
| - Pinhole filesystem mappings, as declared by packages. | ||
| - Network connectivity when necessary. | ||
|
|
||
| ## The session sandbox | ||
|
|
||
| A [session](./sessions.md) is hosted in a sandbox of its own: the one you drop | ||
| into when you attach a shell. Its contents are the union of the project's | ||
| `[session]` packages, the packages the repository's `[stack]` declares, and | ||
| whatever your applied [loadouts](./loadouts.md) contribute. The session | ||
| sandbox's working directory is the session's workspace, seeded with a copy of | ||
| your project files at activation. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| --- | ||
| description: Layered, composable model for declarative software dependencies, from public registries to organization-internal packages. | ||
| --- | ||
|
|
||
| # The Minimal software supply chain | ||
|
|
||
| To enable declarative, reproducible, and understandable software in Minimal, all software exists within a layered model | ||
| we refer to as the _software supply chain_. | ||
|
|
||
| The lowest layer of the software supply chain is your codebase, within which you invoke commands in the Minimal CLI. Each layer | ||
| depends on configuration and software in the layer above it, which is itself a codebase. Each codebase links to the layer above it | ||
| via the `[upstream]` section in the [`minimal.toml`](../reference/minimal-dot-toml.md) file: | ||
|
|
||
| ```toml | ||
| [upstream] | ||
| repo = "<git remote URL>" | ||
| branch = "<git branch>" | ||
| locked_commit = "<git commit hash the repo is currently pinned to>" | ||
| ``` | ||
|
|
||
| These all form the software supply chain. | ||
|
|
||
| The most common upstream is our public packages registry: | ||
|
|
||
| ```toml | ||
| [upstream] # Source of software & tooling | ||
| repo = "https://github.com/gominimal/pkgs" | ||
| branch = "main" | ||
| # locked_commit is filled in when you run `min update` | ||
| ``` | ||
|
|
||
| For more information, check out the [minimal.toml page](../reference/minimal-dot-toml.md) in the References section. | ||
|
|
||
| ## Software is configuration | ||
|
|
||
| Software in any real environment is more complicated than binaries copied to disk, and these nuances are the basis of how Minimal operates. By building security into the way that products are created, Minimal makes software secure by default and removes pressure from the engineering staff. | ||
|
|
||
| A piece of software is encapsulated in a [package](./packages.md), defined at any layer of the software supply chain and usable | ||
| by packages at that layer or below. Packages themselves are defined by build specifications: config-as-code that describes the detailed | ||
| semantics of what the software needs to function as well as an exact, reproducible description of how to build the software. | ||
|
|
||
| Dependencies of packages are references to other packages, an approach that enables Minimal to build exact production and development environments | ||
| from simple, composable descriptions of individual packages. | ||
|
|
||
| For readers familiar with Nix/Nixpkgs, package build-specs are similar to Nix derivations. | ||
|
|
||
| ## Everything is (composable) configuration | ||
|
|
||
| In addition to declaring packages, each software supply chain layer also declares [stacks](./stacks.md) and [profiles](./profiles.md). Each layer inherits the definitions from the layers above it and adds its own on top; defining the same name twice is an error, with one exception: a profile may extend an upstream profile via `from_profile`. Your codebase can use packages from the public registry while defining its own stacks or profiles, and an organization's internal registry can add private packages on top of the public one. | ||
|
|
||
| This composability means you can: | ||
|
|
||
| - **Use the public registry as-is** for standard open-source tooling | ||
| - **Add an internal layer** with organization-specific packages, stacks, or policies | ||
| - **Define project-local packages** for software unique to your codebase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| --- | ||
| description: Stacks wire Minimal to build codebases with specific languages and tools. Covers available stacks, usage, and how to define custom ones. | ||
| --- | ||
|
|
||
| # Stacks | ||
|
|
||
| Stacks wire Minimal to operate a codebase with specific tools and commands. When you name a stack | ||
| in your [`minimal.toml`](../reference/minimal-dot-toml.md) file, build-plane commands like `mip build` just work. Additionally, your [tasks](../reference/tasks.md) inherit these familiar tools and semantics automatically. | ||
|
|
||
| Learn more about stacks in [the reference section](../reference/stack-specs.md). | ||
|
|
||
| ## Available stacks | ||
|
|
||
| The [Minimal Public Package Registry](https://github.com/gominimal/pkgs) ships stacks for most common languages and build systems. The registry is the source of truth; the highlights: | ||
|
|
||
| | Stack | Detected by | Build command | | ||
| |---------|-------------|---------------| | ||
| | `go` | `go.mod` + `go.sum` | `go build` | | ||
| | `rust` | `Cargo.toml` | `cargo build --release` | | ||
| | `pnpm` | `pnpm-lock.yaml` | `pnpm install && pnpm build` | | ||
| | `npm` | `package-lock.json` | `npm ci && npm run build` | | ||
| | `bun` | `bun.lock` | `bun install && bun run build` | | ||
| | `deno` | `deno.json` | `deno compile` | | ||
| | `uv` | `uv.lock`, `pyproject.toml` | `uv sync && uv build` | | ||
| | `pip` | `requirements.txt`, `setup.py` | `pip3 install --target ./build .` | | ||
| | `gradle` | `build.gradle` | `gradle build -x test` | | ||
| | `make` | `Makefile` | `./configure && make` | | ||
| | `meson` | `meson.build` | `meson setup && ninja` | | ||
| | `cmake` | `CMakeLists.txt` | `cmake && make` | | ||
| | `zig` | `build.zig` | `zig build -Doptimize=ReleaseSafe` | | ||
|
|
||
| ## Using a stack | ||
|
|
||
| Stacks are declared in the [`[stack]` section](../reference/minimal-dot-toml.md#stack) of your `minimal.toml` file. | ||
|
|
||
| ```toml | ||
| [stack] | ||
| use = "<stack name>" | ||
| ``` | ||
|
|
||
| Any stack defined in your codebase (or defined in any layer in your [software supply chain](./software-supply-chain.md)) can be used. | ||
|
|
||
| When a stack is defined, any [task](../reference/tasks.md) inherits the packages and environment the stack | ||
| declares; the stack also contributes default tasks, such as `build`. | ||
|
|
||
| ### Declaring additional dependencies | ||
|
|
||
| It's quite common for a specific codebase to need additional dependencies, either during the build (build packages) or | ||
| on any system the built software runs (runtime packages). Both can be declared in your `minimal.toml`: | ||
|
|
||
| ```toml | ||
| [stack] | ||
| use = "..." | ||
| build_packages = ["perl"] | ||
| runtime_packages = ["openssl"] | ||
| ``` | ||
|
|
||
| Dependencies can also be added automatically with `min add --build` or `min add --runtime` (see the [`min` reference](../reference/cli-min.md)). | ||
|
|
||
| ### Automatic initialization | ||
|
|
||
| Minimal will auto-detect the stack to use for most languages and package managers. To have the Minimal CLI detect an appropriate stack | ||
| and pre-fill your codebase's [`minimal.toml`](../reference/minimal-dot-toml.md) file, run `min init` in the root directory. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we mention the session sandbox here? Its the sandbox you drop into when you use a session, the union of
[session]and[stack]stuff in yourminimal.tomlas well as your loadout.