diff --git a/README.md b/README.md index 4c9e2594..a8a6e8a7 100644 --- a/README.md +++ b/README.md @@ -278,7 +278,7 @@ These are open pull requests, not yet part of Lens: ([#406](https://github.com/adeelahmad/lense/pull/406)). - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)). - **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates - for ([#602](https://github.com/adeelahmad/lense/pull/602)). + for ([#607](https://github.com/adeelahmad/lense/pull/607)). ## Planned diff --git a/changelog.d/mesh-compute.md b/changelog.d/mesh-compute.md new file mode 100644 index 00000000..165e9a4e --- /dev/null +++ b/changelog.d/mesh-compute.md @@ -0,0 +1,5 @@ +- **Compute nodes run the assistant's code** (in progress). A compute node's install line now has + `LENS_NODE_SANDBOX=1`, which starts an OpenShell gateway on that machine. `lens node` hands Lens the gateway's + client certificate once, kept sealed, and reports whether the gateway answers. Lens reaches the gateway over the + mesh only, with mutual TLS, so `sandbox_nodes` lists the node and `run_python` and `run_bash` can run there + ([Compute nodes](docs/mesh.md#compute-nodes)). diff --git a/docker-compose.node.yml b/docker-compose.node.yml index 871be6eb..a670cf93 100644 --- a/docker-compose.node.yml +++ b/docker-compose.node.yml @@ -1,7 +1,9 @@ # A machine on another Lens install's mesh (docs/mesh.md, Nodes): what install.sh starts with LENS_JOIN and # LENS_JOIN_TOKEN. No database, no web app, nothing published: `mesh` runs Nebula, and `node` (`lens node`) joins once, # then follows the primary's config over the mesh and reports every minute; on a worker node it runs `lens worker` too, -# against the primary's database over the mesh. +# against the primary's database over the mesh. A compute node (LENS_NODE_SANDBOX=1 in its install line, which sets +# COMPOSE_PROFILES=sandbox) also runs an OpenShell gateway, which the primary reaches through the mesh container's relay +# (port 7443 on the mesh address, open to the primary only) to run the assistant's code here. # LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… docker compose -p lens-node -f docker-compose.node.yml up -d --build name: lens-node @@ -20,6 +22,7 @@ services: environment: LENS_MESH_DIR: /mesh LENS_MESH_STATE: /state + LENS_MESH_RELAYS: 7443=openshell-gateway:17670 # a compute node's sandbox gateway, for the primary volumes: - ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro - mesh-data:/mesh @@ -50,12 +53,88 @@ services: # a unit of an auto scaling group: its name, and the steps its worker runs (docs/scaling.md) LENS_SCALING_UNIT: ${LENS_SCALING_UNIT:-} LENS_WORKER_STEPS: ${LENS_WORKER_STEPS:-} + # a compute node's gateway: its client certificate, which `lens node` hands the primary once (docs/mesh.md) + LENS_SANDBOX_TLS_DIR: /sandbox-key + LENS_NODE_GATEWAY: openshell-gateway:17670 volumes: - mesh-data:/mesh - node-data:/data + - sandbox-key:/sandbox-key:ro + restart: unless-stopped + + # A compute node's NVIDIA OpenShell gateway, as on the primary (docker-compose.yml, profile `sandbox`): one sandbox + # container per conversation, made through this machine's Docker; Linux with Landlock (6.2 or newer) only. + # openshell-init makes its certificates once, openshell-key copies the client's for `lens node`, and the sandboxes' + # supervisors reach the gateway on this machine's 127.0.0.1:17670. Nothing else reaches it but the mesh's relay. + openshell-init: + image: ghcr.io/nvidia/openshell/gateway:0.1.4 + profiles: [sandbox] + user: "0" + network_mode: none + command: ["generate-certs", "--output-dir", "/var/lib/openshell/tls", "--server-san", "openshell-gateway"] + environment: + HOME: /var/lib/openshell + volumes: + - &openshell-data + type: bind + source: /var/lib/openshell + target: /var/lib/openshell + bind: + create_host_path: true + restart: "no" + + openshell-key: + image: python:3.12-slim + profiles: [sandbox] + user: "0" + network_mode: none + depends_on: + openshell-init: + condition: service_completed_successfully + command: + - python3 + - -c + - | + import os, shutil + for src, dst in (("ca.crt", "ca.crt"), ("client/tls.crt", "tls.crt"), ("client/tls.key", "tls.key")): + shutil.copyfile("/tls/" + src, "/key/" + dst + ".new") + os.chown("/key/" + dst + ".new", 1000, 1000) + os.chmod("/key/" + dst + ".new", 0o400) + os.replace("/key/" + dst + ".new", "/key/" + dst) + volumes: + - /var/lib/openshell/tls:/tls:ro + - sandbox-key:/key + restart: "no" + + openshell-gateway: + image: ghcr.io/nvidia/openshell/gateway:0.1.4 + profiles: [sandbox] + user: "0" # it drives Docker through its socket + command: ["--bind-address", "0.0.0.0", "--port", "17670"] + security_opt: ["no-new-privileges:true"] + cap_drop: [ALL] + read_only: true + tmpfs: [/tmp] + depends_on: + openshell-key: + condition: service_completed_successfully + environment: + OPENSHELL_GATEWAY_CONFIG: /etc/openshell/gateway.toml + OPENSHELL_DB_URL: "sqlite:/var/lib/openshell/gateway.db?mode=rwc" + OPENSHELL_LOCAL_TLS_DIR: /var/lib/openshell/tls + OPENSHELL_TELEMETRY_ENABLED: "false" + XDG_DATA_HOME: /var/lib/openshell + HOME: /var/lib/openshell + ports: + - "127.0.0.1:17670:17670" # this machine only: the sandboxes' supervisors, on the host's network + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - *openshell-data + - ./sandbox/gateway.toml:/etc/openshell/gateway.toml:ro restart: unless-stopped volumes: mesh-data: mesh-state: node-data: + sandbox-key: diff --git a/docs/assistant.md b/docs/assistant.md index 5a92663d..29a3b472 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -546,9 +546,9 @@ people it's turned on for: those who can edit a namespace, and admins (`sandbox. ### Compute nodes -In progress: the seam is built, and the mesh that joins the nodes is being built in its own pull request. Once the -install's mesh registers compute nodes (its other machines, each with its own sandbox gateway, reached only over the -mesh's own network), the assistant can run a conversation's code on one of them: +Once a compute node on the install's [mesh](mesh.md#compute-nodes) has its own sandbox gateway (its install line +starts one, and hands Lens the gateway's certificate), the assistant can run a conversation's code on it, reached +only over the mesh's own network: - **`sandbox_nodes`** lists where code can run: this machine (`here`) and each compute node, with whether it can run code now and, if not, why (it hasn't reported in the last 3 minutes, it isn't on the mesh, its sandbox gateway isn't diff --git a/docs/mesh.md b/docs/mesh.md index 598e9df4..d5cd1ee6 100644 --- a/docs/mesh.md +++ b/docs/mesh.md @@ -3,8 +3,8 @@ Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API under `/api/v1/mesh`, Nebula running on the primary in the `mesh` container, relaying the database and the API, -`lens node` with the install line that joins another machine, worker nodes running jobs, and Settings › Network and -Nodes in the web app. Coming in its own pull request: running code on compute nodes. +`lens node` with the install line that joins another machine, worker nodes running jobs, compute nodes running the +assistant's code, and Settings › Network and Nodes in the web app. Not yet tried between two real machines. Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a @@ -22,7 +22,7 @@ install. | --- | --- | --- | | **primary** | This install: the lighthouse other machines find each other through, the relay for machines that can't reach each other directly, and the certificate authority | | | **worker** | Runs jobs against this install's database, as `lens worker` does on this machine: transcribing and telling voices apart with the models on that machine ([Worker nodes](#worker-nodes)) | the database port (8000) and the API (8080) | -| **compute** | Runs code in sandboxes for the assistant. It gets nothing of Lens's: no database, no keys | the API (8080) | +| **compute** | Runs the assistant's code in sandboxes, on a gateway of its own ([Compute nodes](#compute-nodes)). It gets nothing of Lens's: no database, no keys | the API (8080) | A worker node needs this install's database to be a SurrealDB server, which Docker installs have (`SURREAL_URL`); on an install with the embedded database only compute nodes can be added. A node may serve some namespaces only @@ -64,7 +64,8 @@ Over the mesh, with its node token (`Authorization: Bearer lensnode1.…`): | --- | --- | | `GET /api/v1/mesh/node/config` | its Nebula config as it is now, and a `version` that changes when the config does (the blocklist grew, the primary's addresses changed): the node reloads Nebula when it sees a new one | | `POST /api/v1/mesh/node/renew` | a new certificate for its public key (the same or a new one), with the groups its row has now. A node renews at a third of `mesh.cert_days` before its certificate runs out | -| `POST /api/v1/mesh/node/beat` | its report, every minute: whether Nebula runs, its peers, its machine, the sandboxes it can run, Nebula's last lines. Shown on the Nodes page | +| `POST /api/v1/mesh/node/beat` | its report, every minute: whether Nebula runs, its peers, its machine, whether its sandbox gateway answers, Nebula's last lines. Shown on the Nodes page | +| `PUT /api/v1/mesh/node/sandbox` | a compute node's sandbox gateway's client certificate (`ca`, `cert`, `key`, PEM), once, and again only when it changes: kept sealed ([Compute nodes](#compute-nodes)) | A removed node is refused on all three. Nodes make these calls at the primary's mesh address (`10.73.0.1:8080` by default), which isn't a host name the web app is served at: the API admits it for these paths alone, while the mesh @@ -116,6 +117,26 @@ and its last lines are in the node's report on the Nodes page. - **Its name** is the node's, or the auto scaling unit's (`LENS_SCALING_UNIT`), and it runs only the steps it's given, never routines or the other services this machine's worker runs. +## Compute nodes + +A compute node's install line has `LENS_NODE_SANDBOX=1` (Settings › Nodes adds it to a compute node's line), which +also starts an NVIDIA OpenShell gateway on that machine, as Settings › Code sandbox does on this one +([Code sandbox](assistant.md#code-sandbox)): `docker-compose.node.yml`'s `sandbox` profile, kept in the node's +`.env` for later runs. Its machine needs what the gateway needs here: Linux with Landlock (6.2 or newer) and Docker. + +- **Its certificate, once.** The gateway makes its certificates on its first start. `lens node` hands this install + the client certificate Lens shows the gateway (`PUT /api/v1/mesh/node/sandbox`), which is kept sealed in the node's + row like a setting's secret; it's sent again only when it changes. +- **Over the mesh only.** Lens reaches the gateway at the node's mesh address, port 7443, which the node's mesh + container relays to the gateway and Nebula's firewall opens to the primary only, with mutual TLS. The gateway's + certificate is checked against the name it was made for (`openshell-gateway`), not the address. +- **Ready when it answers.** Each beat says whether the gateway answers; one that doesn't is in the node's error + ("isn't answering") and the assistant won't pick it. A compute node without a gateway says how to start it. +- **What the assistant gets.** Once a compute node has handed over its certificate, the code sandbox lists it + (`sandbox_nodes`) and runs a conversation's code there when asked ([Compute nodes](assistant.md#compute-nodes)); + removing the node takes it away again, and an undo brings it back. Its sandboxes have the same image, limits and + network setting as this machine's, and none of Lens's data or keys. + ## On this machine Lens's server can't create network devices, and nothing in Lens but Nebula needs to, so on the primary Nebula runs in @@ -197,7 +218,7 @@ In the web app: machine shows in the list when it joins. Every machine is listed, this one first, with its role, mesh address, online or when it was last seen (offline after three minutes without a report), Nebula, its Lens version, its processors, memory and system, when its certificate runs out, the namespaces it serves (ticked in place), its - workers and Nebula's last lines. Click the pencil to rename a node (Enter saves, Escape keeps the name). Remove + workers, on a compute node whether its sandbox gateway runs, and Nebula's last lines. Click the pencil to rename a node (Enter saves, Escape keeps the name). Remove asks nothing: the row stays, dimmed and marked Removed, with Undo while its certificate is good. Join tokens not used yet are listed under it, each with Withdraw. The list refreshes every 15 seconds. @@ -213,6 +234,9 @@ In the web app: the primary's calls to a node carry its agent key. - A worker node never gets the archive's secret, a vault's files or another namespace's: it reads the files of the namespaces it serves, plain, over the mesh. +- A compute node gets nothing of Lens's. Lens holds its gateway's client certificate, sealed, and reaches that gateway + only over the mesh, with mutual TLS on top of Nebula's; the gateway listens on the node's loopback and the mesh + relay, nowhere else. - Removing a node blocks every certificate it was given, on every machine, and its credentials stop at once. - The CA's key and the tokens' secrets are sealed in the database like any setting's secret (`ARCHIVE_SECRET_KEY` or `secret.key`); node tokens are kept only as hashes. @@ -235,4 +259,5 @@ names an address that forwards to it). Nebula's own isolation is upstream's. - A second transport for networks that block UDP. - Rotating the CA. -- Compute nodes running the assistant's sandboxes through the mesh, and their load shown on the Nodes page. +- Compute nodes' load on the Nodes page, and the assistant picking a ready compute node by itself when this machine + has no gateway (a Raspberry Pi primary): today it runs here unless the person or the model names a node. diff --git a/fastapi_backend/app/api/v1/routes/mesh.py b/fastapi_backend/app/api/v1/routes/mesh.py index e220bdc7..7b7ce1a5 100644 --- a/fastapi_backend/app/api/v1/routes/mesh.py +++ b/fastapi_backend/app/api/v1/routes/mesh.py @@ -26,6 +26,7 @@ MeshNodeChange, MeshNodeConfig, MeshRenew, + MeshSandboxGateway, MeshStatus, MeshToken, MeshTokenCreate, @@ -117,7 +118,9 @@ def make_mesh_token(body: MeshTokenCreate, user: AdminWriter, request: Request, auth.audit(db, user.as_audit(), "settings.save", "mesh", ["enabled"]) auth.audit(db, user.as_audit(), "mesh.token.create", f"mesh_token:{tok['id']}", {"role": body.role, "uses": body.uses}) url = web_origin(request) - return MeshTokenCreated(token=MeshToken(**tok), secret=secret, join_url=url, install_line=mesh.join_line(cfg, secret, url)) + return MeshTokenCreated( + token=MeshToken(**tok), secret=secret, join_url=url, install_line=mesh.join_line(cfg, secret, url, role=body.role) + ) @router.delete("/tokens/{token_id}") @@ -220,8 +223,23 @@ def chunks(): return StreamingResponse(chunks(), media_type="application/octet-stream", headers={"ETag": tag}) +@router.put("/node/sandbox") +def node_sandbox(body: MeshSandboxGateway, node: Node, db: Db, cfg: Cfg) -> Ok: + """The client certificate of the calling compute node's sandbox gateway, which Lens keeps sealed and reaches that + gateway over the mesh with. 403 for a node of another role, 400 for what isn't a certificate and key.""" + try: + mesh.sandbox_gateway(db, cfg, node, body.model_dump()) + except PermissionError as e: + raise HTTPException(403, str(e)) from None + except ValueError as e: + raise HTTPException(400, str(e)) from None + return Ok() + + @router.post("/node/beat") def node_beat(body: MeshBeat, node: Node, db: Db) -> Ok: - """The calling node's report, every minute: Nebula, its peers, its machine.""" + """The calling node's report, every minute: Nebula, its peers, its machine, its sandbox gateway.""" mesh.beat(db, node, body.model_dump(exclude_none=True)) + if node["role"] == "compute": + mesh.offer_sandboxes(db) # this process may have started since the node gave its gateway's certificate return Ok() diff --git a/fastapi_backend/app/domain/mesh.py b/fastapi_backend/app/domain/mesh.py index d767bbfe..6a9f0d5b 100644 --- a/fastapi_backend/app/domain/mesh.py +++ b/fastapi_backend/app/domain/mesh.py @@ -564,9 +564,10 @@ def make_token(db, cfg, by=None, role="worker", name=None, namespaces=None, hour return _token_view(row), f"{TOKEN_PREFIX}{tid}.{secret}" -def join_line(cfg, token, url=None): +def join_line(cfg, token, url=None, role=None): """The install line another machine joins with (docs/mesh.md, Joining): install.sh with this install's address - (the web app's: `url`, else FRONTEND_URL) and the token. Shown once, with the token.""" + (the web app's: `url`, else FRONTEND_URL) and the token, and for a compute node LENS_NODE_SANDBOX=1, which starts + its sandbox gateway too. Shown once, with the token.""" from app.config import settings as env repo = cfg["updates"].get("repo") or "adeelahmad/lense" @@ -574,7 +575,8 @@ def join_line(cfg, token, url=None): if not re.fullmatch(r"https?://[A-Za-z0-9.\-\[\]:]+(/[A-Za-z0-9._~/-]*)?", url): raise ValueError(f"{url!r} isn't an address a machine can join at") parse_token(token) - return f"curl -fsSL https://raw.githubusercontent.com/{repo}/main/install.sh | LENS_JOIN={url} LENS_JOIN_TOKEN={token} sh" + extra = " LENS_NODE_SANDBOX=1" if role == "compute" else "" + return f"curl -fsSL https://raw.githubusercontent.com/{repo}/main/install.sh | LENS_JOIN={url} LENS_JOIN_TOKEN={token}{extra} sh" def revoke_token(db, tid): @@ -828,6 +830,74 @@ def media(db, cfg, row, rid): return path +# ---------- compute nodes: the assistant's sandboxes run there ---------- +# A compute node runs its own OpenShell gateway (docker-compose.node.yml, profile `sandbox`), which makes sandboxes in +# that machine's Docker, and the mesh container relays AGENT_PORT on its mesh address (open to the primary only) to it. +# The node hands Lens the gateway's client certificate once (sandbox_gateway), which is kept sealed in its row: Lens +# then reaches the gateway over the mesh with mutual TLS, checking the gateway's certificate against the name it was +# made for (SANDBOX_NAME). The sandbox registers the install's nodes once one has a gateway (offer_sandboxes), so the +# assistant can run code there; a compute node gets nothing of Lens's in return. +SANDBOX_NAME = "openshell-gateway" +PEM_BYTES = 16_384 + + +def _pems(bundle): + """{ca, cert, key} as PEM text, each loaded to be sure it is one; ValueError otherwise.""" + from cryptography import x509 + from cryptography.hazmat.primitives import serialization + + out = {} + for k in ("ca", "cert", "key"): + v = str((bundle or {}).get(k) or "").strip() + if not v or len(v) > PEM_BYTES: + raise ValueError(f"the gateway's {k} is missing or too long") + try: + if k == "key": + serialization.load_pem_private_key(v.encode(), password=None) + else: + x509.load_pem_x509_certificate(v.encode()) + except (ValueError, TypeError): + raise ValueError(f"the gateway's {k} isn't PEM") from None + out[k] = v + "\n" + return out + + +def sandbox_gateway(db, cfg, row, bundle): + """Keep the client certificate of a compute node's gateway, sealed in its row, and offer the assistant its + sandboxes. PermissionError for a node of another role; ValueError for what isn't a certificate and key.""" + if row["role"] != "compute": + raise PermissionError("only a compute node runs sandboxes: this one is a " + row["role"]) + pems = _pems(bundle) + sealed = settings.seal(cfg, json.dumps(pems), f"mesh:sandbox.{int(row['id'])}") + db.q("UPDATE $r SET sandbox_tls = $s", r=R("mesh_node", int(row["id"])), s=sealed) + offer_sandboxes(db) + + +def gateway_driver(db, cfg, node_): + """The sandbox's Driver for a compute node's gateway, at its mesh address; None until the node gave Lens its + certificate.""" + from . import sandbox + + sealed = (db.one("SELECT sandbox_tls FROM $r", r=R("mesh_node", int(node_["id"]))) or {}).get("sandbox_tls") + if not sealed or not node_.get("ip"): + return None + pem = json.loads(settings.unseal(cfg, sealed, f"mesh:sandbox.{int(node_['id'])}")) + return sandbox.OpenShell(cfg, endpoint=f"{node_['ip']}:{AGENT_PORT}", pem=pem, server_name=SANDBOX_NAME) + + +def offer_sandboxes(db): + """Let the assistant's code run on compute nodes while one on the mesh has a gateway, and not otherwise (the + sandbox then has no node to pick, and no sandbox_nodes tool).""" + from . import sandbox + + has = db.one("SELECT count() AS n FROM mesh_node WHERE role = 'compute' AND revoked_at = NONE AND sandbox_tls != NONE GROUP ALL") + if (has or {}).get("n"): + if not sandbox.nodes_on(): + sandbox.register_nodes(lambda d: nodes(d, revoked=False), gateway_driver) + elif sandbox.nodes_on(): + sandbox.register_nodes(None, None) + + # ---------- the admin's changes ---------- def update(db, nid, name=None, namespaces=None): """Rename a node, or change the namespaces it serves (its certificate's groups follow when it next renews, which @@ -862,6 +932,8 @@ def remove(db, nid, by=None): db.q("UPDATE $r SET revoked_at = $t, revoked_by = $b", r=R("mesh_node", int(nid)), t=store.now(), b=by) if n["role"] == "worker": revoke_database(db, nid) + if n["role"] == "compute": + offer_sandboxes(db) return node(db, nid) @@ -874,6 +946,8 @@ def undo_remove(db, nid, cfg): db.q("UPDATE $r SET revoked_at = NONE, revoked_by = NONE", r=R("mesh_node", int(nid))) if n["role"] == "worker": grant_database(db, cfg, nid) # a new password: the node fetches it with its next worker_access + if n["role"] == "compute": + offer_sandboxes(db) return node(db, nid) diff --git a/fastapi_backend/app/domain/mesh_node.py b/fastapi_backend/app/domain/mesh_node.py index 23d39b4f..765e9fb9 100644 --- a/fastapi_backend/app/domain/mesh_node.py +++ b/fastapi_backend/app/domain/mesh_node.py @@ -16,11 +16,16 @@ database through the mesh as the node's own user of it (mesh.worker_access), started again when it stops or that user's password changes, and stopped while the node is removed. The worker's jobs read their files from the primary over the mesh (media()). +4. **A compute node runs sandboxes.** Its install line starts an OpenShell gateway beside it (docker-compose.node.yml, + profile `sandbox`). It hands the primary that gateway's client certificate once (Gateway, mesh.sandbox_gateway) and + reports the gateway in each beat while it answers, so the assistant can run code here; the primary reaches the + gateway through the mesh container's relay, never anything else on this machine. """ from __future__ import annotations import datetime as dt +import hashlib import json import os import pathlib @@ -104,6 +109,7 @@ def __init__(self, state, shared, join_url=None, token=None, name=None, http=Non self.last_beat = 0.0 self._reported = None self.worker = Worker(self) + self.gateway = Gateway(self) # ---------- what's kept ---------- @property @@ -216,16 +222,21 @@ def follow(self, pub): if got["version"] != self.node.get("version") or got.get("not_after") != self.node.get("not_after"): self.node.update(config=got["config"], version=got["version"], not_after=got.get("not_after"), renew_days=got.get("renew_days")) self.save() - if self.node["node"]["role"] == "worker": + role = self.node["node"]["role"] + if role == "worker": self.worker.sync() + sandbox = self.gateway.sync() if role == "compute" else None s = self.service() + mine = self.worker.error or self.gateway.error report = { "nebula": s.get("state") or ("running" if s.get("nebula") else "stopped"), "version": updates.current(), "machine": machine(), - # this call reached the primary: its own errors are past; the mesh container's and the worker's aren't - "error": (str(s["error"]) if s.get("error") else self.worker.error or "")[:500] or None, - "log": ([str(x) for x in (s.get("log") or [])] + self.worker.log)[-40:], + "sandbox": sandbox, + # this call reached the primary: its own errors are past; the mesh container's, the worker's and the + # gateway's aren't + "error": (str(s["error"]) if s.get("error") else mine or "")[:500] or None, + "log": ([str(x) for x in (s.get("log") or [])] + self.worker.log + self.gateway.log)[-40:], } self.http.post(self.api("/node/beat"), json={k: v for k, v in report.items() if v is not None}, headers=self._headers()) @@ -396,6 +407,63 @@ def stop(self): self.key = None +class Gateway: + """A compute node's sandbox gateway (the module's docstring, 4): Agent.follow syncs it every minute. Hands the + primary the gateway's client certificate when it's new, and says whether the gateway answers.""" + + FILES = {"ca": "ca.crt", "cert": "tls.crt", "key": "tls.key"} + + def __init__(self, agent, connect=socket.create_connection): + self.agent, self.connect = agent, connect + self.error = None + self.log = [] + + def _say(self, msg): + say(msg) + self.log = (self.log + [f"{_now().strftime('%H:%M:%S')} {msg}"])[-10:] + + def _fail(self, msg): + if msg != self.error: # said once, not every minute + self._say(msg) + self.error = msg + + def bundle(self): + """The gateway's client certificate as openshell-key left it, {ca, cert, key}; None before it has.""" + d = pathlib.Path(os.environ.get("LENS_SANDBOX_TLS_DIR") or "/sandbox-key") + try: + return {k: (d / f).read_text() for k, f in self.FILES.items()} + except OSError: + return None + + def sync(self): + """The report's `sandbox`: {gateway: "running"} while the gateway answers, None otherwise (error says why).""" + a = self.agent + found = self.bundle() + if not found: + self._fail("this compute node runs no sandbox gateway yet: its install line starts it (LENS_NODE_SANDBOX=1)") + return None + digest = hashlib.sha256(json.dumps(found, sort_keys=True).encode()).hexdigest() + if a.node.get("sandbox_sent") != digest: + r = a.http.put(a.api("/node/sandbox"), json=found, headers=a._headers()) + if r.status_code in (400, 403): + self._fail(f"the install didn't take the gateway's certificate: {_detail(r)}") + return None + r.raise_for_status() + a.node["sandbox_sent"] = digest + a.save() + self._say("gave the install the sandbox gateway's certificate") + host, _, port = (os.environ.get("LENS_NODE_GATEWAY") or "openshell-gateway:17670").rpartition(":") + try: + self.connect((host, int(port)), timeout=3).close() + except (OSError, ValueError) as e: + self._fail(f"the sandbox gateway isn't answering at {host}:{port}: {getattr(e, 'strerror', None) or e}") + return None + if self.error: + self._say("the sandbox gateway answers") + self.error = None + return {"gateway": "running"} + + # ---------- in a worker node's `lens worker`: the files its jobs read ---------- MEDIA_HOURS = 6 diff --git a/fastapi_backend/app/domain/sandbox.py b/fastapi_backend/app/domain/sandbox.py index cc4751c8..caba1f81 100644 --- a/fastapi_backend/app/domain/sandbox.py +++ b/fastapi_backend/app/domain/sandbox.py @@ -346,9 +346,11 @@ def _bundle_at(d): class OpenShell: """An OpenShell gateway, through its Python client (the `sandbox` extra), over mutual TLS when Lens has the - gateway's client certificate (tls_dir).""" + gateway's client certificate (tls_dir). A compute node's gateway is reached at `endpoint` (its mesh address) with + the certificate it gave Lens (`pem`: ca, cert, key), checked against the name its certificate is for + (`server_name`), since that's not its address.""" - def __init__(self, cfg, certs=None, timeout=30.0): + def __init__(self, cfg, certs=None, timeout=30.0, endpoint=None, pem=None, server_name=None): try: import openshell except ImportError: @@ -356,6 +358,9 @@ def __init__(self, cfg, certs=None, timeout=30.0): "the sandbox's client (openshell) isn't installed yet: it's fetched once the sandbox is on (Settings › Components)" ) from None self.sdk = openshell + if pem is not None: + self.client = _client_for(openshell, endpoint, pem, server_name, timeout) + return found = _bundle_at(certs or tls_dir()) tls = openshell.TlsConfig(ca_path=found[0], cert_path=found[1], key_path=found[2]) if found else None self.client = openshell.SandboxClient(settings(cfg)["gateway"], tls=tls, timeout=timeout) @@ -431,6 +436,24 @@ def close(self): self.client.close() +def _client_for(openshell, endpoint, pem, server_name, timeout): + """A SandboxClient over mutual TLS from certificates in memory, checking the gateway's against `server_name`. + OpenShell's client takes neither, so its channel is swapped for one that does (pinned: openshell < 0.2).""" + import grpc + from openshell._proto import openshell_pb2_grpc + from openshell.errors import _error_mapping_channel + + creds = grpc.ssl_channel_credentials( + root_certificates=pem["ca"].encode(), private_key=pem["key"].encode(), certificate_chain=pem["cert"].encode() + ) + options = [("grpc.ssl_target_name_override", server_name)] if server_name else [] + client = openshell.SandboxClient(endpoint, timeout=timeout) # an insecure channel, never used: replaced here + client._channel.close() + client._channel = grpc.secure_channel(endpoint, creds, options=options) + client._stub = openshell_pb2_grpc.OpenShellStub(_error_mapping_channel(client._channel)) + return client + + _driver: Any = None # what tests use instead of a gateway: a function of cfg that returns a Driver diff --git a/fastapi_backend/app/schemas/mesh.py b/fastapi_backend/app/schemas/mesh.py index 541777ac..8e060333 100644 --- a/fastapi_backend/app/schemas/mesh.py +++ b/fastapi_backend/app/schemas/mesh.py @@ -142,11 +142,17 @@ class MeshRenew(RequestModel): public_key: str = Field(max_length=500, description="its Nebula public key, the same or a new one") +class MeshSandboxGateway(RequestModel): + ca: str = Field(max_length=16384, description="the gateway's CA certificate, PEM") + cert: str = Field(max_length=16384, description="the client certificate Lens shows the gateway, PEM") + key: str = Field(max_length=16384, description="its private key, PEM: kept sealed") + + class MeshBeat(RequestModel): nebula: str | None = Field(None, max_length=40, description="running, starting, stopped or failed") version: str | None = Field(None, max_length=40, description="the Lens version it runs") peers: list[dict[str, Any]] | None = Field(None, max_length=500, description="the machines it has tunnels to") machine: dict[str, Any] | None = Field(None, description="its processors, memory and system") - sandbox: dict[str, Any] | None = Field(None, description="the sandboxes it can run") + sandbox: dict[str, Any] | None = Field(None, description="a compute node's sandbox gateway, when it answers: {gateway}") error: str | None = Field(None, max_length=500) log: list[str] | None = Field(None, max_length=40, description="Nebula's last lines") diff --git a/fastapi_backend/tests/api/test_mesh.py b/fastapi_backend/tests/api/test_mesh.py index cce8cd3c..b4ee5f41 100644 --- a/fastapi_backend/tests/api/test_mesh.py +++ b/fastapi_backend/tests/api/test_mesh.py @@ -77,7 +77,9 @@ def test_adding_a_node_makes_the_ca_a_token_and_turns_the_mesh_on(client, db, cf assert r.status_code == 200, r.text made = r.json() assert made["secret"].startswith(mesh.TOKEN_PREFIX) and made["join_url"].startswith("http://") - assert made["install_line"].endswith(f"install.sh | LENS_JOIN={made['join_url']} LENS_JOIN_TOKEN={made['secret']} sh") + assert made["install_line"].endswith( + f"install.sh | LENS_JOIN={made['join_url']} LENS_JOIN_TOKEN={made['secret']} LENS_NODE_SANDBOX=1 sh" + ) # a compute node's assert made["token"]["role"] == "compute" and made["token"]["uses"] == 1 and made["token"]["used"] == 0 after = client.get("/api/v1/mesh", headers=h).json() diff --git a/fastapi_backend/tests/api/test_mesh_compute.py b/fastapi_backend/tests/api/test_mesh_compute.py new file mode 100644 index 00000000..fc16a868 --- /dev/null +++ b/fastapi_backend/tests/api/test_mesh_compute.py @@ -0,0 +1,217 @@ +"""Compute nodes (docs/mesh.md, Compute nodes): a machine that joined as a compute node runs its own OpenShell gateway, +hands this install the gateway's client certificate once, and reports whether the gateway answers. Lens keeps the +certificate sealed and then reaches that gateway over the mesh with mutual TLS, checking it against the name its +certificate was made for, so the assistant's code can run there. The gateway is tests/fake_openshell.py, over TLS.""" + +from __future__ import annotations + +import datetime as dt +import json + +import grpc +import pytest +from cryptography import x509 +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric import ec +from cryptography.x509.oid import NameOID +from openshell._proto import openshell_pb2_grpc + +from app.domain import mesh, sandbox, store +from tests import fake_openshell +from tests.api.test_mesh import admin, nebula, server_db # noqa: F401 - fixtures +from tests.api.test_mesh_node import agent, due, running + + +def _pem(obj): + if isinstance(obj, x509.Certificate): + return obj.public_bytes(serialization.Encoding.PEM).decode() + return obj.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption()).decode() + + +def _cert(subject, key, issuer=None, issuer_key=None, ca=False, sans=()): + now = dt.datetime.now(dt.UTC) + b = ( + x509.CertificateBuilder() + .subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, subject)])) + .issuer_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, issuer or subject)])) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - dt.timedelta(minutes=1)) + .not_valid_after(now + dt.timedelta(days=1)) + .add_extension(x509.BasicConstraints(ca=ca, path_length=None), critical=True) + ) + if sans: + b = b.add_extension(x509.SubjectAlternativeName(list(sans)), critical=False) + return b.sign(issuer_key or key, hashes.SHA256()) + + +@pytest.fixture(scope="module") +def pki(): + """What openshell-init makes: a CA, the gateway's certificate for openshell-gateway, and the client's.""" + ca_key, srv_key, cli_key = (ec.generate_private_key(ec.SECP256R1()) for _ in range(3)) + ca = _cert("openshell-ca", ca_key, ca=True) + srv = _cert("openshell-gateway", srv_key, "openshell-ca", ca_key, sans=[x509.DNSName("openshell-gateway")]) + cli = _cert("openshell-client", cli_key, "openshell-ca", ca_key) + return {"ca": _pem(ca), "server": (_pem(srv_key), _pem(srv)), "cert": _pem(cli), "key": _pem(cli_key)} + + +@pytest.fixture +def gateway(pki, tmp_path): + """The compute node's gateway, which lets in only the client certificate its CA signed.""" + gw = fake_openshell.Gateway(tmp_path / "boxes") + server = grpc.server(fake_openshell.concurrent.futures.ThreadPoolExecutor(max_workers=4)) + openshell_pb2_grpc.add_OpenShellServicer_to_server(gw, server) + creds = grpc.ssl_server_credentials( + [(pki["server"][0].encode(), pki["server"][1].encode())], root_certificates=pki["ca"].encode(), require_client_auth=True + ) + port = server.add_secure_port("127.0.0.1:0", creds) + server.start() + yield gw, port + server.stop(None) + + +@pytest.fixture +def key_dir(tmp_path, pki, monkeypatch): + """The sandbox-key volume as openshell-key fills it on the node.""" + d = tmp_path / "sandbox-key" + d.mkdir() + for k, f in (("ca", "ca.crt"), ("cert", "tls.crt"), ("key", "tls.key")): + (d / f).write_text(pki[k]) + monkeypatch.setenv("LENS_SANDBOX_TLS_DIR", str(d)) + return d + + +@pytest.fixture(autouse=True) +def no_nodes(): + sandbox.register_nodes(None, None) + yield + sandbox.register_nodes(None, None) + + +class Answers: + """The gateway's port as `lens node` checks it: answering, or not.""" + + def __init__(self): + self.up, self.asked = True, [] + + def __call__(self, addr, timeout=None): + self.asked.append(addr) + if not self.up: + raise ConnectionRefusedError(111, "Connection refused") + return self + + def close(self): + pass + + +def on_mesh(client, cfg, h, folders, role="compute"): + made = client.post("/api/v1/mesh/tokens", json={"role": role}, headers=h).json() + a = agent(client, folders, made["secret"], name="gpu box") + a.gateway.connect = Answers() + a.tick() + (folders[1] / "host.pub").write_text(mesh.keygen(cfg)[1]) + running(folders[1]) + a.tick() + return made, a + + +@pytest.fixture +def folders(tmp_path): + return tmp_path / "node", tmp_path / "mesh" + + +def test_a_compute_nodes_install_line_starts_its_gateway(client, db, cfg, nebula, server_db): # noqa: F811 + h = admin(client, db) + compute = client.post("/api/v1/mesh/tokens", json={"role": "compute"}, headers=h).json() + worker = client.post("/api/v1/mesh/tokens", json={"role": "worker"}, headers=h).json() + assert compute["install_line"].endswith(" LENS_NODE_SANDBOX=1 sh") + assert "LENS_NODE_SANDBOX" not in worker["install_line"] + + +def test_a_compute_node_hands_over_its_gateway_and_the_assistant_may_run_code_there(client, db, cfg, nebula, key_dir, pki, folders): # noqa: F811 + h = admin(client, db) + assert not sandbox.nodes_on() + _, a = on_mesh(client, cfg, h, folders) + nid = a.node["node"]["id"] + row = db.one("SELECT sandbox_tls FROM $r", r=store.R("mesh_node", nid)) + assert row["sandbox_tls"] and "PRIVATE KEY" not in json.dumps(row) # kept sealed + assert a.node["sandbox_sent"] and a.gateway.connect.asked == [("openshell-gateway", 17670)] + me = client.get("/api/v1/mesh/nodes", headers=h).json()[1] + assert me["status"]["sandbox"] == {"gateway": "running"} and me["status"].get("error") is None + assert any("gave the install the sandbox gateway's certificate" in line for line in me["status"]["log"]) + + # the sandbox now has the node to run on, ready + assert sandbox.nodes_on() + listed = {n["id"]: n for n in sandbox.node_list(db)["nodes"]} + assert listed[nid]["ready"] is True and listed[nid]["name"] == "gpu box" + + # sent once; again only when the certificate changes + puts = [] + real = a.http.put + a.http.put = lambda *x, **k: puts.append(x) or real(*x, **k) + due(a) + a.tick() + assert not puts + + # the gateway stops answering: the node says so, and the sandbox won't pick it + a.gateway.connect.up = False + due(a) + a.tick() + me = client.get("/api/v1/mesh/nodes", headers=h).json()[1] + assert "sandbox" not in me["status"] and "isn't answering at openshell-gateway:17670" in me["status"]["error"] + assert "its sandbox gateway isn't running" in {n["id"]: n for n in sandbox.node_list(db)["nodes"]}[nid]["why_not"] + + # removing the node takes its sandboxes away from the assistant; an undo gives them back + client.delete(f"/api/v1/mesh/nodes/{nid}", headers=h) + assert not sandbox.nodes_on() + client.post(f"/api/v1/mesh/nodes/{nid}/restore", headers=h) + assert sandbox.nodes_on() + + +def test_without_its_gateway_a_compute_node_says_how_to_start_it(client, db, cfg, nebula, folders, tmp_path, monkeypatch): # noqa: F811 + monkeypatch.setenv("LENS_SANDBOX_TLS_DIR", str(tmp_path / "none")) + h = admin(client, db) + _, a = on_mesh(client, cfg, h, folders) + me = client.get("/api/v1/mesh/nodes", headers=h).json()[1] + assert "LENS_NODE_SANDBOX=1" in me["status"]["error"] and not sandbox.nodes_on() + + +def test_only_a_compute_node_gives_a_gateway_and_only_a_certificate(client, db, cfg, nebula, server_db, key_dir, pki, folders): # noqa: F811 + h = admin(client, db) + _, w = on_mesh(client, cfg, h, folders, role="worker") + r = client.put("/api/v1/mesh/node/sandbox", json={"ca": pki["ca"], "cert": pki["cert"], "key": pki["key"]}, headers=w._headers()) + assert r.status_code == 403 + _, c = on_mesh(client, cfg, h, (folders[0].parent / "c", folders[1].parent / "c-mesh")) + bad = client.put("/api/v1/mesh/node/sandbox", json={"ca": pki["ca"], "cert": "hello", "key": pki["key"]}, headers=c._headers()) + assert bad.status_code == 400 and "cert isn't PEM" in bad.json()["detail"] + nobody = client.put("/api/v1/mesh/node/sandbox", json={"ca": pki["ca"], "cert": pki["cert"], "key": pki["key"]}) + assert nobody.status_code == 401 + + +def test_lens_reaches_the_nodes_gateway_with_its_certificate_checked_by_name(db, cfg, pki, gateway, monkeypatch): + gw, port = gateway + monkeypatch.setattr(mesh, "AGENT_PORT", port) + nid = 7 + db.q("CREATE $r CONTENT {role: 'compute', name: 'gpu box', ip: '127.0.0.1'}", r=store.R("mesh_node", nid)) + row = {"id": nid, "role": "compute"} + with pytest.raises(PermissionError): + mesh.sandbox_gateway(db, cfg, {**row, "role": "worker"}, pki) + mesh.sandbox_gateway(db, cfg, row, pki) + d = mesh.gateway_driver(db, cfg, {"id": nid, "ip": "127.0.0.1"}) + try: + d.health() # TLS checked against openshell-gateway, not 127.0.0.1, and Lens let in with the client certificate + d.create("lens-chat-1", {"image": "python:3.12-slim", "network": "off", "hosts": [], "cpu": 1, "memory": "512Mi"}) + ran = d.exec("lens-chat-1", ["python3", "-c", "print(6 * 7)"], None, 30, 10_000) + assert ran.stdout.strip() == b"42" + finally: + d.close() + + # a client the gateway's CA didn't sign isn't let in + other_key = ec.generate_private_key(ec.SECP256R1()) + stranger = {**pki, "cert": _pem(_cert("someone", other_key)), "key": _pem(other_key)} + d = sandbox.OpenShell(cfg, endpoint=f"127.0.0.1:{port}", pem=stranger, server_name=mesh.SANDBOX_NAME, timeout=5) + try: + with pytest.raises(grpc.RpcError): + d.health() + finally: + d.close() diff --git a/install.sh b/install.sh index 324f142f..c81e9f29 100755 --- a/install.sh +++ b/install.sh @@ -44,6 +44,7 @@ # default, with Docker: no web app, no database, nothing published # LENS_JOIN_TOKEN the join token, used once and never written down # LENS_NODE_NAME the name it joins with (default: LENS_SCALING_UNIT, an auto scaling unit's, else this machine's) +# LENS_NODE_SANDBOX 1 on a compute node's line: it also runs the assistant's code sandbox gateway. Kept for later runs set -eu # All in a function, so nothing runs until the whole script has arrived (curl | sh streams it). @@ -416,6 +417,11 @@ if [ "$NODE" = 1 ]; then eval "v=\${$k:-}" [ -z "$v" ] || setenv "$k" "$v" done + # a compute node also runs the sandbox gateway (docker-compose.node.yml, profile sandbox), kept for later runs + if [ "${LENS_NODE_SANDBOX:-}" = 1 ] || grep -qx 'LENS_NODE_SANDBOX=1' .env; then + setenv LENS_NODE_SANDBOX 1 + setenv COMPOSE_PROFILES sandbox + fi name="${LENS_NODE_NAME:-$(grep '^LENS_NODE_NAME=' .env | cut -d= -f2- || true)}" name="${name:-${LENS_SCALING_UNIT:-}}" setenv LENS_NODE_NAME "${name:-$(hostname 2>/dev/null || uname -n)}" diff --git a/nextjs-frontend/__tests__/settings-mesh.test.tsx b/nextjs-frontend/__tests__/settings-mesh.test.tsx index 97735ced..4a522352 100644 --- a/nextjs-frontend/__tests__/settings-mesh.test.tsx +++ b/nextjs-frontend/__tests__/settings-mesh.test.tsx @@ -192,6 +192,20 @@ describe("Settings › Nodes", () => { }); }); + it("says whether a compute node's sandbox gateway runs", async () => { + const box = { ...GPU, id: 3, name: "sandbox-box", role: "compute", ip: "10.73.0.3" }; + rows = [ + PRIMARY, + { ...box, status: { ...GPU.status, sandbox: { gateway: "running" } } }, + { ...box, id: 4, name: "old-box", ip: "10.73.0.4" }, + ]; + show("nodes"); + const up = await screen.findByRole("listitem", { name: "sandbox-box" }); + expect(within(up).getByText(/sandbox gateway running/)).toBeInTheDocument(); + const down = screen.getByRole("listitem", { name: "old-box" }); + expect(within(down).getByText(/no sandbox gateway/)).toBeInTheDocument(); + }); + it("says when a node was last seen, and its machine", () => { const now = Date.parse("2026-10-18T12:00:00Z"); expect(seenText("2026-10-18T11:58:00Z", now)).toEqual({ online: true, text: "Online · seen 2 minutes ago" }); diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index 8ca209fb..e0dd3aef 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -2477,6 +2477,7 @@ export type { MeshNodesResponses, MeshRelay, MeshRenew, + MeshSandboxGateway, MeshService, MeshStatus, MeshStatusData, @@ -2568,6 +2569,11 @@ export type { NodeMediaErrors, NodeMediaResponses, NodeParam, + NodeSandboxData, + NodeSandboxError, + NodeSandboxErrors, + NodeSandboxResponse, + NodeSandboxResponses, NodeTrace, NodeType, NodeWorkerData, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 252ede34..71acfb7b 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -1276,6 +1276,9 @@ import type { NodeMediaData, NodeMediaErrors, NodeMediaResponses, + NodeSandboxData, + NodeSandboxErrors, + NodeSandboxResponses, NodeWorkerData, NodeWorkerResponses, NotSameSpeakerData, @@ -13702,10 +13705,29 @@ export class Mesh { }); } + /** + * Node Sandbox + * + * The client certificate of the calling compute node's sandbox gateway, which Lens keeps sealed and reaches that + * gateway over the mesh with. 403 for a node of another role, 400 for what isn't a certificate and key. + */ + public static nodeSandbox( + options: Options, + ): RequestResult { + return (options.client ?? client).put({ + url: "/api/v1/mesh/node/sandbox", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } + /** * Node Beat * - * The calling node's report, every minute: Nebula, its peers, its machine. + * The calling node's report, every minute: Nebula, its peers, its machine, its sandbox gateway. */ public static nodeBeat( options: Options, diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 4af7ecec..d44fe7b2 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -13106,7 +13106,7 @@ export type MeshBeat = { /** * Sandbox * - * the sandboxes it can run + * a compute node's sandbox gateway, when it answers: {gateway} */ sandbox?: { [key: string]: unknown; @@ -13377,6 +13377,30 @@ export type MeshRenew = { public_key: string; }; +/** + * MeshSandboxGateway + */ +export type MeshSandboxGateway = { + /** + * Ca + * + * the gateway's CA certificate, PEM + */ + ca: string; + /** + * Cert + * + * the client certificate Lens shows the gateway, PEM + */ + cert: string; + /** + * Key + * + * its private key, PEM: kept sealed + */ + key: string; +}; + /** * MeshService */ @@ -50500,6 +50524,31 @@ export type NodeMediaResponses = { 200: unknown; }; +export type NodeSandboxData = { + body: MeshSandboxGateway; + path?: never; + query?: never; + url: "/api/v1/mesh/node/sandbox"; +}; + +export type NodeSandboxErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type NodeSandboxError = NodeSandboxErrors[keyof NodeSandboxErrors]; + +export type NodeSandboxResponses = { + /** + * Successful Response + */ + 200: Ok; +}; + +export type NodeSandboxResponse = NodeSandboxResponses[keyof NodeSandboxResponses]; + export type NodeBeatData = { body: MeshBeat; path?: never; diff --git a/nextjs-frontend/components/settings/mesh-nodes.tsx b/nextjs-frontend/components/settings/mesh-nodes.tsx index dd65b7d0..843e05d2 100644 --- a/nextjs-frontend/components/settings/mesh-nodes.tsx +++ b/nextjs-frontend/components/settings/mesh-nodes.tsx @@ -35,7 +35,7 @@ const ROLES = [ { value: "compute", label: "Compute", - hint: "Runs the assistant’s code in sandboxes, and gets nothing of Lens’s: no database, no keys.", + hint: "Runs the assistant’s code in sandboxes on a gateway of its own (Linux 6.2 or newer, with Docker), and gets nothing of Lens’s: no database, no keys.", }, ]; const ROLE_LABEL: Record = { primary: "This machine", worker: "Worker", compute: "Compute" }; @@ -48,6 +48,7 @@ type Report = { info?: Machine | null; error?: string | null; log?: string[] | null; + sandbox?: { gateway?: string } | null; }; const upper = (s: string) => (s ? s[0].toUpperCase() + s.slice(1) : s); @@ -127,9 +128,9 @@ export function NodesBody() { {(tokens.data ?? []).length > 0 && }

- Agents do the same over /api/v1/mesh: nodes, tokens and the network. In - progress: compute nodes running the assistant’s code. Today a node joins, renews its certificate and reports - here, and a worker node takes this install’s jobs. + Agents do the same over /api/v1/mesh: nodes, tokens and the network. A node + joins, renews its certificate and reports here; a worker node takes this install’s jobs, and a compute node runs + the assistant’s code.

); @@ -259,6 +260,7 @@ function NodeRow({ node: n }: { node: MeshNode }) { machineText(r.machine ?? r.info), n.not_after ? `certificate until ${shortDate(n.not_after)}` : null, n.workers?.length ? `${n.workers.length} worker${n.workers.length === 1 ? "" : "s"}` : null, + n.role === "compute" && !removed ? (r.sandbox?.gateway ? "sandbox gateway running" : "no sandbox gateway") : null, ].filter(Boolean); const served = n.namespaces ?? []; const error = remove.error ?? restore.error ?? serve.error; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 4c76dd25..672a5d08 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -35041,13 +35041,55 @@ } } }, + "/api/v1/mesh/node/sandbox": { + "put": { + "tags": [ + "mesh" + ], + "summary": "Node Sandbox", + "description": "The client certificate of the calling compute node's sandbox gateway, which Lens keeps sealed and reaches that\ngateway over the mesh with. 403 for a node of another role, 400 for what isn't a certificate and key.", + "operationId": "node_sandbox", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MeshSandboxGateway" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Ok" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/mesh/node/beat": { "post": { "tags": [ "mesh" ], "summary": "Node Beat", - "description": "The calling node's report, every minute: Nebula, its peers, its machine.", + "description": "The calling node's report, every minute: Nebula, its peers, its machine, its sandbox gateway.", "operationId": "node_beat", "requestBody": { "content": { @@ -58002,7 +58044,7 @@ } ], "title": "Sandbox", - "description": "the sandboxes it can run" + "description": "a compute node's sandbox gateway, when it answers: {gateway}" }, "error": { "anyOf": [ @@ -58415,6 +58457,36 @@ ], "title": "MeshRenew" }, + "MeshSandboxGateway": { + "properties": { + "ca": { + "type": "string", + "maxLength": 16384, + "title": "Ca", + "description": "the gateway's CA certificate, PEM" + }, + "cert": { + "type": "string", + "maxLength": 16384, + "title": "Cert", + "description": "the client certificate Lens shows the gateway, PEM" + }, + "key": { + "type": "string", + "maxLength": 16384, + "title": "Key", + "description": "its private key, PEM: kept sealed" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "ca", + "cert", + "key" + ], + "title": "MeshSandboxGateway" + }, "MeshService": { "properties": { "available": { diff --git a/threat-model.json b/threat-model.json index f43336fa..75162926 100644 --- a/threat-model.json +++ b/threat-model.json @@ -147,6 +147,7 @@ "POST /api/v1/mesh/join (a machine joining the mesh)", "/api/v1/mesh/node/config, /node/renew, /node/beat (a node's own calls, with its node token)", "GET /api/v1/mesh/node/worker, /node/media/{id} (a worker node's database user and its jobs' files, with its node token)", + "PUT /api/v1/mesh/node/sandbox (a compute node's sandbox gateway's client certificate, with its node token), and Lens's calls to that gateway on the node's mesh address, port 7443", "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh", "UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the mesh container, while the mesh is on)" ], @@ -791,6 +792,15 @@ "provenance": "documented", "source": "docs/mesh.md \"Worker nodes\"" }, + { + "entry_point": "PUT /api/v1/mesh/node/sandbox (a compute node's sandbox gateway's client certificate, with its node token), and Lens's calls to that gateway on the node's mesh address, port 7443", + "parameter": "a CA certificate, a client certificate and its key, PEM", + "attacker_controllable": "yes", + "component": "optional", + "caller_must_enforce": "compute nodes only (403 for others); each part capped and loaded as a certificate or key (400 otherwise); the key kept sealed like a setting's secret; Lens reaches the gateway over the mesh only (7443 open to the primary only, relayed to the gateway by the node's mesh container), with mutual TLS checked against the name the gateway's certificate was made for; the node gets nothing of Lens's", + "provenance": "documented", + "source": "docs/mesh.md \"Compute nodes\"" + }, { "entry_point": "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh", "parameter": "a node's role, name, namespaces, token hours and uses; the mesh's network, port, hosts, cipher", @@ -1550,7 +1560,7 @@ "property": "P-MESH", "violation_symptom": "info-leak, integrity-bypass", "severity_tier": "security", - "conditions": "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only; a worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads its namespaces' files over the mesh, never a vault's, without the archive's secret", + "conditions": "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only; a worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads its namespaces' files over the mesh, never a vault's, without the archive's secret; a compute node runs the assistant's sandboxes on a gateway of its own, which Lens reaches over the mesh only with mutual TLS, holding its client certificate sealed, and the node gets nothing of Lens's", "provenance": "documented", "source": "docs/mesh.md \"What keeps it safe\"" } diff --git a/threat-model.md b/threat-model.md index a277ce8f..ea6d570d 100644 --- a/threat-model.md +++ b/threat-model.md @@ -337,6 +337,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /api/v1/mesh/join` (a machine joining the mesh) | a join token's id, a nonce, the machine's name and Nebula public key, an HMAC of those keyed with the token's secret, facts about the machine | yes, anyone reaching the web app, once a token exists | x-credential, data | no sign-in: the token's secret never crosses the wire, the HMAC is checked in constant time, wrong proofs count towards a throttle per address (8 in 15 minutes), a token is spent on use (or after `uses`) and expires, and the answer (certificate, config, node token) is sealed with a key derived from the secret, so an eavesdropper on plain http:// learns nothing | *(documented, docs/mesh.md "Joining")* | | `/api/v1/mesh/node/config`, `/node/renew`, `/node/beat` (a node's own calls, with its node token) | a public key to certify; a status report: whether Nebula runs, its peers, its machine, its log lines | yes, a machine that joined (a removed one is refused) | x-credential, data, x-peer | the node token is 256 random bits kept only as a hash; a node gets its own config and a certificate for its own address and groups only, and can't widen them; reports are capped and kept as its status for admins; they arrive at the primary's mesh address, which the API's host check admits for these paths alone while the mesh is on | *(documented, docs/mesh.md "After joining")* | | `GET /api/v1/mesh/node/worker`, `/node/media/{id}` (a worker node's database user and its jobs' files, with its node token) | a recording id; the node's role and namespaces | yes, a worker node that joined (a removed one is refused) | x-credential, data, x-peer | worker nodes only (403 for others); the database user (`lens_node_`) owns Lens's database and nothing else on the server, never root, is removed with the node and made again with a new password on an undo, and isn't kept on the node's disk; files are those of the namespaces the node serves, never a vault's, sent plain over the mesh and cached on the node for 6 hours; the archive's secret never leaves the primary | *(documented, docs/mesh.md "Worker nodes")* | +| `PUT /api/v1/mesh/node/sandbox` (a compute node's sandbox gateway's client certificate, with its node token), and Lens's calls to that gateway on port 7443 of the node's mesh address | a CA certificate, a client certificate and its key, PEM | yes, a compute node that joined (a removed one is refused) | x-credential, x-peer | compute nodes only (403 for others); each part capped and loaded as a certificate or key (400 otherwise); the key kept sealed like a setting's secret; Lens reaches the gateway over the mesh only (Nebula's firewall opens 7443 to the primary only, and the node's mesh container relays it to the gateway), with mutual TLS checked against the name the gateway's certificate was made for; a compute node gets nothing of Lens's in return | *(documented, docs/mesh.md "Compute nodes")* | | `POST /api/v1/mesh/tokens`, `PATCH` and `DELETE /api/v1/mesh/nodes/{id}`, `PUT /api/v1/settings/mesh` | a node's role, name, namespaces, token hours and uses; the mesh's network, port, hosts, cipher | admin only | x-env-config | admin is trusted; audited (`mesh.token.create`, `mesh.join`, `mesh.node.*`); the network can't change once machines have addresses in it | *(documented, docs/mesh.md "Settings")* | | UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the `mesh` container, while the mesh is on) | Nebula handshakes and tunnelled packets from anyone reaching the port; on the mesh, TCP to the database and the API | yes, anyone reaching UDP 4242; on the mesh, a machine with a certificate the mesh's CA signed | data, x-peer | Nebula drops what no certificate of the mesh's CA signs and anything on the blocklist; its firewall, rendered by Lens, lets only `lens-worker` reach 8000 and `lens-node` 8080; the relays listen on the mesh address only; the container is the only one holding NET_ADMIN, makes its own key pair (the private key stays in its own volume) and runs only Nebula's release with the checksum kept in Lens | *(documented, docs/mesh.md "On this machine")* | @@ -595,7 +596,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-BROWSER-GUARD | The Lens extension's side panel reads and acts only on sites the person allowed in it (and the browser granted), and opens an address only on such a site. It asks the person before a click, submit or Enter that the model marks consequential or whose control's words or posting form look so, and, once the answer has read text from outside Lens (`ai.untrusted_asks`), before opening or downloading an address or typing text the person's own message doesn't contain. It never types into password, card or file fields, never reports their values, and shows each action as it happens. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Letting it use your browser")* | Voided by: The consequential check is the model's flag plus a word list (`extensions/browser/src/core/act.js`): a control with no telling words that a misled model doesn't mark runs without asking; synthetic events reach only the page's own handlers. Removing a site from the options stops the side panel acting there, but leaves the browser's access to it for the clipper. Lens sends the ask (`confirm`) and the extension honours it: an extension older than this doesn't ask. | | P-SANDBOX | Code the assistant runs (`run_python`, `run_bash`; `sandbox.enabled`, off by default) runs only for people `sandbox.who` allows (editors and admins, or admins; never viewers), in a sandbox of that conversation's own on the OpenShell gateway, never in Lens's processes. The sandbox gets none of Lens's data, keys or database address: only the code and the files a run copies in, which must be attached to that conversation by that person (25 MB each, 100 MB a run). Its policy allows no egress with `sandbox.network: off`, only pypi.org, files.pythonhosted.org and registry.npmjs.org on 443 with `packages`, and those and the names in `sandbox.hosts` on 80 and 443 with `hosts`; every endpoint is held to public addresses (no private, loopback, link-local or reserved range), so a listed name that resolves to one is refused. A file leaves it only as one of the person's held uploads (25 MB) placed in a namespace by an `import_files` approval, or as a download (`share_sandbox_file`, 25 MB, 20 a conversation) kept encrypted with the conversation, served only to its person as an attachment with `nosniff`, and deleted with the conversation or their account. A sandbox stops after `sandbox.idle_minutes` idle and is deleted after `sandbox.keep_hours` unused or with its conversation. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Code sandbox")* | Voided by: isolation inside the sandbox is OpenShell's (§1.9); `landlock.compatibility: best_effort` runs without Landlock on a kernel that lacks it — `domain/sandbox.py`; whoever the gateway lets in can run sandboxes of their own (in the `sandbox` profile, a holder of its client certificate); a file shared to download is readable by whoever holds `secret.key` or `ARCHIVE_SECRET_KEY` (Q14); what the code prints reaches the model as third-party text (N-PROMPT). | | P-RESEARCH | A research plan (`domain/research.py`) runs only once the person who asked for it approves it: signed in, or with a "yes" in their chat room from someone in `bridge.users`; an API key, an app or an agent (MCP, the assistant) can only decline it, unless an admin turns on `research.agents_skip_review`. The run acts as that person with their access at run time and stops if their account is disabled. It is offered only the read tools of the sources its plan names (never a tool that changes anything), makes at most the plan's tool calls (`research.max_tool_calls`, 40 by default), stops looking things up after `research.max_minutes` and writes up what it found, and checks the budgets of its namespaces and `all` before each stage, waiting or stopping as they say. Its web searches and page reads, and code runs with the network when the plan names the code sandbox, don't ask again after outside text, since the plan approved them. A person has one run going at a time. One process carries a run out at a time (a claim on `research_run:` with a heartbeat every 10 seconds, taken over only after two minutes of silence). Every call it makes is in the activity ledger for the conversation and `research:`. | integrity-bypass | correctness-only | *(documented, docs/assistant.md "Deep research")* | Voided by: `research.agents_skip_review: true` lets an agent's plan run without review — `domain/research.py` `agents_may_approve`; a run's web queries can carry archive text it read, within its plan's tool calls (the misled model of N-PROMPT); a "yes" in a room is the room's people's (N-BRIDGE). | -| P-MESH | The mesh between Lens's machines (`mesh.enabled`, off until an admin adds a node) admits only machines that joined with a token an admin made, which is spent on use and expires; the token never crosses the wire (an HMAC proves it, the answer is sealed with a key derived from it), each machine's certificate names only its own address, role and namespaces, and Nebula's firewall, rendered by Lens, lets worker nodes reach the database port and every node the API port on the primary, and the primary a node's agent port, and nothing else. Removing a node puts every certificate it was given on the blocklist every machine's config carries, and its node token stops working at once. Nodes still sign in to Lens as any client does: the mesh is a transport, not an identity. On the primary Nebula runs in the `mesh` container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the checksum kept in Lens; its relays listen on the mesh address only. A node runs `lens node` and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only. A worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads the files of the namespaces it serves over the mesh, never a vault's, without the archive's secret. | info-leak, integrity-bypass | security-critical | *(documented, docs/mesh.md "What keeps it safe")* | Voided by: the join request goes to whatever address is put in the install line, so a wrong address gives a stranger the token's id (not its secret); a worker node owns Lens's database and keeps plain copies of the files it worked on for a few hours (a trusted peer, Q9); the lighthouse must be reachable on UDP `mesh.port`; Nebula's own isolation is upstream's (§1.9) — `domain/mesh.py`. | +| P-MESH | The mesh between Lens's machines (`mesh.enabled`, off until an admin adds a node) admits only machines that joined with a token an admin made, which is spent on use and expires; the token never crosses the wire (an HMAC proves it, the answer is sealed with a key derived from it), each machine's certificate names only its own address, role and namespaces, and Nebula's firewall, rendered by Lens, lets worker nodes reach the database port and every node the API port on the primary, and the primary a node's agent port, and nothing else. Removing a node puts every certificate it was given on the blocklist every machine's config carries, and its node token stops working at once. Nodes still sign in to Lens as any client does: the mesh is a transport, not an identity. On the primary Nebula runs in the `mesh` container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the checksum kept in Lens; its relays listen on the mesh address only. A node runs `lens node` and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only. A worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads the files of the namespaces it serves over the mesh, never a vault's, without the archive's secret. A compute node runs the assistant's sandboxes on an OpenShell gateway of its own, which Lens reaches over the mesh only, with mutual TLS, holding its client certificate sealed; the node gets nothing of Lens's. | info-leak, integrity-bypass | security-critical | *(documented, docs/mesh.md "What keeps it safe")* | Voided by: the join request goes to whatever address is put in the install line, so a wrong address gives a stranger the token's id (not its secret); a worker node owns Lens's database and keeps plain copies of the files it worked on for a few hours (a trusted peer, Q9); the lighthouse must be reachable on UDP `mesh.port`; Nebula's own isolation is upstream's (§1.9) — `domain/mesh.py`. | ### Worked routing examples diff --git a/threat-model.yaml b/threat-model.yaml index 1dba6f48..64815bc8 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -590,6 +590,15 @@ entry_points: control_kinds: [x-credential, data, x-peer] caller_must_enforce: "worker nodes only (403 for others); the database user is the node's own, of Lens's database only, never the server's root, removed with the node and made again with a new password on an undo; files are those of the namespaces the node serves, never a vault's, sent plain over the mesh; the archive's secret never leaves the primary" provenance: {kind: documented, source: 'docs/mesh.md "Worker nodes"'} + - id: "PUT /api/v1/mesh/node/sandbox (a compute node's sandbox gateway's client certificate, with its node token), and Lens's calls to that gateway on the node's mesh address, port 7443" + component: optional + parameters: + - name: "a CA certificate, a client certificate and its key, PEM" + attacker_controllable: true + x-condition: "a compute node that joined (a removed one is refused)" + control_kinds: [x-credential, x-peer] + caller_must_enforce: "compute nodes only (403 for others); each part capped and loaded as a certificate or key (400 otherwise); the key kept sealed like a setting's secret; Lens reaches the gateway over the mesh only (7443 open to the primary only, relayed to the gateway by the node's mesh container), with mutual TLS checked against the name the gateway's certificate was made for; the node gets nothing of Lens's" + provenance: {kind: documented, source: 'docs/mesh.md "Compute nodes"'} - id: "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh" component: optional parameters: @@ -1992,7 +2001,7 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only; a worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads its namespaces' files over the mesh, never a vault's, without the archive's secret" + conditions: "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only; a worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads its namespaces' files over the mesh, never a vault's, without the archive's secret; a compute node runs the assistant's sandboxes on a gateway of its own, which Lens reaches over the mesh only with mutual TLS, holding its client certificate sealed, and the node gets nothing of Lens's" violation_symptoms: [info-leak, integrity-bypass] x-voided-by: "a wrong address in the install line gives a stranger the token's id (not its secret); a worker node owns Lens's database and keeps plain copies of its files for a few hours (trusted peer, Q9); the lighthouse must be reachable on UDP mesh.port; Nebula's own isolation is upstream's (domain/mesh.py)" provenance: {kind: documented, source: 'docs/mesh.md "What keeps it safe"'}