Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ These are open pull requests, not yet part of Lens:
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
- **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates
for ([#602](https://github.com/adeelahmad/lense/pull/602)).
for ([#607](https://github.com/adeelahmad/lense/pull/607)).

## Planned

Expand Down
5 changes: 5 additions & 0 deletions changelog.d/mesh-compute.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Compute nodes run the assistant's code** (in progress). A compute node's install line now has
`LENS_NODE_SANDBOX=1`, which starts an OpenShell gateway on that machine. `lens node` hands Lens the gateway's
client certificate once, kept sealed, and reports whether the gateway answers. Lens reaches the gateway over the
mesh only, with mutual TLS, so `sandbox_nodes` lists the node and `run_python` and `run_bash` can run there
([Compute nodes](docs/mesh.md#compute-nodes)).
81 changes: 80 additions & 1 deletion docker-compose.node.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# A machine on another Lens install's mesh (docs/mesh.md, Nodes): what install.sh starts with LENS_JOIN and
# LENS_JOIN_TOKEN. No database, no web app, nothing published: `mesh` runs Nebula, and `node` (`lens node`) joins once,
# then follows the primary's config over the mesh and reports every minute; on a worker node it runs `lens worker` too,
# against the primary's database over the mesh.
# against the primary's database over the mesh. A compute node (LENS_NODE_SANDBOX=1 in its install line, which sets
# COMPOSE_PROFILES=sandbox) also runs an OpenShell gateway, which the primary reaches through the mesh container's relay
# (port 7443 on the mesh address, open to the primary only) to run the assistant's code here.
# LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… docker compose -p lens-node -f docker-compose.node.yml up -d --build
name: lens-node

Expand All @@ -20,6 +22,7 @@ services:
environment:
LENS_MESH_DIR: /mesh
LENS_MESH_STATE: /state
LENS_MESH_RELAYS: 7443=openshell-gateway:17670 # a compute node's sandbox gateway, for the primary
volumes:
- ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro
- mesh-data:/mesh
Expand Down Expand Up @@ -50,12 +53,88 @@ services:
# a unit of an auto scaling group: its name, and the steps its worker runs (docs/scaling.md)
LENS_SCALING_UNIT: ${LENS_SCALING_UNIT:-}
LENS_WORKER_STEPS: ${LENS_WORKER_STEPS:-}
# a compute node's gateway: its client certificate, which `lens node` hands the primary once (docs/mesh.md)
LENS_SANDBOX_TLS_DIR: /sandbox-key
LENS_NODE_GATEWAY: openshell-gateway:17670
volumes:
- mesh-data:/mesh
- node-data:/data
- sandbox-key:/sandbox-key:ro
restart: unless-stopped

# A compute node's NVIDIA OpenShell gateway, as on the primary (docker-compose.yml, profile `sandbox`): one sandbox
# container per conversation, made through this machine's Docker; Linux with Landlock (6.2 or newer) only.
# openshell-init makes its certificates once, openshell-key copies the client's for `lens node`, and the sandboxes'
# supervisors reach the gateway on this machine's 127.0.0.1:17670. Nothing else reaches it but the mesh's relay.
openshell-init:
image: ghcr.io/nvidia/openshell/gateway:0.1.4
profiles: [sandbox]
user: "0"
network_mode: none
command: ["generate-certs", "--output-dir", "/var/lib/openshell/tls", "--server-san", "openshell-gateway"]
environment:
HOME: /var/lib/openshell
volumes:
- &openshell-data
type: bind
source: /var/lib/openshell
target: /var/lib/openshell
bind:
create_host_path: true
restart: "no"

openshell-key:
image: python:3.12-slim
profiles: [sandbox]
user: "0"
network_mode: none
depends_on:
openshell-init:
condition: service_completed_successfully
command:
- python3
- -c
- |
import os, shutil
for src, dst in (("ca.crt", "ca.crt"), ("client/tls.crt", "tls.crt"), ("client/tls.key", "tls.key")):
shutil.copyfile("/tls/" + src, "/key/" + dst + ".new")
os.chown("/key/" + dst + ".new", 1000, 1000)
os.chmod("/key/" + dst + ".new", 0o400)
os.replace("/key/" + dst + ".new", "/key/" + dst)
volumes:
- /var/lib/openshell/tls:/tls:ro
- sandbox-key:/key
restart: "no"

openshell-gateway:
image: ghcr.io/nvidia/openshell/gateway:0.1.4
profiles: [sandbox]
user: "0" # it drives Docker through its socket
command: ["--bind-address", "0.0.0.0", "--port", "17670"]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
read_only: true
tmpfs: [/tmp]
depends_on:
openshell-key:
condition: service_completed_successfully
environment:
OPENSHELL_GATEWAY_CONFIG: /etc/openshell/gateway.toml
OPENSHELL_DB_URL: "sqlite:/var/lib/openshell/gateway.db?mode=rwc"
OPENSHELL_LOCAL_TLS_DIR: /var/lib/openshell/tls
OPENSHELL_TELEMETRY_ENABLED: "false"
XDG_DATA_HOME: /var/lib/openshell
HOME: /var/lib/openshell
ports:
- "127.0.0.1:17670:17670" # this machine only: the sandboxes' supervisors, on the host's network
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- *openshell-data
- ./sandbox/gateway.toml:/etc/openshell/gateway.toml:ro
restart: unless-stopped

volumes:
mesh-data:
mesh-state:
node-data:
sandbox-key:
6 changes: 3 additions & 3 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -546,9 +546,9 @@ people it's turned on for: those who can edit a namespace, and admins (`sandbox.

### Compute nodes

In progress: the seam is built, and the mesh that joins the nodes is being built in its own pull request. Once the
install's mesh registers compute nodes (its other machines, each with its own sandbox gateway, reached only over the
mesh's own network), the assistant can run a conversation's code on one of them:
Once a compute node on the install's [mesh](mesh.md#compute-nodes) has its own sandbox gateway (its install line
starts one, and hands Lens the gateway's certificate), the assistant can run a conversation's code on it, reached
only over the mesh's own network:

- **`sandbox_nodes`** lists where code can run: this machine (`here`) and each compute node, with whether it can run
code now and, if not, why (it hasn't reported in the last 3 minutes, it isn't on the mesh, its sandbox gateway isn't
Expand Down
37 changes: 31 additions & 6 deletions docs/mesh.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the
token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API
under `/api/v1/mesh`, Nebula running on the primary in the `mesh` container, relaying the database and the API,
`lens node` with the install line that joins another machine, worker nodes running jobs, and Settings › Network and
Nodes in the web app. Coming in its own pull request: running code on compute nodes.
`lens node` with the install line that joins another machine, worker nodes running jobs, compute nodes running the
assistant's code, and Settings › Network and Nodes in the web app. Not yet tried between two real machines.

Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code
on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a
Expand All @@ -22,7 +22,7 @@ install.
| --- | --- | --- |
| **primary** | This install: the lighthouse other machines find each other through, the relay for machines that can't reach each other directly, and the certificate authority | |
| **worker** | Runs jobs against this install's database, as `lens worker` does on this machine: transcribing and telling voices apart with the models on that machine ([Worker nodes](#worker-nodes)) | the database port (8000) and the API (8080) |
| **compute** | Runs code in sandboxes for the assistant. It gets nothing of Lens's: no database, no keys | the API (8080) |
| **compute** | Runs the assistant's code in sandboxes, on a gateway of its own ([Compute nodes](#compute-nodes)). It gets nothing of Lens's: no database, no keys | the API (8080) |

A worker node needs this install's database to be a SurrealDB server, which Docker installs have (`SURREAL_URL`);
on an install with the embedded database only compute nodes can be added. A node may serve some namespaces only
Expand Down Expand Up @@ -64,7 +64,8 @@ Over the mesh, with its node token (`Authorization: Bearer lensnode1.…`):
| --- | --- |
| `GET /api/v1/mesh/node/config` | its Nebula config as it is now, and a `version` that changes when the config does (the blocklist grew, the primary's addresses changed): the node reloads Nebula when it sees a new one |
| `POST /api/v1/mesh/node/renew` | a new certificate for its public key (the same or a new one), with the groups its row has now. A node renews at a third of `mesh.cert_days` before its certificate runs out |
| `POST /api/v1/mesh/node/beat` | its report, every minute: whether Nebula runs, its peers, its machine, the sandboxes it can run, Nebula's last lines. Shown on the Nodes page |
| `POST /api/v1/mesh/node/beat` | its report, every minute: whether Nebula runs, its peers, its machine, whether its sandbox gateway answers, Nebula's last lines. Shown on the Nodes page |
| `PUT /api/v1/mesh/node/sandbox` | a compute node's sandbox gateway's client certificate (`ca`, `cert`, `key`, PEM), once, and again only when it changes: kept sealed ([Compute nodes](#compute-nodes)) |

A removed node is refused on all three. Nodes make these calls at the primary's mesh address (`10.73.0.1:8080` by
default), which isn't a host name the web app is served at: the API admits it for these paths alone, while the mesh
Expand Down Expand Up @@ -116,6 +117,26 @@ and its last lines are in the node's report on the Nodes page.
- **Its name** is the node's, or the auto scaling unit's (`LENS_SCALING_UNIT`), and it runs only the steps it's
given, never routines or the other services this machine's worker runs.

## Compute nodes

A compute node's install line has `LENS_NODE_SANDBOX=1` (Settings › Nodes adds it to a compute node's line), which
also starts an NVIDIA OpenShell gateway on that machine, as Settings › Code sandbox does on this one
([Code sandbox](assistant.md#code-sandbox)): `docker-compose.node.yml`'s `sandbox` profile, kept in the node's
`.env` for later runs. Its machine needs what the gateway needs here: Linux with Landlock (6.2 or newer) and Docker.

- **Its certificate, once.** The gateway makes its certificates on its first start. `lens node` hands this install
the client certificate Lens shows the gateway (`PUT /api/v1/mesh/node/sandbox`), which is kept sealed in the node's
row like a setting's secret; it's sent again only when it changes.
- **Over the mesh only.** Lens reaches the gateway at the node's mesh address, port 7443, which the node's mesh
container relays to the gateway and Nebula's firewall opens to the primary only, with mutual TLS. The gateway's
certificate is checked against the name it was made for (`openshell-gateway`), not the address.
- **Ready when it answers.** Each beat says whether the gateway answers; one that doesn't is in the node's error
("isn't answering") and the assistant won't pick it. A compute node without a gateway says how to start it.
- **What the assistant gets.** Once a compute node has handed over its certificate, the code sandbox lists it
(`sandbox_nodes`) and runs a conversation's code there when asked ([Compute nodes](assistant.md#compute-nodes));
removing the node takes it away again, and an undo brings it back. Its sandboxes have the same image, limits and
network setting as this machine's, and none of Lens's data or keys.

## On this machine

Lens's server can't create network devices, and nothing in Lens but Nebula needs to, so on the primary Nebula runs in
Expand Down Expand Up @@ -197,7 +218,7 @@ In the web app:
machine shows in the list when it joins. Every machine is listed, this one first, with its role, mesh address,
online or when it was last seen (offline after three minutes without a report), Nebula, its Lens version, its
processors, memory and system, when its certificate runs out, the namespaces it serves (ticked in place), its
workers and Nebula's last lines. Click the pencil to rename a node (Enter saves, Escape keeps the name). Remove
workers, on a compute node whether its sandbox gateway runs, and Nebula's last lines. Click the pencil to rename a node (Enter saves, Escape keeps the name). Remove
asks nothing: the row stays, dimmed and marked Removed, with Undo while its certificate is good. Join tokens not
used yet are listed under it, each with Withdraw. The list refreshes every 15 seconds.

Expand All @@ -213,6 +234,9 @@ In the web app:
the primary's calls to a node carry its agent key.
- A worker node never gets the archive's secret, a vault's files or another namespace's: it reads the files of the
namespaces it serves, plain, over the mesh.
- A compute node gets nothing of Lens's. Lens holds its gateway's client certificate, sealed, and reaches that gateway
only over the mesh, with mutual TLS on top of Nebula's; the gateway listens on the node's loopback and the mesh
relay, nowhere else.
- Removing a node blocks every certificate it was given, on every machine, and its credentials stop at once.
- The CA's key and the tokens' secrets are sealed in the database like any setting's secret (`ARCHIVE_SECRET_KEY` or
`secret.key`); node tokens are kept only as hashes.
Expand All @@ -235,4 +259,5 @@ names an address that forwards to it). Nebula's own isolation is upstream's.

- A second transport for networks that block UDP.
- Rotating the CA.
- Compute nodes running the assistant's sandboxes through the mesh, and their load shown on the Nodes page.
- Compute nodes' load on the Nodes page, and the assistant picking a ready compute node by itself when this machine
has no gateway (a Raspberry Pi primary): today it runs here unless the person or the model names a node.
22 changes: 20 additions & 2 deletions fastapi_backend/app/api/v1/routes/mesh.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
MeshNodeChange,
MeshNodeConfig,
MeshRenew,
MeshSandboxGateway,
MeshStatus,
MeshToken,
MeshTokenCreate,
Expand Down Expand Up @@ -117,7 +118,9 @@ def make_mesh_token(body: MeshTokenCreate, user: AdminWriter, request: Request,
auth.audit(db, user.as_audit(), "settings.save", "mesh", ["enabled"])
auth.audit(db, user.as_audit(), "mesh.token.create", f"mesh_token:{tok['id']}", {"role": body.role, "uses": body.uses})
url = web_origin(request)
return MeshTokenCreated(token=MeshToken(**tok), secret=secret, join_url=url, install_line=mesh.join_line(cfg, secret, url))
return MeshTokenCreated(
token=MeshToken(**tok), secret=secret, join_url=url, install_line=mesh.join_line(cfg, secret, url, role=body.role)
)


@router.delete("/tokens/{token_id}")
Expand Down Expand Up @@ -220,8 +223,23 @@ def chunks():
return StreamingResponse(chunks(), media_type="application/octet-stream", headers={"ETag": tag})


@router.put("/node/sandbox")
def node_sandbox(body: MeshSandboxGateway, node: Node, db: Db, cfg: Cfg) -> Ok:
"""The client certificate of the calling compute node's sandbox gateway, which Lens keeps sealed and reaches that
gateway over the mesh with. 403 for a node of another role, 400 for what isn't a certificate and key."""
try:
mesh.sandbox_gateway(db, cfg, node, body.model_dump())
except PermissionError as e:
raise HTTPException(403, str(e)) from None
except ValueError as e:
raise HTTPException(400, str(e)) from None
return Ok()


@router.post("/node/beat")
def node_beat(body: MeshBeat, node: Node, db: Db) -> Ok:
"""The calling node's report, every minute: Nebula, its peers, its machine."""
"""The calling node's report, every minute: Nebula, its peers, its machine, its sandbox gateway."""
mesh.beat(db, node, body.model_dump(exclude_none=True))
if node["role"] == "compute":
mesh.offer_sandboxes(db) # this process may have started since the node gave its gateway's certificate
return Ok()
Loading
Loading