Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ These are open pull requests, not yet part of Lens:
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
- **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates
for ([#553](https://github.com/adeelahmad/lense/pull/553)).
for ([#591](https://github.com/adeelahmad/lense/pull/591)).

## Planned

Expand Down
5 changes: 5 additions & 0 deletions changelog.d/mesh-node.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Joining a node** (in progress). Another machine joins with the install line and `LENS_JOIN` and `LENS_JOIN_TOKEN`:
it asks nothing, publishes no port, and runs Nebula and `lens node`, which joins once and then follows the primary's
config, renews its certificate and reports over the mesh. Adding a node token now returns that install line. Worker
nodes running jobs and the Settings pages follow in their own pull requests
([On a node](docs/mesh.md#on-a-node)).
61 changes: 61 additions & 0 deletions docker-compose.node.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# A machine on another Lens install's mesh (docs/mesh.md, Nodes): what install.sh starts with LENS_JOIN and
# LENS_JOIN_TOKEN. No database, no web app, nothing published: `mesh` runs Nebula, and `node` (`lens node`) joins once,
# then follows the primary's config over the mesh and reports every minute.
# LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… docker compose -p lens-node -f docker-compose.node.yml up -d --build
name: lens-node

services:
mesh:
image: python:3.12-slim # the plain image: the script is the repo's, Nebula is fetched with its checksum checked
command: ["python", "-I", "/lens-mesh/sidecar.py"]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
cap_add: [NET_ADMIN]
devices: ["/dev/net/tun:/dev/net/tun"]
read_only: true
tmpfs: [/tmp]
mem_limit: 256m
pids_limit: 256
environment:
LENS_MESH_DIR: /mesh
LENS_MESH_STATE: /state
volumes:
- ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro
- mesh-data:/mesh
- mesh-state:/state
restart: unless-stopped

node:
image: lens-backend:${LENS_VERSION:-prod}
build:
context: fastapi_backend
target: ${LENS_TARGET:-lean}
command: ["lens", "node"]
# the mesh container's network: what it reaches over Nebula, this reaches too, and nothing listens outside it
network_mode: service:mesh
depends_on: [mesh]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
read_only: true
tmpfs: [/tmp]
mem_limit: 512m
healthcheck:
disable: true # the image's check is the API's, which a node doesn't run
environment:
LENS_MESH_DIR: /mesh
LENS_NODE_DIR: /data/node
LENS_JOIN: ${LENS_JOIN:-}
LENS_JOIN_TOKEN: ${LENS_JOIN_TOKEN:-} # spent on the first start; never written to .env
LENS_NODE_NAME: ${LENS_NODE_NAME:-}
# a unit of an auto scaling group: its name, and the steps its worker runs (docs/scaling.md)
LENS_SCALING_UNIT: ${LENS_SCALING_UNIT:-}
LENS_WORKER_STEPS: ${LENS_WORKER_STEPS:-}
volumes:
- mesh-data:/mesh
- node-data:/data
restart: unless-stopped

volumes:
mesh-data:
mesh-state:
node-data:
38 changes: 33 additions & 5 deletions docs/mesh.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the
token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API
under `/api/v1/mesh`, and Nebula running on the primary in the `mesh` container, relaying the database and the API.
Coming in their own pull requests: `lens node` and the install line that joins another machine, Settings › Network and
Nodes in the web app, and running code on compute nodes.
under `/api/v1/mesh`, Nebula running on the primary in the `mesh` container, relaying the database and the API, and
`lens node` with the install line that joins another machine. Coming in their own pull requests: worker nodes running
jobs, Settings › Network and Nodes in the web app, and running code on compute nodes.

Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code
on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a
Expand Down Expand Up @@ -38,7 +38,10 @@ default), and withdrawn before that. The machine joins at this install's address
curl -fsSL https://raw.githubusercontent.com/adeelahmad/lense/main/install.sh | LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… sh
```

*(the install line's node path lands in its own pull request; the API it calls is here)*
`LENS_JOIN` is this install's address, the web app's (the one Settings › Nodes shows); with it the install line makes
the machine a node (below) instead of installing Lens. `LENS_NODE_NAME` names it (default: the machine's own name, or
for a unit an [auto scaling group](scaling.md) starts, `LENS_SCALING_UNIT`, which is kept with `LENS_WORKER_STEPS` for
the node's worker). `POST /api/v1/mesh/tokens` answers with the whole line (`install_line`).

The token never crosses the wire. The machine makes its own Nebula key pair (`nebula-cert keygen`; the private key
never leaves it) and sends `POST /api/v1/mesh/join` with the token's id, a random nonce, its name, its public key and
Expand All @@ -63,7 +66,30 @@ Over the mesh, with its node token (`Authorization: Bearer lensnode1.…`):
| `POST /api/v1/mesh/node/renew` | a new certificate for its public key (the same or a new one), with the groups its row has now. A node renews at a third of `mesh.cert_days` before its certificate runs out |
| `POST /api/v1/mesh/node/beat` | its report, every minute: whether Nebula runs, its peers, its machine, the sandboxes it can run, Nebula's last lines. Shown on the Nodes page |

A removed node is refused on all three.
A removed node is refused on all three. Nodes make these calls at the primary's mesh address (`10.73.0.1:8080` by
default), which isn't a host name the web app is served at: the API admits it for these paths alone, while the mesh
is on, and each call still needs its node token.

## On a node

The install line with `LENS_JOIN` makes the machine a node, in `~/lens-node` (`LENS_DIR` for another folder), with
Docker, which it installs as for Lens. It asks nothing, publishes no port and starts two containers
(`docker-compose.node.yml`, in the smaller `lean` image by default):

- **`mesh`**, the same container as on the primary (`mesh/sidecar.py`): it makes the node's key pair and runs Nebula
with the config it's given. Nothing listens on the node but Nebula, on a port it picks.
- **`node`** runs `lens node`, in the `mesh` container's network, with the folder the two share. It needs no database
and no settings. It turns the mesh on in that folder, so the container makes its key pair; joins with the token (as
above); keeps its node token and agent key in its own volume (`node.json`, readable only by it); and writes the
config for the container. Then, every minute, over the mesh only: it fetches its config (a removed machine's
certificate on the blocklist reaches it within the minute), renews its certificate when a third of its life is
left or the container's key changed, and sends its report.

The install line waits until the node has joined and says its mesh address. A token that was used, expired or
withdrawn isn't tried again: the line says why, and is run again with a new token. The token is in the first start's
environment only, never in `.env`, and isn't needed again: running the line again updates the node, which stays the
same node. A node that's removed keeps asking, so an undo in Settings › Nodes brings it back; `docker compose -f
docker-compose.node.yml exec node lens node status` says what it is on the mesh, without its credentials.

## On this machine

Expand Down Expand Up @@ -158,6 +184,8 @@ names an address that forwards to it). Nebula's own isolation is upstream's.

## Refine later

- The upgrade agent on nodes: today a node is updated by running its install line again.

- A second transport for networks that block UDP.
- Rotating the CA.
- Compute nodes running the assistant's sandboxes through the mesh, and their load shown on the Nodes page.
3 changes: 2 additions & 1 deletion fastapi_backend/app/api/v1/routes/mesh.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,8 @@ def make_mesh_token(body: MeshTokenCreate, user: AdminWriter, request: Request,
settings.save(db, request.app.state.archive.base, "mesh", {"enabled": True}, user.email)
auth.audit(db, user.as_audit(), "settings.save", "mesh", ["enabled"])
auth.audit(db, user.as_audit(), "mesh.token.create", f"mesh_token:{tok['id']}", {"role": body.role, "uses": body.uses})
return MeshTokenCreated(token=MeshToken(**tok), secret=secret, join_url=web_origin(request))
url = web_origin(request)
return MeshTokenCreated(token=MeshToken(**tok), secret=secret, join_url=url, install_line=mesh.join_line(cfg, secret, url))


@router.delete("/tokens/{token_id}")
Expand Down
4 changes: 4 additions & 0 deletions fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,10 @@ def main(argv=None):
config, i = argv[i].split("=", 1)[1], i + 1
else:
break
if i < len(argv) and argv[i] == "node": # a machine on another install's mesh: no database, no settings
from .domain import mesh_node

return mesh_node.main(argv[i + 1 :])
if i < len(argv) and argv[i] in PLATFORM_CMDS:
return platform_main(argv[i:], config)
if i < len(argv) and argv[i] == "ext": # extensions' manifests as files (cli_ext.py)
Expand Down
15 changes: 13 additions & 2 deletions fastapi_backend/app/core/middleware.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,15 +59,26 @@ def host_name(host: str) -> str:
return (host.split("]")[0] + "]" if host.startswith("[") else host.rsplit(":", 1)[0]).lower()


def mesh_call(request: Request, cfg: dict[str, Any]) -> bool:
"""A node's own call over the mesh (docs/mesh.md, After joining): addressed to the primary's mesh address, which
only the mesh reaches, on the paths nodes use, while the mesh is on. Each still carries its node token."""
from app.domain import mesh

if not request.url.path.startswith(mesh.NODE_PATHS) or not mesh.conf(cfg)["enabled"]:
return False
return host_name(request.headers.get("host", "")) == mesh.primary_ip(cfg)


def install(app: FastAPI) -> None:
@app.middleware("http")
async def guard(request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
# The Host header must be one we serve: this stops DNS-rebinding attacks against a server on localhost.
conf = getattr(request.app.state, "settings", None)
if conf is not None:
server = conf.current()["server"]
current = conf.current()
server = current["server"]
allowed = [h.lower() for h in server["allowed_hosts"]]
if "*" not in allowed and host_name(request.headers.get("host", "")) not in allowed:
if "*" not in allowed and host_name(request.headers.get("host", "")) not in allowed and not mesh_call(request, current):
return PlainTextResponse("Invalid host header", status_code=400)
frames = " ".join(server["embed_frame_ancestors"]) or "'none'"
else:
Expand Down
15 changes: 15 additions & 0 deletions fastapi_backend/app/domain/mesh.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@
# node's agent serves the primary only
DB_PORT, API_PORT, AGENT_PORT = 8000, 8080, 7443
INTERFACE = "lens-mesh"
# what nodes call on the primary, at its mesh address (core/middleware.py lets that address in for these alone)
NODE_PATHS = ("/api/v1/mesh/node/",)
TOKEN_PREFIX = "lensjoin1."
NODE_PREFIX = "lensnode1."
TOKEN_HOURS = (1, 24 * 30)
Expand Down Expand Up @@ -562,6 +564,19 @@ def make_token(db, cfg, by=None, role="worker", name=None, namespaces=None, hour
return _token_view(row), f"{TOKEN_PREFIX}{tid}.{secret}"


def join_line(cfg, token, url=None):
"""The install line another machine joins with (docs/mesh.md, Joining): install.sh with this install's address
(the web app's: `url`, else FRONTEND_URL) and the token. Shown once, with the token."""
from app.config import settings as env

repo = cfg["updates"].get("repo") or "adeelahmad/lense"
url = (url or env.FRONTEND_URL).rstrip("/")
if not re.fullmatch(r"https?://[A-Za-z0-9.\-\[\]:]+(/[A-Za-z0-9._~/-]*)?", url):
raise ValueError(f"{url!r} isn't an address a machine can join at")
parse_token(token)
return f"curl -fsSL https://raw.githubusercontent.com/{repo}/main/install.sh | LENS_JOIN={url} LENS_JOIN_TOKEN={token} sh"


def revoke_token(db, tid):
if not db.one("SELECT id FROM $r", r=R("mesh_token", int(tid))):
raise KeyError(tid)
Expand Down
Loading
Loading