From 94de5bd9f1418deadc22b87808c0abcc03fc3282 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 03:22:00 +0000 Subject: [PATCH 1/6] feat(mesh): run Nebula on the primary in an optional mesh container The primary's half of the mesh: Lens's server can't create network devices, so Nebula runs in a container of its own, `mesh` (mesh/sidecar.py in the plain Python image), the only one holding NET_ADMIN and /dev/net/tun. It makes this machine's key pair itself and keeps the private key in its own volume; it puts the public key in the folder it shares with Lens's server, and the keeper (mesh.Keeper, one process at a time on a lease) signs it and writes the config back. The container checks each config with `nebula -test`, reloads Nebula on SIGHUP when it can (the blocklist, the firewall) and restarts it when it must, and relays port 8000 on the mesh address to the database and 8080 to the API, for the machines Nebula's firewall lets in. It fetches Nebula's release with the same pinned checksums as Lens. Adding a node asks the upgrade agent to start the container (profile `mesh`), and turning the mesh off stops it; GET /api/v1/mesh shows how it runs under `service`, and the primary's row carries its report. Default installs are unchanged: the profile is off and nothing listens. Tested with a stand-in Nebula in CI, and with Nebula 1.11.2 in Docker: a worker node reached the database and the API through the container, a compute node the API only, and a removed node nothing. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- CHANGELOG.md | 6 +- docker-compose.prod.yml | 36 ++ docker-compose.yml | 35 ++ docs/assistant.md | 5 +- docs/mesh.md | 43 +- docs/updates.md | 6 +- fastapi_backend/app/cli.py | 2 + fastapi_backend/app/core/runtime.py | 2 + fastapi_backend/app/domain/mesh.py | 196 +++++++- fastapi_backend/app/domain/services.py | 11 +- fastapi_backend/app/schemas/mesh.py | 24 +- .../tests/api/test_mesh_service.py | 229 ++++++++++ fastapi_backend/tests/api/test_services.py | 1 + mesh/project.json | 27 ++ mesh/sidecar.py | 421 ++++++++++++++++++ nextjs-frontend/app/openapi-client/index.ts | 2 + .../app/openapi-client/types.gen.ts | 102 ++++- nextjs-frontend/openapi.json | 147 +++++- packaging/host/lens-upgrade.sh | 1 + packaging/host/test-services.sh | 11 +- threat-model.json | 14 +- threat-model.md | 3 +- threat-model.yaml | 11 +- 23 files changed, 1310 insertions(+), 25 deletions(-) create mode 100644 fastapi_backend/tests/api/test_mesh_service.py create mode 100644 mesh/project.json create mode 100644 mesh/sidecar.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7824212ad..3d0804c21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,8 +79,10 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version a key derived from it) and gets an address, a certificate naming its role and namespaces, and its Nebula config, whose firewall lets worker nodes reach the database and every node the API, and nothing else; a node renews, reports and refreshes its config with its own node token, and removing one blocklists its certificates on every machine. - Off until a node is added; settings in `mesh`. Nebula running on the primary, the install line that joins a machine - and the Settings pages follow in their own pull requests ([Nodes and the mesh](docs/mesh.md)). + Off until a node is added; settings in `mesh`. On the primary Nebula runs in the optional `mesh` container, which the + upgrade agent starts when a node is added: it makes its own key pair, checks Nebula's download against the kept + checksums and relays the database and the API on the mesh address only. The install line that joins a machine and + the Settings pages follow in their own pull requests ([Nodes and the mesh](docs/mesh.md)). - **Export a conversation.** `GET /api/v1/chats//export.md`, `.pdf` or `.json` gives the version showing, one message (`?message=`) or, as JSON, every version (`?all=true`). The Markdown has each question and answer, its citations as footnotes linking back to the moment in Lens and the tools it used; the PDF is that Markdown printed by diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index ecdc2480b..79fec64d8 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -74,6 +74,8 @@ x-backend: &backend LENS_GRAFANA_URL: ${LENS_GRAFANA_URL:-} # the volume shared with the matterbridge service (its config, written from Settings → Chat rooms) LENS_MATTERBRIDGE_DIR: /matterbridge + # the volume shared with the mesh service (its config, written while the mesh is on; docs/mesh.md) + LENS_MESH_DIR: /mesh # the MCP runner (the optional `mcp` service, docs/assistant.md): MCP servers that run as a program, sandboxed there LENS_MCP_RUNNER_URL: http://mcp:8090 LENS_MCP_TOKEN_FILE: /mcp-key/token @@ -96,6 +98,7 @@ x-backend: &backend - ./docs:/docs:ro - ${AUDIO_DIR:-./local-audio}:/audio:ro - matterbridge-data:/matterbridge + - mesh-data:/mesh - mcp-key:/mcp-key:ro - sandbox-key:/sandbox-key:ro depends_on: @@ -449,9 +452,42 @@ services: restart: unless-stopped networks: [lens] + # The mesh (docs/mesh.md): Nebula, the private network between Lens's machines, on this machine (mesh/sidecar.py). + # Lens's server writes its config into the volume the two share; it makes this machine's key pair itself and holds + # NET_ADMIN for the mesh's network device, which nothing else in Lens does. It relays the database and the API to + # the machines the mesh's firewall lets in. Started by the upgrade agent when an admin adds a node (Settings › + # Nodes), or `docker compose --profile mesh up -d`. Other machines reach it on UDP 4242 (LENS_MESH_PORT). + mesh: + image: python:3.12-slim # the plain image: the script is the repo's, Nebula is fetched with its checksum checked + profiles: [mesh] + command: ["python", "-I", "/lens-mesh/sidecar.py"] + security_opt: ["no-new-privileges:true"] + cap_drop: [ALL] + cap_add: [NET_ADMIN] + devices: ["/dev/net/tun:/dev/net/tun"] + read_only: true + tmpfs: [/tmp] + mem_limit: 256m + pids_limit: 256 + environment: + LENS_MESH_DIR: /mesh + LENS_MESH_STATE: /state + LENS_MESH_LISTEN_PORT: "4242" + LENS_MESH_RELAYS: 8000=surrealdb:8000,8080=backend:8000 + ports: + - "${LENS_MESH_BIND:-0.0.0.0}:${LENS_MESH_PORT:-4242}:4242/udp" + volumes: + - ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro + - mesh-data:/mesh + - mesh-state:/state + restart: unless-stopped + networks: [lens] + volumes: surreal-data: matterbridge-data: + mesh-data: + mesh-state: mcp-data: mcp-key: sandbox-key: diff --git a/docker-compose.yml b/docker-compose.yml index 7f1ed18f6..c6328116c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -46,6 +46,8 @@ x-backend: &backend OPENAPI_OUTPUT_FILE: ./shared-data/openapi.json # the volume shared with the matterbridge service (its config, written from Settings → Chat rooms) LENS_MATTERBRIDGE_DIR: /matterbridge + # the volume shared with the mesh service (its config, written while the mesh is on; docs/mesh.md) + LENS_MESH_DIR: /mesh # the MCP runner (the optional `mcp` service, docs/assistant.md): MCP servers that run as a program, sandboxed there LENS_MCP_RUNNER_URL: http://mcp:8090 LENS_MCP_TOKEN_FILE: /mcp-key/token @@ -65,6 +67,7 @@ x-backend: &backend - ./docs:/docs:ro - ${AUDIO_DIR:-./local-audio}:/audio:ro - matterbridge-data:/matterbridge + - mesh-data:/mesh - mcp-key:/mcp-key:ro - sandbox-key:/sandbox-key:ro depends_on: @@ -337,9 +340,41 @@ services: - fedora-data:/fcrepo-home networks: [lens] + # The mesh (docs/mesh.md): Nebula, the private network between Lens's machines, on this machine (mesh/sidecar.py). + # Lens's server writes its config into the volume the two share; it makes this machine's key pair itself and holds + # NET_ADMIN for the mesh's network device, which nothing else in Lens does. It relays the database and the API to + # the machines the mesh's firewall lets in. Started by the upgrade agent when an admin adds a node (Settings › + # Nodes), or `docker compose --profile mesh up -d`. Other machines reach it on UDP 4242 (LENS_MESH_PORT). + mesh: + image: python:3.12-slim # the plain image: the script is the repo's, Nebula is fetched with its checksum checked + profiles: [mesh] + command: ["python", "-I", "/lens-mesh/sidecar.py"] + security_opt: ["no-new-privileges:true"] + cap_drop: [ALL] + cap_add: [NET_ADMIN] + devices: ["/dev/net/tun:/dev/net/tun"] + read_only: true + tmpfs: [/tmp] + mem_limit: 256m + pids_limit: 256 + environment: + LENS_MESH_DIR: /mesh + LENS_MESH_STATE: /state + LENS_MESH_LISTEN_PORT: "4242" + LENS_MESH_RELAYS: 8000=surrealdb:8000,8080=backend:8000 + ports: + - "${LENS_MESH_BIND:-0.0.0.0}:${LENS_MESH_PORT:-4242}:4242/udp" + volumes: + - ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro + - mesh-data:/mesh + - mesh-state:/state + networks: [lens] + volumes: surreal-data: matterbridge-data: + mesh-data: + mesh-state: mcp-data: mcp-key: sandbox-key: diff --git a/docs/assistant.md b/docs/assistant.md index 52e986195..73ad8ac5d 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -857,8 +857,9 @@ machine** under **Ready-made** in **Add MCP server**, then **Add Browser**, whic ### Optional containers -The MCP runner, the browser, the [code sandbox](#the-gateway)'s gateway and monitoring -([Prometheus and Grafana](prometheus.md)) are containers Lens can run next to it, off until an admin turns one on. The web app can't start containers itself, so the machine's upgrade agent does +The MCP runner, the browser, the [code sandbox](#the-gateway)'s gateway, monitoring +([Prometheus and Grafana](prometheus.md)) and the mesh ([Nodes and the mesh](mesh.md#on-this-machine), started when an +admin adds a node) are containers Lens can run next to it, off until an admin turns one on. The web app can't start containers itself, so the machine's upgrade agent does ([Updates](updates.md)): it adds the profile to `COMPOSE_PROFILES` in Lens's `.env`, or takes it out, and starts or stops that profile's containers, one change at a time. Without an agent (Lens wasn't installed with `install.sh`), the switch shows the line to run on the machine instead. Agents use the same switches: `GET /api/v1/admin/services` and diff --git a/docs/mesh.md b/docs/mesh.md index 68f9afd19..75887bd16 100644 --- a/docs/mesh.md +++ b/docs/mesh.md @@ -1,10 +1,10 @@ # Nodes and the mesh Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the -token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings and the API -under `/api/v1/mesh`. Coming in their own pull requests: Nebula running on the primary and relaying the database, -`lens node` and the install line that joins another machine, Settings › Network and Nodes in the web app, and running -code on compute nodes. +token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API +under `/api/v1/mesh`, and Nebula running on the primary in the `mesh` container, relaying the database and the API. +Coming in their own pull requests: `lens node` and the install line that joins another machine, Settings › Network and +Nodes in the web app, and running code on compute nodes. Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a @@ -65,6 +65,37 @@ Over the mesh, with its node token (`Authorization: Bearer lensnode1.…`): A removed node is refused on all three. +## On this machine + +Lens's server can't create network devices, and nothing in Lens but Nebula needs to, so on the primary Nebula runs in +a container of its own, `mesh` (`mesh/sidecar.py`, in the plain Python image). It's an optional service like the +browser or monitoring: adding a node asks the machine's upgrade agent to start it, and turning the mesh off +(`mesh.enabled`) stops it. Without an upgrade agent, start it with the line Settings › Network shows: + +```sh +docker compose -f docker-compose.prod.yml --profile mesh up -d +``` + +- **Its own key.** The container makes this machine's Nebula key pair the first time and keeps the private key in its + own volume. It shares a folder with Lens's server (`LENS_MESH_DIR`, `/mesh`): it puts the public key there, Lens + signs it and writes the config back (`lens.json`), and the container adds the private key and runs Nebula. Lens + signs it again when a third of its life is left, or when the container's key changes. +- **Checked before it runs.** It fetches Nebula's release for the machine on first use, checked against the same + checksums Lens keeps, and runs each new config past `nebula -test` first. A change Nebula can take while running + (a removed node's certificate on the blocklist, the firewall) is reloaded without dropping tunnels; a new network + device or port restarts it. +- **Relays.** It listens on this machine's mesh address only, and passes port 8000 to the database and port 8080 to + the API: what a worker node and every node reach here, as Nebula's firewall allows (below). +- **The port.** Other machines reach it on UDP 4242, which Docker publishes (`LENS_MESH_PORT` in `.env` publishes + another; `mesh.port` is the port nodes are told to use, so change both, or set `mesh.hosts` to `host:port` when a + router forwards another port to it). +- **What it says.** It writes how it runs (`status.json`): Nebula on or not, since when, the relays and Nebula's last + lines. `GET /api/v1/mesh` shows it under `service`, and the primary's own row in `/api/v1/mesh/nodes` carries it as + a node's report does. + +It holds `NET_ADMIN` and `/dev/net/tun` and nothing else: no other capability, a read-only image, 256 MB of memory, +none of Lens's data or keys but its own key pair and the config. + ## The config Each machine's config is rendered by Lens, from the settings and the node's row; the node adds only its private key. @@ -91,7 +122,7 @@ Everything is in **Settings › Network** (`mesh`), editable in the app with def | --- | --- | --- | | `enabled` | off | on when a node is added; off stops the mesh | | `network` | `10.73.0.0/16` | the mesh's addresses, a private IPv4 range from /8 to /28; the primary is its first. It can't change once machines have addresses in it | -| `port` | `4242` | the UDP port the primary listens on; nodes must reach it | +| `port` | `4242` | the UDP port nodes reach the primary at: the one Docker publishes (`LENS_MESH_PORT`), unless a router forwards another | | `hosts` | none | where nodes reach that port from outside the mesh (`host` or `host:port`, up to 8). Empty: the address each node joined this install at | | `cipher` | `aes` | or `chachapoly` (faster on machines without AES hardware, such as some Raspberry Pis); the same on every machine | | `cert_days` | 30 | how long a machine's certificate lasts; it renews at two thirds | @@ -107,6 +138,8 @@ removes it (undo with `POST /nodes/{id}/restore` while its certificate is good). ## What keeps it safe - Nothing listens until an admin adds a node, and single-machine installs never do. +- On the primary only the `mesh` container holds `NET_ADMIN`; it makes its own key pair, runs only Nebula's release with + the checksum kept in Lens, and its relays listen on the mesh address alone. - Only a machine with a token an admin made joins; the token is spent on use, expires, and never crosses the wire. - A machine's certificate names only its own address, role and namespaces; it can't widen them by renewing. - Nebula's firewall, rendered by Lens, opens only what each role needs. Lens itself never trusts the mesh's address: diff --git a/docs/updates.md b/docs/updates.md index dd65857d9..72eebbfeb 100644 --- a/docs/updates.md +++ b/docs/updates.md @@ -65,9 +65,9 @@ The same agent turns Lens's [optional containers](assistant.md#optional-containe `action=services`, a `profile` and `turn=on` or `off` (in the same inbox file, the one every agent watches) adds or removes that profile in `COMPOSE_PROFILES` in `.env` and starts or stops its containers, and nothing else. The profiles are fixed in the agent: `mcp` (the MCP runner), `browser` (the browser and its proxy), `sandbox` (the -[code sandbox](assistant.md#the-gateway)'s gateway, its certificates and the client's key) and `monitoring` -([Charts and history](prometheus.md#prometheus-and-grafana-ready-made): Prometheus, node-exporter and Grafana); it -refuses any other. It reports in `services-status` and `services.log`. A request from an older Lens, with +[code sandbox](assistant.md#the-gateway)'s gateway, its certificates and the client's key), `monitoring` +([Charts and history](prometheus.md#prometheus-and-grafana-ready-made): Prometheus, node-exporter and Grafana) and +`mesh` ([Nodes and the mesh](mesh.md#on-this-machine): Nebula on the primary); it refuses any other. It reports in `services-status` and `services.log`. A request from an older Lens, with `monitoring=on` or `off` and no profile, still works. With no agent (Lens wasn't installed with install.sh, the machine has neither systemd nor launchd, or install.sh ran diff --git a/fastapi_backend/app/cli.py b/fastapi_backend/app/cli.py index ef5ad9456..1c06fd024 100644 --- a/fastapi_backend/app/cli.py +++ b/fastapi_backend/app/cli.py @@ -279,6 +279,7 @@ def platform_main(argv, config): jobs, local_llm, logs, + mesh, nfs_share, notify, passkeys, @@ -405,6 +406,7 @@ def password(): shares.start(db, C, stop, log=say, name=wk.name) # and the file shares (S3, WebDAV), while they're on channels.start_all(db, C, stop, lambda: cfg, name=wk.name, log_fn=say) # and the chat channels, while they're on tunnel.start(db, C, stop, name=wk.name, log=say) # and the Cloudflare tunnel, while it's on + mesh.start(db, C, stop, name=wk.name, log=say) # and the mesh container's config, while the mesh is on https.start(db, C, stop, name=wk.name, log=say) # and HTTPS, where the ports are (LENS_HTTPS_LISTEN) local_llm.start(db, C, stop, name=wk.name, log=say) # and the local chat model, while it's on nfs_share.start(db, C, stop, name=wk.name, log=say) # and the file tree over NFS, while it's on diff --git a/fastapi_backend/app/core/runtime.py b/fastapi_backend/app/core/runtime.py index e67a5668e..fcd15cf27 100644 --- a/fastapi_backend/app/core/runtime.py +++ b/fastapi_backend/app/core/runtime.py @@ -20,6 +20,7 @@ https, jobs, local_llm, + mesh, notify, routines, sensors, @@ -100,6 +101,7 @@ def start_background(self) -> None: sensors.start(self.db, self.current, self.stop, log=log.info, name=f"api-{os.getpid()}") channels.start_all(self.db, self.current, self.stop, lambda: self.base, name=f"api-{os.getpid()}", log_fn=log.warning) tunnel.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info) + mesh.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info) https.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info) local_llm.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info) updates.start(self.db, self.current, self.stop, log=log.info) diff --git a/fastapi_backend/app/domain/mesh.py b/fastapi_backend/app/domain/mesh.py index dcc30b26f..ca1ded1dc 100644 --- a/fastapi_backend/app/domain/mesh.py +++ b/fastapi_backend/app/domain/mesh.py @@ -41,6 +41,8 @@ import subprocess import tarfile import tempfile +import threading +import time import zipfile import httpx @@ -424,7 +426,8 @@ def primary(db, cfg, public_key=None): if public_key: crt, info = _sign(db, cfg, "primary", row["ip"], row["groups"], public_key) _record_cert(db, row["id"], info, crt) - return {**_view(row), "crt": crt, "fingerprint": info["fingerprint"], "not_after": info["not_after"]} + db.q("UPDATE $r SET public_key = $p", r=R("mesh_node", row["id"]), p=public_key) + return {**_view(row), "crt": crt, "fingerprint": info["fingerprint"], "not_after": info["not_after"], "public_key": public_key} return _view(row) @@ -776,8 +779,196 @@ def undo_remove(db, nid): return node(db, nid) +# ---------- Nebula on this machine: the `mesh` container ---------- +# Lens's server can't hold NET_ADMIN, so the primary's Nebula runs in the optional `mesh` container (mesh/sidecar.py), +# which shares a folder with it (LENS_MESH_DIR): Lens writes lens.json there (on or off, the config, the public key it +# signed), the container writes its public key (host.pub) and how it runs (status.json). The keeper below does +# Lens's half, in one process at a time (a lease on app_service:mesh_keeper), and asks the machine's upgrade agent to +# start the container when the mesh is turned on and stop it when it's turned off (services.py, profile `mesh`). +SERVICE = "mesh" +KEEPER_ROW = R("app_service", "mesh_keeper") +KEEPER_SECONDS, LEASE_SECONDS, BEAT_SECONDS = 10, 45, 60 +STATUS_STALE = 30 + + +def shared_dir(): + """The folder shared with the `mesh` container, or None where there is none (not Docker).""" + d = os.environ.get("LENS_MESH_DIR") + return pathlib.Path(d) if d else None + + +def _write(path, text): + tmp = path.with_name(f".{path.name}.{os.getpid()}.tmp") + tmp.write_text(text) + tmp.chmod(0o644) + tmp.replace(path) + + +def service(cfg=None): + """How the `mesh` container says it runs (status.json): {} when there's no container or it hasn't said; + running False when what it said is old (it stopped).""" + d = shared_dir() + if not d: + return {} + try: + s = json.loads((d / "status.json").read_text()) + except (OSError, ValueError): + return {} + if not isinstance(s, dict): + return {} + try: + age = (_now() - dt.datetime.fromisoformat(str(s.get("at")).replace("Z", "+00:00"))).total_seconds() + except ValueError: + age = STATUS_STALE + 1 + s["fresh"] = age <= STATUS_STALE + if not s["fresh"]: + s["nebula"] = False + return s + + +def _parse_time(value): + try: + return dt.datetime.fromisoformat(str(value).replace("Z", "+00:00")) + except ValueError: + return None + + +def lens_file(db, cfg, public_key=None): + """What goes in lens.json now: off, on and waiting for the container's key, or on with the primary's config + (its certificate signed again when the key is new or a third of its life is left).""" + c = conf(cfg) + if not c["enabled"]: + return {"on": False} + row = primary(db, cfg) + out = {"on": True, "ip": row["ip"], "public_key": None, "config": None, "version": None} + if not public_key: + return out + have = db.one("SELECT crt, public_key, not_after FROM $r", r=R("mesh_node", row["id"])) or {} + ends = _parse_time(have.get("not_after")) + if not have.get("crt") or have.get("public_key") != public_key or not ends or ends - _now() < dt.timedelta(days=renew_days(cfg)): + have = primary(db, cfg, public_key=public_key) + config = render(db, cfg, row, have["crt"]) + return {**out, "public_key": public_key, "config": config, "version": version(config)} + + +class Keeper: + """Keeps lens.json as the mesh is, the primary's certificate signed, and the container started or stopped.""" + + def __init__(self, db, cfg_fn, name, log=None): + self.db, self.cfg_fn, self.name, self.log = db, cfg_fn, name, log or (lambda *a: None) + self.last_beat = 0.0 + self.error = None + + def _lease(self): + now = _now() + return bool( + self.db.rows( + "UPSERT $r MERGE {owner: $me, at: $t} WHERE owner = NONE OR owner = $me OR at < $stale RETURN AFTER", + r=KEEPER_ROW, + me=self.name, + t=_ts(now), + stale=_ts(now - dt.timedelta(seconds=LEASE_SECONDS)), + ) + ) + + def _profile(self, on): + """Ask the upgrade agent to start (or stop) the container, once for each turn of mesh.enabled, so an admin + who stops it in Settings › Services isn't overruled.""" + from . import services + + row = self.db.one("SELECT asked_for FROM $r", r=KEEPER_ROW) or {} + if row.get("asked_for") is on or (row.get("asked_for") is None and not on): + return + try: + services.ask(self.db, SERVICE, on, by="lens") + except services.ServiceError as e: + if "upgrade agent" not in str(e): + return # busy: next tick + self.log("mesh: %s", e) + self.db.q("UPDATE $r SET asked_for = $o", r=KEEPER_ROW, o=on) + + def tick(self): + d = shared_dir() + if not d or not self._lease(): + return + cfg = self.cfg_fn() + on = conf(cfg)["enabled"] + self._profile(on) + pub = None + try: + text = (d / "host.pub").read_text() + pub = text if PUBLIC_KEY_RX.fullmatch(text.replace("\r\n", "\n")) else None + except OSError: + pass + want = lens_file(self.db, cfg, pub if on else None) + text = json.dumps(want, indent=1, sort_keys=True) + try: + same = (d / "lens.json").read_text() == text + except OSError: + same = False + if not same: + _write(d / "lens.json", text) + self.log("mesh: wrote the primary's config (%s)", "on" if on else "off") + if on and time.monotonic() - self.last_beat > BEAT_SECONDS: + s = service(cfg) + row = primary(self.db, cfg) + beat( + self.db, + row, + {"nebula": bool(s.get("nebula")), "version": s.get("version"), "error": s.get("error"), "log": s.get("log") or []}, + ) + self.last_beat = time.monotonic() + + def loop(self, stop): + while not stop.is_set(): + try: + self.tick() + self.error = None + except Exception as e: # noqa: BLE001 - keep the keeper alive; the error shows in the status + self.error = f"{type(e).__name__}: {e}" + try: + self.db.q("UPDATE $r SET error = $e WHERE owner = $me", r=KEEPER_ROW, e=self.error, me=self.name) + except Exception: # noqa: BLE001 + pass + stop.wait(KEEPER_SECONDS) + try: + self.db.q("UPDATE $r SET owner = NONE WHERE owner = $me", r=KEEPER_ROW, me=self.name) + except Exception: # noqa: BLE001 + pass + + +def start(db, cfg_fn, stop, name, log=None): + """The keeper, where there is a `mesh` container to keep (LENS_MESH_DIR); None elsewhere.""" + if not shared_dir(): + return None + k = Keeper(db, cfg_fn, name, log) + threading.Thread(target=k.loop, args=(stop,), daemon=True, name="mesh").start() + return k + + +def service_view(db, cfg): + """The `mesh` container as Settings › Network shows it.""" + from . import services + + s = service(cfg) + keeper = db.one("SELECT error FROM $r", r=KEEPER_ROW) or {} + return { + "available": bool(shared_dir()), + "state": s.get("state") or ("off" if s else None), + "nebula": bool(s.get("nebula")), + "since": s.get("since"), + "restarts": s.get("restarts") or 0, + "relays": s.get("relays") or [], + "error": s.get("error") or keeper.get("error"), + "log": (s.get("log") or [])[-40:], + "seen_at": s.get("at"), + "manual": services.manual(SERVICE), + } + + def status(db, cfg): - """What Settings › Network shows: on or off, the network, the CA, the primary's address, the counts.""" + """What Settings › Network shows: on or off, the network, the CA, the primary's address, the counts, and how + Nebula runs on this machine.""" c, row = conf(cfg), ca_row(db) every = nodes(db) return { @@ -792,4 +983,5 @@ def status(db, cfg): "tokens": len(tokens(db)), "nebula": NEBULA_VERSION, "tools": bool(binaries(cfg)), + "service": service_view(db, cfg), } diff --git a/fastapi_backend/app/domain/services.py b/fastapi_backend/app/domain/services.py index 535a3a767..121108a30 100644 --- a/fastapi_backend/app/domain/services.py +++ b/fastapi_backend/app/domain/services.py @@ -1,5 +1,5 @@ -"""Lens's optional containers, turned on and off from the web app: the MCP runner, the browser and monitoring -(docs/assistant.md, docs/prometheus.md). +"""Lens's optional containers, turned on and off from the web app: the MCP runner, the browser, monitoring and the +mesh (docs/assistant.md, docs/prometheus.md, docs/mesh.md). The containers can't start other containers, so the machine's upgrade agent does it, as it installs releases (updates.py, packaging/host/lens-upgrade.sh): a request with action=services, profile= and turn=on|off in its @@ -40,6 +40,13 @@ "Prometheus, node-exporter and Grafana: charts of Lens and this machine over time", None, ), + # started and stopped by the mesh's keeper as mesh.enabled turns (mesh.py), not only from here + "mesh": ( + "Mesh", + "Nebula, the private network between Lens's machines (Settings › Nodes): the only container holding NET_ADMIN, " + "listening on UDP 4242", + None, + ), } ACTIVE = ("queued", "running") FINAL = ("done", "failed") diff --git a/fastapi_backend/app/schemas/mesh.py b/fastapi_backend/app/schemas/mesh.py index b620d17d4..e553c5e2e 100644 --- a/fastapi_backend/app/schemas/mesh.py +++ b/fastapi_backend/app/schemas/mesh.py @@ -18,10 +18,31 @@ class MeshAuthority(ResponseModel): created_at: str | None = None +class MeshRelay(ResponseModel): + port: int = Field(description="the port on this machine's mesh address") + to: str = Field(description="the container it passes connections to") + listening: bool + open: int = Field(0, description="connections through it now") + error: str | None = None + + +class MeshService(ResponseModel): + available: bool = Field(description="this install runs the primary's Nebula in the `mesh` container (Docker installs)") + state: str | None = Field(None, description="off, waiting (for its key to be signed), starting, running or error") + nebula: bool = Field(False, description="Nebula runs on this machine now") + since: str | None = Field(None, description="when Nebula last started") + restarts: int = 0 + relays: list[MeshRelay] = Field(default_factory=list, description="what the mesh reaches on this machine") + error: str | None = None + log: list[str] = Field(default_factory=list, description="Nebula's last lines") + seen_at: str | None = Field(None, description="when the container last said how it runs") + manual: str = Field(description="the line that starts the container on this machine, where there's no upgrade agent") + + class MeshStatus(ResponseModel): enabled: bool = Field(description="the mesh is on (mesh.enabled): adding a node turns it on") network: str = Field(description="the mesh's addresses (mesh.network)") - port: int = Field(description="the UDP port the primary listens on (mesh.port)") + port: int = Field(description="the UDP port other machines reach the primary at (mesh.port)") hosts: list[str] = Field(description="where other machines reach that port (mesh.hosts); empty: where each joined") primary_ip: str = Field(description="this machine's address on the mesh") ca: MeshAuthority | None = Field(None, description="the mesh's certificate authority, once made") @@ -30,6 +51,7 @@ class MeshStatus(ResponseModel): tokens: int = Field(description="join tokens that can still be used") nebula: str = Field(description="the Nebula version Lens runs") tools: bool = Field(description="Nebula's programs are on this server (else they're fetched when first needed)") + service: MeshService = Field(description="Nebula on this machine, in the `mesh` container") class MeshNode(ResponseModel): diff --git a/fastapi_backend/tests/api/test_mesh_service.py b/fastapi_backend/tests/api/test_mesh_service.py new file mode 100644 index 000000000..8cad09de8 --- /dev/null +++ b/fastapi_backend/tests/api/test_mesh_service.py @@ -0,0 +1,229 @@ +"""Nebula on the primary (docs/mesh.md, "On this machine"): the optional `mesh` container (mesh/sidecar.py) runs it +with the config Lens's keeper writes into the folder the two share. The container makes its own key pair and +publishes the public key; the keeper signs it and writes the config; the container checks it with `nebula -test`, +runs Nebula, reloads it when the config changes (a removed node's certificate on the blocklist) and relays the +database and the API on the mesh address. Nebula is a stand-in here; the same flow was run with Nebula 1.11.2 in +Docker: a worker node reached the database and the API through the container, a compute node only the API, and a +removed node nothing.""" + +from __future__ import annotations + +import hashlib +import importlib.util +import io +import json +import pathlib +import socket +import sys +import tarfile +import threading +import time + +import pytest + +from app.domain import mesh, services, settings +from tests.api.test_mesh import admin, joined, nebula, server_db # noqa: F401 - fixtures + +SIDECAR = pathlib.Path(__file__).resolve().parents[3] / "mesh" / "sidecar.py" + + +def load_sidecar(): + spec = importlib.util.spec_from_file_location("lens_mesh_sidecar", SIDECAR) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +@pytest.fixture +def sidecar(): + return load_sidecar() + + +@pytest.fixture +def stub_nebula(nebula): # noqa: F811 - the fixture from test_mesh.py + """A nebula that passes -test, then runs until stopped, saying when it's reloaded (SIGHUP).""" + (nebula / "nebula").write_text( + f"#!{sys.executable}\n" + "import json, signal, sys, time\n" + "if '-test' in sys.argv:\n" + " cfg = json.load(open(sys.argv[sys.argv.index('-config') + 1]))\n" + " sys.exit(0 if cfg['pki'].get('key') and cfg['pki'].get('cert') else 1)\n" + "signal.signal(signal.SIGHUP, lambda *_: print(json.dumps({'msg': 'reloaded'}), flush=True))\n" + "print(json.dumps({'msg': 'Nebula interface is active'}), flush=True)\n" + "while True:\n" + " time.sleep(0.1)\n" + ) + return nebula + + +def echo_server(): + s = socket.create_server(("127.0.0.1", 0)) + + def serve(): + while True: + try: + c, _ = s.accept() + except OSError: + return + with c: + c.sendall(b"db:" + c.recv(100)) + + threading.Thread(target=serve, daemon=True).start() + return s + + +def free_port(): + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +def test_the_sidecar_pins_the_nebula_lens_does(sidecar): + assert sidecar.NEBULA_VERSION == mesh.NEBULA_VERSION + assert sidecar.SHA256 == {k: v for k, v in mesh.SHA256.items() if k.startswith("nebula-linux-")} + assert sidecar.release_name("aarch64") == "nebula-linux-arm64.tar.gz" + assert sidecar.relays("8000=surrealdb:8000, 8080=backend:8000") == [(8000, ("surrealdb", 8000)), (8080, ("backend", 8000))] + with pytest.raises(ValueError): + sidecar.relays("8000=surrealdb") + + +def test_the_sidecar_downloads_nebula_only_with_its_checksum(sidecar, tmp_path, monkeypatch): + buf = io.BytesIO() + with tarfile.open(fileobj=buf, mode="w:gz") as t: + for name in ("nebula", "nebula-cert"): + data = b"#!/bin/sh\necho " + name.encode() + b"\n" + info = tarfile.TarInfo(name) + info.size = len(data) + t.addfile(info, io.BytesIO(data)) + blob = buf.getvalue() + monkeypatch.delenv("LENS_NEBULA_DIR", raising=False) + monkeypatch.setattr(sidecar.platform, "machine", lambda: "x86_64") + + def opener(url, timeout): + assert url == sidecar.RELEASES + "nebula-linux-amd64.tar.gz" + return io.BytesIO(blob) + + with pytest.raises(RuntimeError, match="checksum"): + sidecar.programs(tmp_path, opener=opener) + assert not any((tmp_path / "bin").rglob("nebula*")) + monkeypatch.setitem(sidecar.SHA256, "nebula-linux-amd64.tar.gz", hashlib.sha256(blob).hexdigest()) + got = sidecar.programs(tmp_path, opener=opener) + assert pathlib.Path(got["nebula"]).read_bytes().endswith(b"echo nebula\n") + assert sidecar.programs(tmp_path, fetch=False) == got # kept: not fetched again + + +def test_the_keeper_and_the_sidecar_bring_up_the_primary(client, db, cfg, server_db, stub_nebula, sidecar, tmp_path, monkeypatch): # noqa: F811 + shared, state = tmp_path / "shared", tmp_path / "state" + shared.mkdir() + monkeypatch.setenv("LENS_MESH_DIR", str(shared)) + target = echo_server() + port = free_port() + svc = sidecar.Service(shared, state, f"{port}=127.0.0.1:{target.getsockname()[1]}", listen_port="4242") + h = admin(client, db) + current = lambda: settings.effective(db, cfg) # noqa: E731 + keeper = mesh.Keeper(db, current, "test") + + # off: nothing written but "off"; the container runs nothing + keeper.tick() + assert json.loads((shared / "lens.json").read_text()) == {"on": False} + svc.tick() + assert svc.status()["state"] == "off" and not (shared / "host.pub").exists() + + # adding a node turns the mesh on: the keeper says so, and asks the machine to start the container + asked = [] + monkeypatch.setattr(services, "ask", lambda db_, profile, on, by=None: asked.append((profile, on))) + token = client.post("/api/v1/mesh/tokens", json={"role": "worker"}, headers=h).json()["secret"] + keeper.tick() + assert asked == [("mesh", True)] + lens = json.loads((shared / "lens.json").read_text()) + assert lens["on"] is True and lens["ip"] == "10.73.0.1" and lens["config"] is None # waiting for the key + svc.tick() # makes its key pair; publishes the public key + assert svc.status()["state"] == "waiting" + pub = (shared / "host.pub").read_text() + assert "PUBLIC KEY" in pub and "PRIVATE" not in pub + assert (state / "host.key").stat().st_mode & 0o077 == 0 # the private key stays in the container's own volume + + keeper.tick() # signs it + lens = json.loads((shared / "lens.json").read_text()) + assert lens["public_key"] == pub and lens["config"]["pki"]["cert"] and "key" not in lens["config"]["pki"] + primary = next(n for n in client.get("/api/v1/mesh/nodes", headers=h).json() if n["role"] == "primary") + assert primary["fingerprint"] and primary["not_after"] + keeper.tick() + assert json.loads((shared / "lens.json").read_text())["config"]["pki"]["cert"] == lens["config"]["pki"]["cert"] # not signed again + + # the container runs Nebula with its own key added, and relays the mesh address's ports (127.0.0.1 here) + lens["ip"] = "127.0.0.1" + (shared / "lens.json").write_text(json.dumps(lens)) + svc.tick() + s = svc.status() + assert s["nebula"] is True and s["state"] == "running", s + run = json.loads((state / "nebula.json").read_text()) + assert run["pki"]["key"] == str(state / "host.key") and run["listen"]["port"] == 4242 + with socket.create_connection(("127.0.0.1", port), timeout=5) as c: + c.sendall(b"hello") + assert c.recv(100) == b"db:hello" + + # a node removed: the blocklist grows, the keeper writes it, and Nebula reloads it without a restart + nid, _, _ = joined(client, cfg, token) + pid = s["pid"] + assert client.delete(f"/api/v1/mesh/nodes/{nid}", headers=h).status_code == 200 + keeper.tick() + lens2 = json.loads((shared / "lens.json").read_text()) + assert lens2["config"]["pki"]["blocklist"] and lens2["version"] != lens["version"] + lens2["ip"] = "127.0.0.1" + (shared / "lens.json").write_text(json.dumps(lens2)) + svc.tick() + for _ in range(50): + if any("reloaded" in line for line in svc.status()["log"]): + break + time.sleep(0.05) + assert svc.status()["pid"] == pid and any("reloaded" in line for line in svc.status()["log"]) + + # what Settings › Network shows about it + svc.save() + got = client.get("/api/v1/mesh", headers=h).json()["service"] + assert got["available"] is True and got["nebula"] is True and got["state"] == "running" + assert got["relays"][0]["listening"] is True and got["manual"].endswith("--profile mesh up -d") + + # turned off: the keeper says so and asks the machine to stop the container; Nebula and the relays stop + assert client.put("/api/v1/settings/mesh", json={"enabled": False}, headers=h).status_code == 200 + keeper.tick() + assert asked[-1] == ("mesh", False) + assert json.loads((shared / "lens.json").read_text()) == {"on": False} + svc.tick() + assert svc.status()["nebula"] is False and svc.status()["state"] == "off" + with pytest.raises(OSError): + socket.create_connection(("127.0.0.1", port), timeout=1) + svc.stop() + target.close() + + +def test_a_stopped_nebula_is_started_again_after_a_while(db, cfg, server_db, stub_nebula, sidecar, tmp_path, monkeypatch): # noqa: F811 + shared, state = tmp_path / "shared", tmp_path / "state" + shared.mkdir() + monkeypatch.setenv("LENS_MESH_DIR", str(shared)) + cfg = {**cfg, "mesh": {"enabled": True}} + monkeypatch.setattr(services, "ask", lambda *a, **k: None) + keeper = mesh.Keeper(db, lambda: cfg, "test") + svc = sidecar.Service(shared, state, "") + keeper.tick() + svc.tick() + keeper.tick() + svc.tick() + first = svc.status()["pid"] + assert first + svc.proc.kill() + svc.proc.wait() + svc.tick() + s = svc.status() + assert s["state"] == "error" and "Nebula stopped" in s["error"] and s["restarts"] == 1 + svc.next_try = 0 + svc.tick() + assert svc.status()["nebula"] is True and svc.status()["pid"] != first + svc.stop() + + +def test_no_mesh_folder_no_keeper(db, cfg, monkeypatch): + monkeypatch.delenv("LENS_MESH_DIR", raising=False) + assert mesh.start(db, lambda: cfg, threading.Event(), "test") is None + assert mesh.service_view(db, cfg)["available"] is False diff --git a/fastapi_backend/tests/api/test_services.py b/fastapi_backend/tests/api/test_services.py index 4e5a7788c..5e75856e3 100644 --- a/fastapi_backend/tests/api/test_services.py +++ b/fastapi_backend/tests/api/test_services.py @@ -47,6 +47,7 @@ def test_each_profile_is_turned_on_and_off_on_its_own(client, db, control): ("mcp", False, None, "systemd"), ("browser", False, None, "systemd"), ("monitoring", False, None, "systemd"), + ("mesh", False, None, "systemd"), # started by adding a node (tests/api/test_mesh_service.py) ] assert "--profile browser up -d" in got[1]["manual"] diff --git a/mesh/project.json b/mesh/project.json new file mode 100644 index 000000000..20e3f5560 --- /dev/null +++ b/mesh/project.json @@ -0,0 +1,27 @@ +{ + "$schema": "../node_modules/nx/schemas/project-schema.json", + "name": "mesh", + "projectType": "application", + "tags": ["lang:python"], + "targets": { + "lint": { + "executor": "nx:run-commands", + "options": { + "cwd": "{projectRoot}", + "commands": [ + "uvx ruff check --config ../fastapi_backend/pyproject.toml sidecar.py", + "uvx ruff format --check --config ../fastapi_backend/pyproject.toml sidecar.py" + ], + "parallel": false + } + }, + "test": { + "executor": "nx:run-commands", + "inputs": ["default", "{workspaceRoot}/fastapi_backend/tests/api/test_mesh_service.py"], + "options": { + "cwd": "{workspaceRoot}/fastapi_backend", + "command": "uv run pytest tests/api/test_mesh_service.py -q" + } + } + } +} diff --git a/mesh/sidecar.py b/mesh/sidecar.py new file mode 100644 index 000000000..65317b0b8 --- /dev/null +++ b/mesh/sidecar.py @@ -0,0 +1,421 @@ +#!/usr/bin/env python3 +"""Nebula for Lens: the optional `mesh` container (docs/mesh.md), the only part of Lens that holds NET_ADMIN. + +It runs Nebula with the config Lens's server writes, and relays the ports the mesh serves on this machine to the +containers behind them. It makes this machine's Nebula key pair itself, so the private key never leaves it: Lens +signs the public key and sends back the certificate inside the config. + +Files in the folder it shares with Lens's server (LENS_MESH_DIR, /mesh): + +* `lens.json`, from Lens: {"on", "ip", "public_key", "version", "config"}: the Nebula config without the private key, + and the public key Lens signed it for (none yet: Lens is waiting for this machine's). No file, or "on" false: the + mesh is off, and Nebula stops. +* `host.pub`, from here: this machine's public key, for Lens to sign. +* `status.json`, from here: whether Nebula runs, the relays, Nebula's last lines. + +In its own volume (LENS_MESH_STATE, /state): Nebula's programs, fetched on first use from Nebula's release with their +checksum checked against the ones kept here (the same as fastapi_backend/app/domain/mesh.py's), the private key and +the config Nebula runs. + +LENS_MESH_RELAYS says what to relay: "8000=surrealdb:8000,8080=backend:8000" listens on this machine's mesh address, +port 8000 and 8080, and passes each connection to the database and the API. Nebula's firewall, in the config Lens +writes, decides which machines reach each one. LENS_MESH_LISTEN_PORT is the UDP port Nebula listens on in here (4242; +Docker publishes it). + +Python's standard library only, so it runs in the plain python image. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import platform +import shutil +import signal +import socket +import subprocess +import sys +import tarfile +import threading +import time +import urllib.request + +NEBULA_VERSION = "1.11.2" +RELEASES = f"https://github.com/slackhq/nebula/releases/download/v{NEBULA_VERSION}/" +# Nebula's own SHASUM256.txt for these downloads (kept equal to mesh.py's by tests/api/test_mesh_service.py) +SHA256 = { + "nebula-linux-amd64.tar.gz": "6140d33f2ec21ce7f6b655b5bc820e93a684d97e51d0ddcf907324b5b28aac1e", + "nebula-linux-arm64.tar.gz": "85d10e7bc2d121193c1392a1a919172ded7c413f46e602138281cfa9fa1b0231", + "nebula-linux-arm-7.tar.gz": "dc9c890d7ec014ab709e0444592b33f35db4f396f28245c95887844b1565ff41", + "nebula-linux-arm-6.tar.gz": "d8f86452620eb6ab76bdf9ed9841a309ba8122ac2980c2b7c7e2d1aa8d160e84", +} +LOG_LINES = 60 +TICK = 2.0 +RETRY = (5, 15, 30, 60, 120) +# a change to these needs Nebula started again; the rest it takes on SIGHUP +RESTART_KEYS = ("tun", "listen", "cipher") + + +def say(*parts): + print("lens-mesh:", *parts, flush=True) + + +# ---------- Nebula's programs ---------- +def release_name(machine=None): + arch = {"x86_64": "amd64", "amd64": "amd64", "aarch64": "arm64", "arm64": "arm64", "armv7l": "arm-7", "armv6l": "arm-6"}.get( + (machine or platform.machine()).lower() + ) + if not arch: + raise RuntimeError(f"Nebula has no Linux release for {machine or platform.machine()}") + return f"nebula-linux-{arch}.tar.gz" + + +def programs(state, fetch=True, opener=None): + """{"nebula": path, "nebula-cert": path}: LENS_NEBULA_DIR's, or the release fetched into the state folder.""" + given = os.environ.get("LENS_NEBULA_DIR") + if given: + return {p: os.path.join(given, p) for p in ("nebula", "nebula-cert")} + out = pathlib.Path(state) / "bin" / NEBULA_VERSION + found = {p: str(out / p) for p in ("nebula", "nebula-cert")} + if all(os.access(p, os.X_OK) for p in found.values()): + return found + if not fetch: + return None + name = release_name() + out.mkdir(parents=True, exist_ok=True) + part = out / (name + ".part") + say(f"downloading Nebula {NEBULA_VERSION} ({name})") + h = hashlib.sha256() + with (opener or urllib.request.urlopen)(RELEASES + name, timeout=300) as r, open(part, "wb") as f: + while chunk := r.read(1 << 16): + h.update(chunk) + f.write(chunk) + try: + if h.hexdigest() != SHA256[name]: + raise RuntimeError("the Nebula download doesn't match its checksum") + with tarfile.open(part) as t: + for prog in ("nebula", "nebula-cert"): + src = t.extractfile(t.getmember(prog)) + if src is None: + raise RuntimeError(f"the Nebula download has no {prog}") + with src, open(out / f"{prog}.new", "wb") as f: + shutil.copyfileobj(src, f) + (out / f"{prog}.new").chmod(0o755) + (out / f"{prog}.new").replace(out / prog) + finally: + part.unlink(missing_ok=True) + return found + + +def key_pair(state, cert_tool): + """This machine's key pair, made once: (private key path, public key text).""" + key, pub = pathlib.Path(state) / "host.key", pathlib.Path(state) / "host.pub" + if not (key.exists() and pub.exists()): + for p in (key, pub): + p.unlink(missing_ok=True) + old = os.umask(0o077) + try: + subprocess.run([cert_tool, "keygen", "-out-key", str(key), "-out-pub", str(pub)], check=True, capture_output=True, timeout=60) + finally: + os.umask(old) + key.chmod(0o600) + return str(key), pub.read_text() + + +def write(path, text, mode=0o644): + """Write a file whole, so a reader never sees half of it.""" + path = pathlib.Path(path) + tmp = path.with_name(f".{path.name}.tmp") + tmp.write_text(text) + tmp.chmod(mode) + tmp.replace(path) + + +# ---------- the config ---------- +def read_lens(shared): + """lens.json while the mesh is on, else None (off, or Lens hasn't written it yet).""" + try: + d = json.loads((pathlib.Path(shared) / "lens.json").read_text()) + except (OSError, ValueError): + return None + return d if isinstance(d, dict) and d.get("on") else None + + +def nebula_config(lens, key_path, listen_port=None): + """The config Nebula runs: Lens's, with this machine's private key, and the port Nebula listens on in here.""" + c = json.loads(json.dumps(lens["config"])) + c.setdefault("pki", {})["key"] = key_path + if listen_port: + c.setdefault("listen", {})["port"] = int(listen_port) + return c + + +def needs_restart(old, new): + return any((old or {}).get(k) != new.get(k) for k in RESTART_KEYS) + + +def relays(spec): + """ "8000=surrealdb:8000,8080=backend:8000" as [(8000, ("surrealdb", 8000)), ...].""" + out = [] + for part in (spec or "").split(","): + part = part.strip() + if not part: + continue + port, _, to = part.partition("=") + host, _, to_port = to.rpartition(":") + if not (port.isdigit() and to_port.isdigit() and host): + raise ValueError(f"LENS_MESH_RELAYS: {part!r} isn't port=host:port") + out.append((int(port), (host, int(to_port)))) + return out + + +# ---------- relays ---------- +class Relay: + """Listens on the mesh address and passes each connection on to `to`; the mesh's firewall decides who connects.""" + + def __init__(self, port, to): + self.port, self.to = port, to + self.ip = None + self.sock: socket.socket | None = None + self.error = None + self.open = 0 + + def view(self): + return { + "port": self.port, + "to": f"{self.to[0]}:{self.to[1]}", + "listening": self.sock is not None, + "open": self.open, + "error": self.error, + } + + def ensure(self, ip): + """Listen on ip:port (the mesh address appears once Nebula is up, so this is tried each tick).""" + if self.sock and self.ip == ip: + return + self.close() + try: + s = socket.create_server((ip, self.port), reuse_port=False) + except OSError as e: + self.error = f"can't listen on {ip}:{self.port} yet: {e.strerror or e}" + return + self.ip, self.sock, self.error = ip, s, None + threading.Thread(target=self._accept, args=(s,), daemon=True, name=f"relay-{self.port}").start() + + def close(self): + s, self.sock = self.sock, None + if s: + try: # wakes the accept() blocked on it, which close() alone doesn't on Linux + s.shutdown(socket.SHUT_RDWR) + except OSError: + pass + s.close() + + def _accept(self, s): + while self.sock is s: + try: + conn, _ = s.accept() + except OSError: + return + threading.Thread(target=self._serve, args=(conn,), daemon=True).start() + + def _serve(self, conn): + try: + up = socket.create_connection(self.to, timeout=10) + except OSError as e: + self.error = f"{self.to[0]}:{self.to[1]} didn't answer: {e.strerror or e}" + conn.close() + return + up.settimeout(None) + self.open += 1 + done = threading.Event() + + def pump(a, b): + try: + while chunk := a.recv(1 << 16): + b.sendall(chunk) + except OSError: + pass + finally: + done.set() + + threading.Thread(target=pump, args=(up, conn), daemon=True).start() + pump(conn, up) + done.wait(1) + for x in (conn, up): + try: + x.close() + except OSError: + pass + self.open -= 1 + + +# ---------- the service ---------- +class Service: + def __init__(self, shared, state, relay_spec="", listen_port=None, fetch=True): + self.shared, self.state = pathlib.Path(shared), pathlib.Path(state) + self.relays = [Relay(p, to) for p, to in relays(relay_spec)] + self.listen_port = listen_port + self.fetch = fetch + self.proc: subprocess.Popen | None = None + self.running_config = None + self.lines: list[str] = [] + self.error = None + self.state_word = "off" + self.since = None + self.restarts = 0 + self.fails = 0 + self.next_try = 0.0 + self.config_version = None + self.lock = threading.Lock() + + # what Lens reads + def status(self): + with self.lock: + lines = list(self.lines[-LOG_LINES:]) + return { + "at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "state": self.state_word, + "nebula": self.proc is not None and self.proc.poll() is None, + "pid": self.proc.pid if self.proc and self.proc.poll() is None else None, + "since": self.since, + "restarts": self.restarts, + "version": NEBULA_VERSION, + "config_version": self.config_version, + "relays": [r.view() for r in self.relays], + "error": self.error, + "log": lines, + } + + def save(self): + write(self.shared / "status.json", json.dumps(self.status(), indent=1)) + + def _read(self, proc): + for raw in proc.stdout: + line = raw.rstrip() + if line: + print(line, flush=True) # docker logs shows it too + with self.lock: + self.lines = (self.lines + [line[:400]])[-LOG_LINES:] + + def stop(self): + p, self.proc = self.proc, None + for r in self.relays: + r.close() + if p and p.poll() is None: + p.terminate() + try: + p.wait(10) + except subprocess.TimeoutExpired: + p.kill() + self.running_config = None + + def _start(self, exe, path, config): + self.proc = subprocess.Popen([exe, "-config", str(path)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1) + threading.Thread(target=self._read, args=(self.proc,), daemon=True, name="nebula-log").start() + self.running_config = config + self.since = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + say(f"started Nebula (pid {self.proc.pid})") + + def _failed(self, msg): + self.error = msg + self.state_word = "error" + self.fails += 1 + self.next_try = time.monotonic() + RETRY[min(self.fails - 1, len(RETRY) - 1)] + + def tick(self): + self.state.mkdir(parents=True, exist_ok=True) + lens = read_lens(self.shared) + if not lens: + if self.proc: + say("the mesh is off: stopping Nebula") + self.stop() + self.state_word, self.error, self.config_version = "off", None, None + return + if time.monotonic() < self.next_try: + return + try: + exe = programs(self.state, fetch=self.fetch) + if not exe: + return self._failed("Nebula isn't here yet") + key, pub = key_pair(self.state, exe["nebula-cert"]) + mine = self.shared / "host.pub" + if not mine.exists() or mine.read_text() != pub: + write(mine, pub) + except Exception as e: # noqa: BLE001 - says why in status.json, tries again later + return self._failed(f"{type(e).__name__}: {e}") + if not isinstance(lens.get("config"), dict) or not lens.get("ip") or (lens.get("public_key") or "").strip() != pub.strip(): + self.stop() + self.state_word, self.error = "waiting", "waiting for Lens to sign this machine's key" + return + config = nebula_config(lens, key, self.listen_port) + path = self.state / "nebula.json" + alive = self.proc is not None and self.proc.poll() is None + if self.proc and not alive: # it stopped by itself: say why, and start it again after a while + code = self.proc.returncode + with self.lock: + last = self.lines[-1] if self.lines else "no output" + self.restarts += 1 + self.stop() + return self._failed(f"Nebula stopped (exit {code}): {last}") + if config != self.running_config or not alive: + nxt = self.state / "nebula.next.json" + write(nxt, json.dumps(config, indent=1), 0o600) + test = subprocess.run([exe["nebula"], "-test", "-config", str(nxt)], capture_output=True, text=True, timeout=60) + if test.returncode != 0: + return self._failed("Nebula refused the config: " + (test.stderr or test.stdout).strip()[-300:]) + nxt.replace(path) + if alive and not needs_restart(self.running_config, config): + self.proc.send_signal(signal.SIGHUP) + self.running_config = config + say("the config changed: Nebula reloaded it") + else: + self.stop() + self._start(exe["nebula"], path, config) + self.config_version = lens.get("version") + for r in self.relays: + r.ensure(lens["ip"]) + waiting = [r for r in self.relays if not r.sock] + self.state_word = "starting" if waiting else "running" + self.error = waiting[0].error if waiting else None + if not waiting: + self.fails = 0 + + def loop(self, stop): + while not stop.is_set(): + try: + self.tick() + except Exception as e: # noqa: BLE001 - keep running; the error is in status.json + self._failed(f"{type(e).__name__}: {e}") + try: + self.save() + except OSError as e: + say(f"can't write status.json: {e}") + stop.wait(TICK) + self.stop() + self.state_word = "off" + try: + self.save() + except OSError: + pass + + +def main(): + shared = pathlib.Path(os.environ.get("LENS_MESH_DIR") or "/mesh") + state = pathlib.Path(os.environ.get("LENS_MESH_STATE") or "/state") + shared.mkdir(parents=True, exist_ok=True) + shared.chmod(0o1777) # Lens's server runs as another user: it writes lens.json here and reads the rest + os.umask(0o022) + svc = Service(shared, state, os.environ.get("LENS_MESH_RELAYS", ""), os.environ.get("LENS_MESH_LISTEN_PORT") or None) + stop = threading.Event() + for sig in (signal.SIGTERM, signal.SIGINT): + signal.signal(sig, lambda *_: stop.set()) + say(f"waiting for Lens to write {shared / 'lens.json'} (it does once a node is added)") + svc.loop(stop) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index eb6bd463e..96beb1c16 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -2311,7 +2311,9 @@ export type { MeshNodesData, MeshNodesResponse, MeshNodesResponses, + MeshRelay, MeshRenew, + MeshService, MeshStatus, MeshStatusData, MeshStatusResponse, diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 91c73b405..1259a9559 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -12414,6 +12414,39 @@ export type MeshNodeConfig = { [key: string]: unknown; }; +/** + * MeshRelay + */ +export type MeshRelay = { + /** + * Port + * + * the port on this machine's mesh address + */ + port: number; + /** + * To + * + * the container it passes connections to + */ + to: string; + /** + * Listening + */ + listening: boolean; + /** + * Open + * + * connections through it now + */ + open?: number; + /** + * Error + */ + error?: string | null; + [key: string]: unknown; +}; + /** * MeshRenew */ @@ -12426,6 +12459,69 @@ export type MeshRenew = { public_key: string; }; +/** + * MeshService + */ +export type MeshService = { + /** + * Available + * + * this install runs the primary's Nebula in the `mesh` container (Docker installs) + */ + available: boolean; + /** + * State + * + * off, waiting (for its key to be signed), starting, running or error + */ + state?: string | null; + /** + * Nebula + * + * Nebula runs on this machine now + */ + nebula?: boolean; + /** + * Since + * + * when Nebula last started + */ + since?: string | null; + /** + * Restarts + */ + restarts?: number; + /** + * Relays + * + * what the mesh reaches on this machine + */ + relays?: Array; + /** + * Error + */ + error?: string | null; + /** + * Log + * + * Nebula's last lines + */ + log?: Array; + /** + * Seen At + * + * when the container last said how it runs + */ + seen_at?: string | null; + /** + * Manual + * + * the line that starts the container on this machine, where there's no upgrade agent + */ + manual: string; + [key: string]: unknown; +}; + /** * MeshStatus */ @@ -12445,7 +12541,7 @@ export type MeshStatus = { /** * Port * - * the UDP port the primary listens on (mesh.port) + * the UDP port other machines reach the primary at (mesh.port) */ port: number; /** @@ -12494,6 +12590,10 @@ export type MeshStatus = { * Nebula's programs are on this server (else they're fetched when first needed) */ tools: boolean; + /** + * Nebula on this machine, in the `mesh` container + */ + service: MeshService; [key: string]: unknown; }; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 21e0aae43..c50678525 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -54393,6 +54393,49 @@ ], "title": "MeshNodeConfig" }, + "MeshRelay": { + "properties": { + "port": { + "type": "integer", + "title": "Port", + "description": "the port on this machine's mesh address" + }, + "to": { + "type": "string", + "title": "To", + "description": "the container it passes connections to" + }, + "listening": { + "type": "boolean", + "title": "Listening" + }, + "open": { + "type": "integer", + "title": "Open", + "description": "connections through it now", + "default": 0 + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "port", + "to", + "listening" + ], + "title": "MeshRelay" + }, "MeshRenew": { "properties": { "public_key": { @@ -54409,6 +54452,101 @@ ], "title": "MeshRenew" }, + "MeshService": { + "properties": { + "available": { + "type": "boolean", + "title": "Available", + "description": "this install runs the primary's Nebula in the `mesh` container (Docker installs)" + }, + "state": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "State", + "description": "off, waiting (for its key to be signed), starting, running or error" + }, + "nebula": { + "type": "boolean", + "title": "Nebula", + "description": "Nebula runs on this machine now", + "default": false + }, + "since": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Since", + "description": "when Nebula last started" + }, + "restarts": { + "type": "integer", + "title": "Restarts", + "default": 0 + }, + "relays": { + "items": { + "$ref": "#/components/schemas/MeshRelay" + }, + "type": "array", + "title": "Relays", + "description": "what the mesh reaches on this machine" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + }, + "log": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Log", + "description": "Nebula's last lines" + }, + "seen_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Seen At", + "description": "when the container last said how it runs" + }, + "manual": { + "type": "string", + "title": "Manual", + "description": "the line that starts the container on this machine, where there's no upgrade agent" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "available", + "manual" + ], + "title": "MeshService" + }, "MeshStatus": { "properties": { "enabled": { @@ -54424,7 +54562,7 @@ "port": { "type": "integer", "title": "Port", - "description": "the UDP port the primary listens on (mesh.port)" + "description": "the UDP port other machines reach the primary at (mesh.port)" }, "hosts": { "items": { @@ -54474,6 +54612,10 @@ "type": "boolean", "title": "Tools", "description": "Nebula's programs are on this server (else they're fetched when first needed)" + }, + "service": { + "$ref": "#/components/schemas/MeshService", + "description": "Nebula on this machine, in the `mesh` container" } }, "additionalProperties": true, @@ -54488,7 +54630,8 @@ "removed", "tokens", "nebula", - "tools" + "tools", + "service" ], "title": "MeshStatus" }, diff --git a/packaging/host/lens-upgrade.sh b/packaging/host/lens-upgrade.sh index 823ca7625..a8eae7f73 100755 --- a/packaging/host/lens-upgrade.sh +++ b/packaging/host/lens-upgrade.sh @@ -29,6 +29,7 @@ services_of() { mcp) echo "mcp" ;; browser) echo "browser browser-proxy" ;; sandbox) echo "openshell-init openshell-key openshell-gateway" ;; # the certificates, the client's key, the gateway + mesh) echo "mesh" ;; esac } diff --git a/packaging/host/test-services.sh b/packaging/host/test-services.sh index 81933e0ad..92e67b064 100644 --- a/packaging/host/test-services.sh +++ b/packaging/host/test-services.sh @@ -81,7 +81,16 @@ case "$(last)" in *"-f docker-compose.prod.yml -f docker-compose.https.yml --pro [ "$(profiles)" = matterbridge,monitoring ] || fail "COMPOSE_PROFILES is $(profiles)" ok "the HTTPS compose file comes along when HTTPS is on" -# 7. Anything else is refused and runs nothing: a profile it doesn't know, a turn that isn't on or off, a microsandbox +# 7. The mesh (Nebula on the primary, docs/mesh.md) starts its one container, and stops it +ask id=n1 action=services profile=mesh turn=on +[ "$(said state)" = "done" ] && [ "$(profiles)" = matterbridge,monitoring,mesh ] || fail "COMPOSE_PROFILES is $(profiles)" +case "$(last)" in *"--profile mesh up -d mesh") ;; *) fail "ran: $(last)" ;; esac +ask id=n2 action=services profile=mesh turn=off +[ "$(profiles)" = matterbridge,monitoring ] || fail "COMPOSE_PROFILES is $(profiles)" +case "$(last)" in *"--profile mesh rm -s -f mesh") ;; *) fail "ran: $(last)" ;; esac +ok "the mesh starts and stops its own container" + +# 8. Anything else is refused and runs nothing: a profile it doesn't know, a turn that isn't on or off, a microsandbox calls="$(wc -l <"$work/calls")" ask id=x1 action=services profile=matterbridge turn=on [ "$(said state)" = failed ] || fail "a profile it doesn't know: $(said state)" diff --git a/threat-model.json b/threat-model.json index c723bb66d..9dc7a4cb6 100644 --- a/threat-model.json +++ b/threat-model.json @@ -146,7 +146,8 @@ "SMB share (port 1445 on the host, 4445 in the worker)", "POST /api/v1/mesh/join (a machine joining the mesh)", "/api/v1/mesh/node/config, /node/renew, /node/beat (a node's own calls, with its node token)", - "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh" + "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh", + "UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the mesh container, while the mesh is on)" ], "touches": [ "network", @@ -770,6 +771,15 @@ "caller_must_enforce": "admin is trusted; audited (mesh.token.create, mesh.join, mesh.node.*); the network can't change once machines have addresses in it", "provenance": "documented", "source": "docs/mesh.md \"Settings\"" + }, + { + "entry_point": "UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the mesh container, while the mesh is on)", + "parameter": "Nebula handshakes and tunnelled packets from anyone reaching the port; on the mesh, TCP to the database and the API", + "attacker_controllable": "yes", + "component": "optional", + "caller_must_enforce": "Nebula drops what no certificate of the mesh's CA signs and anything on the blocklist; its firewall, rendered by Lens, lets only lens-worker reach 8000 and lens-node 8080; the relays listen on the mesh address only; the container is the only one holding NET_ADMIN, makes its own key pair (the private key stays in its own volume) and runs only Nebula's release with the checksum kept in Lens", + "provenance": "documented", + "source": "docs/mesh.md \"On this machine\"" } ], "environment": { @@ -1496,7 +1506,7 @@ "property": "P-MESH", "violation_symptom": "info-leak, integrity-bypass", "severity_tier": "security", - "conditions": "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does", + "conditions": "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only", "provenance": "documented", "source": "docs/mesh.md \"What keeps it safe\"" } diff --git a/threat-model.md b/threat-model.md index c85412f2b..462f3a51a 100644 --- a/threat-model.md +++ b/threat-model.md @@ -333,6 +333,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /api/v1/mesh/join` (a machine joining the mesh) | a join token's id, a nonce, the machine's name and Nebula public key, an HMAC of those keyed with the token's secret, facts about the machine | yes, anyone reaching the web app, once a token exists | x-credential, data | no sign-in: the token's secret never crosses the wire, the HMAC is checked in constant time, wrong proofs count towards a throttle per address (8 in 15 minutes), a token is spent on use (or after `uses`) and expires, and the answer (certificate, config, node token) is sealed with a key derived from the secret, so an eavesdropper on plain http:// learns nothing | *(documented, docs/mesh.md "Joining")* | | `/api/v1/mesh/node/config`, `/node/renew`, `/node/beat` (a node's own calls, with its node token) | a public key to certify; a status report: whether Nebula runs, its peers, its machine, its log lines | yes, a machine that joined (a removed one is refused) | x-credential, data, x-peer | the node token is 256 random bits kept only as a hash; a node gets its own config and a certificate for its own address and groups only, and can't widen them; reports are capped and kept as its status for admins | *(documented, docs/mesh.md "After joining")* | | `POST /api/v1/mesh/tokens`, `PATCH` and `DELETE /api/v1/mesh/nodes/{id}`, `PUT /api/v1/settings/mesh` | a node's role, name, namespaces, token hours and uses; the mesh's network, port, hosts, cipher | admin only | x-env-config | admin is trusted; audited (`mesh.token.create`, `mesh.join`, `mesh.node.*`); the network can't change once machines have addresses in it | *(documented, docs/mesh.md "Settings")* | +| UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the `mesh` container, while the mesh is on) | Nebula handshakes and tunnelled packets from anyone reaching the port; on the mesh, TCP to the database and the API | yes, anyone reaching UDP 4242; on the mesh, a machine with a certificate the mesh's CA signed | data, x-peer | Nebula drops what no certificate of the mesh's CA signs and anything on the blocklist; its firewall, rendered by Lens, lets only `lens-worker` reach 8000 and `lens-node` 8080; the relays listen on the mesh address only; the container is the only one holding NET_ADMIN, makes its own key pair (the private key stays in its own volume) and runs only Nebula's release with the checksum kept in Lens | *(documented, docs/mesh.md "On this machine")* | **Stateful postconditions.** A refused or failed one-time secret check consumes nothing; a used one is deleted before Lens trusts it *(documented, docs/authentication.md "works once")*. An upload file only ever holds whole chunks @@ -583,7 +584,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-BROWSER-GUARD | The Lens extension's side panel reads and acts only on sites the person allowed in it (and the browser granted), and opens an address only on such a site. It asks the person before a click, submit or Enter that the model marks consequential or whose control's words or posting form look so, and, once the answer has read text from outside Lens (`ai.untrusted_asks`), before opening or downloading an address or typing text the person's own message doesn't contain. It never types into password, card or file fields, never reports their values, and shows each action as it happens. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Letting it use your browser")* | Voided by: The consequential check is the model's flag plus a word list (`extensions/browser/src/core/act.js`): a control with no telling words that a misled model doesn't mark runs without asking; synthetic events reach only the page's own handlers. Removing a site from the options stops the side panel acting there, but leaves the browser's access to it for the clipper. Lens sends the ask (`confirm`) and the extension honours it: an extension older than this doesn't ask. | | P-SANDBOX | Code the assistant runs (`run_python`, `run_bash`; `sandbox.enabled`, off by default) runs only for people `sandbox.who` allows (editors and admins, or admins; never viewers), in a sandbox of that conversation's own on the OpenShell gateway, never in Lens's processes. The sandbox gets none of Lens's data, keys or database address: only the code and the files a run copies in, which must be attached to that conversation by that person (25 MB each, 100 MB a run). Its policy allows no egress with `sandbox.network: off`, only pypi.org, files.pythonhosted.org and registry.npmjs.org on 443 with `packages`, and those and the names in `sandbox.hosts` on 80 and 443 with `hosts`; every endpoint is held to public addresses (no private, loopback, link-local or reserved range), so a listed name that resolves to one is refused. A file leaves it only as one of the person's held uploads (25 MB) placed in a namespace by an `import_files` approval, or as a download (`share_sandbox_file`, 25 MB, 20 a conversation) kept encrypted with the conversation, served only to its person as an attachment with `nosniff`, and deleted with the conversation or their account. A sandbox stops after `sandbox.idle_minutes` idle and is deleted after `sandbox.keep_hours` unused or with its conversation. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Code sandbox")* | Voided by: isolation inside the sandbox is OpenShell's (§1.9); `landlock.compatibility: best_effort` runs without Landlock on a kernel that lacks it — `domain/sandbox.py`; whoever the gateway lets in can run sandboxes of their own (in the `sandbox` profile, a holder of its client certificate); a file shared to download is readable by whoever holds `secret.key` or `ARCHIVE_SECRET_KEY` (Q14); what the code prints reaches the model as third-party text (N-PROMPT). | | P-RESEARCH | A research plan (`domain/research.py`) runs only once the person who asked for it approves it: signed in, or with a "yes" in their chat room from someone in `bridge.users`; an API key, an app or an agent (MCP, the assistant) can only decline it, unless an admin turns on `research.agents_skip_review`. The run acts as that person with their access at run time and stops if their account is disabled. It is offered only the read tools of the sources its plan names (never a tool that changes anything), makes at most the plan's tool calls (`research.max_tool_calls`, 40 by default), stops looking things up after `research.max_minutes` and writes up what it found, and checks the budgets of its namespaces and `all` before each stage, waiting or stopping as they say. Its web searches and page reads, and code runs with the network when the plan names the code sandbox, don't ask again after outside text, since the plan approved them. A person has one run going at a time. One process carries a run out at a time (a claim on `research_run:` with a heartbeat every 10 seconds, taken over only after two minutes of silence). Every call it makes is in the activity ledger for the conversation and `research:`. | integrity-bypass | correctness-only | *(documented, docs/assistant.md "Deep research")* | Voided by: `research.agents_skip_review: true` lets an agent's plan run without review — `domain/research.py` `agents_may_approve`; a run's web queries can carry archive text it read, within its plan's tool calls (the misled model of N-PROMPT); a "yes" in a room is the room's people's (N-BRIDGE). | -| P-MESH | The mesh between Lens's machines (`mesh.enabled`, off until an admin adds a node) admits only machines that joined with a token an admin made, which is spent on use and expires; the token never crosses the wire (an HMAC proves it, the answer is sealed with a key derived from it), each machine's certificate names only its own address, role and namespaces, and Nebula's firewall, rendered by Lens, lets worker nodes reach the database port and every node the API port on the primary, and the primary a node's agent port, and nothing else. Removing a node puts every certificate it was given on the blocklist every machine's config carries, and its node token stops working at once. Nodes still sign in to Lens as any client does: the mesh is a transport, not an identity. | info-leak, integrity-bypass | security-critical | *(documented, docs/mesh.md "What keeps it safe")* | Voided by: the join request goes to whatever address is put in the install line, so a wrong address gives a stranger the token's id (not its secret); a worker node that reaches the database port holds the database credential (a trusted peer, Q9); the lighthouse must be reachable on UDP `mesh.port`; Nebula's own isolation is upstream's (§1.9) — `domain/mesh.py`. | +| P-MESH | The mesh between Lens's machines (`mesh.enabled`, off until an admin adds a node) admits only machines that joined with a token an admin made, which is spent on use and expires; the token never crosses the wire (an HMAC proves it, the answer is sealed with a key derived from it), each machine's certificate names only its own address, role and namespaces, and Nebula's firewall, rendered by Lens, lets worker nodes reach the database port and every node the API port on the primary, and the primary a node's agent port, and nothing else. Removing a node puts every certificate it was given on the blocklist every machine's config carries, and its node token stops working at once. Nodes still sign in to Lens as any client does: the mesh is a transport, not an identity. On the primary Nebula runs in the `mesh` container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the checksum kept in Lens; its relays listen on the mesh address only. | info-leak, integrity-bypass | security-critical | *(documented, docs/mesh.md "What keeps it safe")* | Voided by: the join request goes to whatever address is put in the install line, so a wrong address gives a stranger the token's id (not its secret); a worker node that reaches the database port holds the database credential (a trusted peer, Q9); the lighthouse must be reachable on UDP `mesh.port`; Nebula's own isolation is upstream's (§1.9) — `domain/mesh.py`. | ### Worked routing examples diff --git a/threat-model.yaml b/threat-model.yaml index 3f06d498c..6ba100ab3 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -581,6 +581,15 @@ entry_points: control_kinds: [x-env-config] caller_must_enforce: "admin is trusted; audited (mesh.token.create, mesh.join, mesh.node.*); the network can't change once machines have addresses in it" provenance: {kind: documented, source: 'docs/mesh.md "Settings"'} + - id: "UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the mesh container, while the mesh is on)" + component: optional + parameters: + - name: "Nebula handshakes and tunnelled packets from anyone reaching the port; on the mesh, TCP to the database and the API" + attacker_controllable: true + x-condition: "anyone reaching UDP 4242; on the mesh, a machine with a certificate the mesh's CA signed" + control_kinds: [data, x-peer] + caller_must_enforce: "Nebula drops what no certificate of the mesh's CA signs and anything on the blocklist; its firewall, rendered by Lens, lets only lens-worker reach 8000 and lens-node 8080; the relays listen on the mesh address only; the container is the only one holding NET_ADMIN, makes its own key pair (the private key stays in its own volume) and runs only Nebula's release with the checksum kept in Lens" + provenance: {kind: documented, source: 'docs/mesh.md "On this machine"'} - id: ".env, archive.yaml, LENS_*, ARCHIVE_SECRET_KEY, RCLONE_BINARY" component: outbound parameters: @@ -1921,7 +1930,7 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does" + conditions: "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only" violation_symptoms: [info-leak, integrity-bypass] x-voided-by: "a wrong address in the install line gives a stranger the token's id (not its secret); a worker node holds the database credential (trusted peer, Q9); the lighthouse must be reachable on UDP mesh.port; Nebula's own isolation is upstream's (domain/mesh.py)" provenance: {kind: documented, source: 'docs/mesh.md "What keeps it safe"'} From df5532e147bddd39469b72346de2d666e1e90d2e Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 06:06:17 +0000 Subject: [PATCH 2/6] docs(readme): the mesh's in-progress line points at its open pull request Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f2affbc04..5f5bbd02b 100644 --- a/README.md +++ b/README.md @@ -233,7 +233,7 @@ These are open pull requests, not yet part of Lens: ([#406](https://github.com/adeelahmad/lense/pull/406)). - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)). - **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates - for ([#546](https://github.com/adeelahmad/lense/pull/546)). + for ([#553](https://github.com/adeelahmad/lense/pull/553)). ## Planned From f92e8620a75bb66c5875acd7ceb3bbe94a9057a2 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 06:18:26 +0000 Subject: [PATCH 3/6] test(mcp): the services tool lists the mesh container too Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- fastapi_backend/tests/api/test_mcp.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index fd0027c52..1a2834f56 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -538,7 +538,7 @@ def test_monitoring_tools(client, db, env, monkeypatch): assert "no upgrade agent" in tool_error(client, ha, "set_monitoring", on=True) # and every optional container, the same way got = tool(client, ha, "services") - assert [s["profile"] for s in got["services"]] == ["mcp", "browser", "monitoring"] + assert [s["profile"] for s in got["services"]] == ["mcp", "browser", "monitoring", "mesh"] assert "no upgrade agent" in tool_error(client, ha, "set_service", profile="browser", on=True) assert "only admins" in tool_error(client, env["h"], "set_service", profile="browser", on=True) From 03a20781c7f3842525932b9c774292a5acbc6be0 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 06:37:24 +0000 Subject: [PATCH 4/6] style(mesh): the sidecar is linted and formatted as the server's code, by pre-commit too Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- mesh/project.json | 4 ++-- mesh/ruff.toml | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 mesh/ruff.toml diff --git a/mesh/project.json b/mesh/project.json index 20e3f5560..458425793 100644 --- a/mesh/project.json +++ b/mesh/project.json @@ -9,8 +9,8 @@ "options": { "cwd": "{projectRoot}", "commands": [ - "uvx ruff check --config ../fastapi_backend/pyproject.toml sidecar.py", - "uvx ruff format --check --config ../fastapi_backend/pyproject.toml sidecar.py" + "uvx ruff check sidecar.py", + "uvx ruff format --check sidecar.py" ], "parallel": false } diff --git a/mesh/ruff.toml b/mesh/ruff.toml new file mode 100644 index 000000000..ba15434ab --- /dev/null +++ b/mesh/ruff.toml @@ -0,0 +1,2 @@ +# mesh/sidecar.py is formatted and linted as Lens's server code is +extend = "../fastapi_backend/pyproject.toml" From 832d55d86b537d13f55ff2f89809c78b82e915d8 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 07:32:16 +0000 Subject: [PATCH 5/6] chore: merge main; the mesh container's changelog entry under Unreleased, and #400 listed as built Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- CHANGELOG.md | 12 ++++++++---- README.md | 6 +++--- docs/requirements.md | 1 + 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f633fcedb..6ba79c7bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased +- **Nebula on the primary** (the private network between Lens's machines, in progress). On the primary Nebula runs in + the optional `mesh` container, which the upgrade agent starts when a node is added: it makes its own key pair, checks + Nebula's download against the kept checksums and relays the database and the API on the mesh address only. The + install line that joins a machine and the Settings pages follow in their own pull requests + ([Nodes and the mesh](docs/mesh.md)). + ## 0.15.0 October 18, 2026 {id="0.15.0"} - **The assistant sends mail, adds events and writes files, after your yes.** New tools `send_mail` (send, or save a @@ -89,10 +95,8 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version a key derived from it) and gets an address, a certificate naming its role and namespaces, and its Nebula config, whose firewall lets worker nodes reach the database and every node the API, and nothing else; a node renews, reports and refreshes its config with its own node token, and removing one blocklists its certificates on every machine. - Off until a node is added; settings in `mesh`. On the primary Nebula runs in the optional `mesh` container, which the - upgrade agent starts when a node is added: it makes its own key pair, checks Nebula's download against the kept - checksums and relays the database and the API on the mesh address only. The install line that joins a machine and - the Settings pages follow in their own pull requests ([Nodes and the mesh](docs/mesh.md)). + Off until a node is added; settings in `mesh`. Nebula running on the primary, the install line that joins a machine + and the Settings pages follow in their own pull requests ([Nodes and the mesh](docs/mesh.md)). - **Export a conversation.** `GET /api/v1/chats//export.md`, `.pdf` or `.json` gives the version showing, one message (`?message=`) or, as JSON, every version (`?all=true`). The Markdown has each question and answer, its citations as footnotes linking back to the moment in Lens and the tools it used; the PDF is that Markdown printed by diff --git a/README.md b/README.md index c97293513..64a82d3a4 100644 --- a/README.md +++ b/README.md @@ -86,11 +86,13 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine (opt-in), any message in one click. Long conversations are summarised before they fill the model's context, and `/clear` and `/compact` start afresh or summarise now. Conversations title themselves after their first answer, unless - you name them. Each namespace can have its own assistant with + you name them. It answers questions about Lens itself from its docs and this server. Each namespace can have its own + assistant with its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages), [mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records), + [questions about Lens](docs/assistant.md#questions-about-lens-itself), [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md); [checks](docs/requirements.md#ask-and-get-cited-answers)) - **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python), @@ -234,8 +236,6 @@ still loading. ([Components](docs/components.md), These are open pull requests, not yet part of Lens: -- **The assistant** answering questions about Lens itself, from its docs and this server - ([#400](https://github.com/adeelahmad/lense/pull/400)). - **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)). - **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)). - **Web clipper:** signing in, or using your API token, not only an import webhook's diff --git a/docs/requirements.md b/docs/requirements.md index 3f4080b1c..68b7b6c57 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -87,6 +87,7 @@ request, to another check that proves the same thing; loosening a requirement or | Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` | | Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` | | Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` | +| It answers questions about Lens itself, from its docs and this server | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_assistant_answers_about_lens_itself`, `test_sections_anchors_and_a_changed_page`, `test_agents_ask_lens_over_mcp` | | Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` | | Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app` | | A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` | From f3980c620ef244cc7e03ddc07271eb78274b2964 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 11:02:53 +0000 Subject: [PATCH 6/6] chore: merge main; the mesh container's changelog entry as a fragment in changelog.d Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- CHANGELOG.md | 6 ------ changelog.d/mesh-container.md | 5 +++++ 2 files changed, 5 insertions(+), 6 deletions(-) create mode 100644 changelog.d/mesh-container.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b9723a01..6ff27e90d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,6 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased -- **Nebula on the primary** (the private network between Lens's machines, in progress). On the primary Nebula runs in - the optional `mesh` container, which the upgrade agent starts when a node is added: it makes its own key pair, checks - Nebula's download against the kept checksums and relays the database and the API on the mesh address only. The - install line that joins a machine and the Settings pages follow in their own pull requests - ([Nodes and the mesh](docs/mesh.md)). - ## 0.16.0 October 18, 2026 {id="0.16.0"} ### Features diff --git a/changelog.d/mesh-container.md b/changelog.d/mesh-container.md new file mode 100644 index 000000000..ab2792703 --- /dev/null +++ b/changelog.d/mesh-container.md @@ -0,0 +1,5 @@ +- **Nebula on the primary** (the private network between Lens's machines, in progress). On the primary Nebula runs in + the optional `mesh` container, which the upgrade agent starts when a node is added: it makes its own key pair, checks + Nebula's download against the kept checksums and relays the database and the API on the mesh address only. The + install line that joins a machine and the Settings pages follow in their own pull requests + ([Nodes and the mesh](docs/mesh.md)).