Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
rclone keeps files being written in a cache on Lens's disk and puts them to Lens once they're closed; a file Lens
can't take shows in the share's log. **Import files people add** in Settings, NFS share (`nfs.add_files`, on) turns it
off and serves the share read-only again ([Adding files](docs/file-shares.md#adding-files)).
- **A private network between Lens's machines** (in progress). Lens is the certificate authority of a Nebula mesh
between the machines it runs on: an admin adds a node with a join token (`POST /api/v1/mesh/tokens`), the machine
joins without ever sending the token (`POST /api/v1/mesh/join`: an HMAC proves it, the answer comes back sealed with
a key derived from it) and gets an address, a certificate naming its role and namespaces, and its Nebula config,
whose firewall lets worker nodes reach the database and every node the API, and nothing else; a node renews, reports
and refreshes its config with its own node token, and removing one blocklists its certificates on every machine.
Off until a node is added; settings in `mesh`. Nebula running on the primary, the install line that joins a machine
and the Settings pages follow in their own pull requests ([Nodes and the mesh](docs/mesh.md)).
- **Export a conversation.** `GET /api/v1/chats/<id>/export.md`, `.pdf` or `.json` gives the version showing, one
message (`?message=`) or, as JSON, every version (`?all=true`). The Markdown has each question and answer, its
citations as footnotes linking back to the moment in Lens and the tools it used; the PDF is that Markdown printed by
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,8 @@ These are open pull requests, not yet part of Lens:
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
- **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates
for ([#546](https://github.com/adeelahmad/lense/pull/546)).

## Planned

Expand Down
130 changes: 130 additions & 0 deletions docs/mesh.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Nodes and the mesh

Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the
token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings and the API
under `/api/v1/mesh`. Coming in their own pull requests: Nebula running on the primary and relaying the database,
`lens node` and the install line that joins another machine, Settings › Network and Nodes in the web app, and running
code on compute nodes.

Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code
on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a
[Nebula](https://github.com/slackhq/nebula) mesh (MIT), never over the LAN or the internet in the clear: every packet
between them is encrypted and signed with certificates Lens issued, and Nebula's firewall lets through only what each
role needs. A single-machine install never turns it on.

**Off until an admin adds a node.** Adding a node (Settings › Nodes, or `POST /api/v1/mesh/tokens`) makes the mesh's
certificate authority the first time and turns `mesh.enabled` on. Nothing goes in `.env`; nothing is asked at
install.

## Roles

| Role | What it does | What it reaches on the primary |
| --- | --- | --- |
| **primary** | This install: the lighthouse other machines find each other through, the relay for machines that can't reach each other directly, and the certificate authority | |
| **worker** | Runs jobs (transcription, OCR, the model steps it can) against this install's database, as `lens worker` does on this machine | the database port (8000) and the API (8080) |
| **compute** | Runs code in sandboxes for the assistant. It gets nothing of Lens's: no database, no keys | the API (8080) |

A worker node needs this install's database to be a SurrealDB server, which Docker installs have (`SURREAL_URL`);
on an install with the embedded database only compute nodes can be added. A node may serve some namespaces only
(`namespaces` on its token, changed later on the node): its certificate carries them as Nebula groups
(`ns-<namespace>`) for the firewall, and its workers take only those namespaces' jobs.

## Joining

An admin makes a join token for a role, shown once. It can be used once (or `uses` times) within `hours` (a day by
default), and withdrawn before that. The machine joins at this install's address with:

```sh
curl -fsSL https://raw.githubusercontent.com/adeelahmad/lense/main/install.sh | LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… sh
```

*(the install line's node path lands in its own pull request; the API it calls is here)*

The token never crosses the wire. The machine makes its own Nebula key pair (`nebula-cert keygen`; the private key
never leaves it) and sends `POST /api/v1/mesh/join` with the token's id, a random nonce, its name, its public key and
a proof: `HMAC-SHA256(secret, "lens-mesh-join\n<id>\n<nonce>\n<name>\n<public key>")`. Lens checks the proof in
constant time, spends the token, gives the machine the next address in `mesh.network`, signs a certificate for its
public key naming its role and namespaces as groups, and answers with its certificate, its whole Nebula config and its
credentials, sealed with AES-256-GCM under a key derived from the token's secret (HKDF-SHA256 with the nonce as salt).
So a join over plain `http://` on the LAN gives an eavesdropper nothing, and a machine in the middle can neither join
nor read the answer. Wrong proofs count towards a throttle per address (8 in 15 minutes).

The answer holds: the node's id, name, role, address and groups; the primary's mesh address and ports; the config
(below); a **node token** (`lensnode1.<id>.<secret>`, kept on the node only, as a hash on the primary) for the node's
own calls; and an **agent key** the primary signs its calls to the node with.

## After joining

Over the mesh, with its node token (`Authorization: Bearer lensnode1.…`):

| Call | |
| --- | --- |
| `GET /api/v1/mesh/node/config` | its Nebula config as it is now, and a `version` that changes when the config does (the blocklist grew, the primary's addresses changed): the node reloads Nebula when it sees a new one |
| `POST /api/v1/mesh/node/renew` | a new certificate for its public key (the same or a new one), with the groups its row has now. A node renews at a third of `mesh.cert_days` before its certificate runs out |
| `POST /api/v1/mesh/node/beat` | its report, every minute: whether Nebula runs, its peers, its machine, the sandboxes it can run, Nebula's last lines. Shown on the Nodes page |

A removed node is refused on all three.

## The config

Each machine's config is rendered by Lens, from the settings and the node's row; the node adds only its private key.

- `pki`: the CA certificate, the machine's certificate, and the **blocklist**: the fingerprints of every certificate a
removed node was given. Nebula drops a blocked peer's tunnels and refuses new ones (`disconnect_invalid`).
- `lighthouse` and `static_host_map`: the primary (the network's first address) is the lighthouse; nodes find it at
`mesh.hosts` on `mesh.port`, or, when that's empty, at the address they joined this install at.
- `relay`: the primary relays for nodes that can't reach each other directly (`mesh.relay`).
- `firewall`: outbound anything; inbound ICMP, and on the primary the database port (8000) to the `lens-worker` group
and the API port (8080) to `lens-node`; on a node its agent port (7443) to `lens-primary` only. Everything else is
dropped.
- `tun`: the `lens-mesh` interface, MTU `mesh.mtu`. `cipher`: `mesh.cipher`, the same on every machine.

Nebula's programs come from Nebula's release for the machine (Linux amd64, arm64, ARMv7 and ARMv6 for a Raspberry
Pi; macOS), pinned to one version with its checksum kept in Lens, fetched into `<data_dir>/bin` on first use; a
`nebula` and `nebula-cert` on `PATH` or in `LENS_NEBULA_DIR` are used instead.

## Settings

Everything is in **Settings › Network** (`mesh`), editable in the app with defaults:

| Setting | Default | |
| --- | --- | --- |
| `enabled` | off | on when a node is added; off stops the mesh |
| `network` | `10.73.0.0/16` | the mesh's addresses, a private IPv4 range from /8 to /28; the primary is its first. It can't change once machines have addresses in it |
| `port` | `4242` | the UDP port the primary listens on; nodes must reach it |
| `hosts` | none | where nodes reach that port from outside the mesh (`host` or `host:port`, up to 8). Empty: the address each node joined this install at |
| `cipher` | `aes` | or `chachapoly` (faster on machines without AES hardware, such as some Raspberry Pis); the same on every machine |
| `cert_days` | 30 | how long a machine's certificate lasts; it renews at two thirds |
| `relay` | on | the primary relays for machines that can't reach each other directly |
| `mtu` | 1300 | the mesh interface's MTU |
| `log_level` | info | Nebula's log level |

**Nodes** (`/api/v1/mesh/nodes`): every machine, this one first, with its role, address, certificate and when it runs
out, last report, and the job workers running on it. `PATCH` renames a node or changes its namespaces; `DELETE`
removes it (undo with `POST /nodes/{id}/restore` while its certificate is good). Tokens: `GET`, `POST`, `DELETE
/api/v1/mesh/tokens`.

## What keeps it safe

- Nothing listens until an admin adds a node, and single-machine installs never do.
- Only a machine with a token an admin made joins; the token is spent on use, expires, and never crosses the wire.
- A machine's certificate names only its own address, role and namespaces; it can't widen them by renewing.
- Nebula's firewall, rendered by Lens, opens only what each role needs. Lens itself never trusts the mesh's address:
a worker node signs in to the database with its credential, a node's calls carry its node token, and the primary's
calls to a node carry its agent key.
- Removing a node blocks every certificate it was given, on every machine, and its credentials stop at once.
- The CA's key and the tokens' secrets are sealed in the database like any setting's secret (`ARCHIVE_SECRET_KEY` or
`secret.key`); node tokens are kept only as hashes.
- Nebula's download is pinned to one version with its checksum kept in Lens, unlike programs fetched from a "latest"
release.

What it doesn't do: a worker node holds the database credential, so it is a trusted peer as a worker on this machine
is; a compute node is not, and gets no database. The lighthouse must be reachable on UDP `mesh.port` from the nodes
(a Raspberry Pi behind a carrier's NAT can be a node, not the primary a node reaches directly, unless `mesh.hosts`
names an address that forwards to it). Nebula's own isolation is upstream's.

## Refine later

- A second transport for networks that block UDP.
- Rotating the CA.
- Compute nodes running the assistant's sandboxes through the mesh, and their load shown on the Nodes page.
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
integrations,
jobs,
marketplaces,
mesh,
metadata,
namespaces,
notebook,
Expand Down Expand Up @@ -153,5 +154,6 @@
vaults,
podcasts,
specs,
mesh,
):
api_router.include_router(module.router)
185 changes: 185 additions & 0 deletions fastapi_backend/app/api/v1/routes/mesh.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
"""The mesh between Lens's machines (docs/mesh.md): the network and its machines for admins (Settings › Network and
Nodes), joining it for a new machine (with a join token), and the calls a machine on it makes (with its node token).

Adding a node makes a join token and turns the mesh on. Making and withdrawing tokens, joining, changing and removing
nodes are audited.
"""

from __future__ import annotations

import urllib.parse
from typing import Annotated, Any

from fastapi import APIRouter, Depends, HTTPException, Request

from app.api.deps import AdminReader, AdminWriter, Cfg, Db, client_ip, domain_errors, web_origin
from app.api.v1.routes.jobs import _worker as worker_info
from app.domain import auth, jobs, mesh, settings
from app.schemas.common import Ok
from app.schemas.mesh import (
MeshBeat,
MeshJoin,
MeshJoined,
MeshNode,
MeshNodeChange,
MeshNodeConfig,
MeshRenew,
MeshStatus,
MeshToken,
MeshTokenCreate,
MeshTokenCreated,
)

router = APIRouter(prefix="/mesh", tags=["mesh"])


@router.get("")
def mesh_status(user: AdminReader, db: Db, cfg: Cfg) -> MeshStatus:
"""Whether the mesh is on, its network and port, where other machines reach it, its CA and how many machines are
on it."""
return MeshStatus(**mesh.status(db, cfg))


def _with_workers(db, rows) -> list[MeshNode]:
"""Each machine with the job workers running on it: a node's report as theirs, the rest (this machine's, and any
started by hand against the database) as the primary's."""
by_node: dict[Any, list] = {}
for w in jobs.workers(db):
by_node.setdefault(w.get("node"), []).append(w)
ids = {n["id"] for n in rows}
out = []
for n in rows:
mine = by_node.get(n["id"], [])
if n["role"] == "primary":
mine = mine + [w for k, ws in by_node.items() if k not in ids for w in ws]
out.append(MeshNode(**n, workers=[worker_info(w) for w in mine]))
return out


@router.get("/nodes")
def mesh_nodes(user: AdminReader, db: Db) -> list[MeshNode]:
"""Every machine: this one (the primary) first, then each node with its role, address, certificate, last report
and the workers running on it. Removed nodes are listed too, with when."""
rows = mesh.nodes(db)
if not rows: # no mesh yet: this machine and its workers
rows = [{"id": 0, "name": "This machine", "role": "primary", "ip": "", "groups": [], "namespaces": []}]
return _with_workers(db, rows)


@router.patch("/nodes/{node_id}")
def change_node(node_id: int, body: MeshNodeChange, user: AdminWriter, db: Db) -> MeshNode:
"""Rename a node, or change the namespaces it serves (its certificate's groups follow when it next renews)."""
with domain_errors():
n = mesh.update(db, node_id, name=body.name, namespaces=body.namespaces)
auth.audit(db, user.as_audit(), "mesh.node.change", f"mesh_node:{node_id}", body.model_dump(exclude_none=True))
return _with_workers(db, [n])[0]


@router.delete("/nodes/{node_id}")
def remove_node(node_id: int, user: AdminWriter, db: Db) -> MeshNode:
"""Take a node off the mesh: its certificates go on every machine's blocklist and its credentials stop working.
POST /mesh/nodes/{id}/restore undoes it while its certificate is good."""
with domain_errors():
n = mesh.remove(db, node_id, by=user.email)
auth.audit(db, user.as_audit(), "mesh.node.remove", f"mesh_node:{node_id}")
return _with_workers(db, [n])[0]


@router.post("/nodes/{node_id}/restore")
def restore_node(node_id: int, user: AdminWriter, db: Db) -> MeshNode:
"""Undo removing a node."""
with domain_errors():
n = mesh.undo_remove(db, node_id)
auth.audit(db, user.as_audit(), "mesh.node.restore", f"mesh_node:{node_id}")
return _with_workers(db, [n])[0]


@router.get("/tokens")
def mesh_tokens(user: AdminReader, db: Db) -> list[MeshToken]:
"""Join tokens that can still be used (their secrets are shown only when made)."""
return [MeshToken(**t) for t in mesh.tokens(db)]


@router.post("/tokens")
def make_mesh_token(body: MeshTokenCreate, user: AdminWriter, request: Request, db: Db, cfg: Cfg) -> MeshTokenCreated:
"""Add a node: a token another machine joins the mesh with, shown this once. Makes the mesh's CA the first time,
and turns the mesh on."""
with domain_errors():
try:
tok, secret = mesh.make_token(db, cfg, user.email, body.role, body.name, body.namespaces, body.hours, body.uses)
except mesh.MeshError as e:
raise HTTPException(503, str(e)) from None
if not mesh.conf(cfg)["enabled"]:
settings.save(db, request.app.state.archive.base, "mesh", {"enabled": True}, user.email)
auth.audit(db, user.as_audit(), "settings.save", "mesh", ["enabled"])
auth.audit(db, user.as_audit(), "mesh.token.create", f"mesh_token:{tok['id']}", {"role": body.role, "uses": body.uses})
return MeshTokenCreated(token=MeshToken(**tok), secret=secret, join_url=web_origin(request))


@router.delete("/tokens/{token_id}")
def withdraw_mesh_token(token_id: int, user: AdminWriter, db: Db) -> Ok:
"""Withdraw a join token: no machine can join with it any more (those that did stay)."""
with domain_errors():
mesh.revoke_token(db, token_id)
auth.audit(db, user.as_audit(), "mesh.token.withdraw", f"mesh_token:{token_id}")
return Ok()


@router.post("/join")
def join_mesh(body: MeshJoin, request: Request, db: Db, cfg: Cfg) -> MeshJoined:
"""A machine joins the mesh with a join token, without sending it: the token's id, the machine's public key, and
an HMAC of the request keyed with the token's secret. The answer is sealed with a key from that secret
(docs/mesh.md, Joining). Too many wrong tries from one address are turned away for a while."""
key = f"mesh-join:{client_ip(request)}"
if auth.throttled(key):
raise HTTPException(429, "too many tries; wait a few minutes")
host = urllib.parse.urlsplit(web_origin(request)).hostname or ""
try:
nid, sealed = mesh.join(db, cfg, body.token_id, body.nonce, body.name, body.public_key, body.proof, host, body.info)
except PermissionError as e:
auth.hit(key)
raise HTTPException(403, str(e)) from None
except ValueError as e:
raise HTTPException(400, str(e)) from None
except mesh.MeshError as e:
raise HTTPException(503, str(e)) from None
auth.audit(db, None, "mesh.join", f"mesh_node:{nid}", {"token": body.token_id, "name": body.name, "from": client_ip(request)})
return MeshJoined(node=nid, sealed=sealed)


def _node(request: Request, db: Db) -> dict[str, Any]:
"""The node calling, from its node token (Authorization: Bearer lensnode1.<id>.<secret>)."""
h = request.headers.get("authorization") or ""
row = mesh.by_token(db, h[7:] if h.lower().startswith("bearer ") else "")
if not row:
raise HTTPException(401, "not a node on this install's mesh (or it was removed)")
return row


Node = Annotated[dict[str, Any], Depends(_node)]


@router.get("/node/config")
def node_config(node: Node, db: Db, cfg: Cfg) -> MeshNodeConfig:
"""The calling node's Nebula config as it is now (the blocklist, the lighthouse's addresses): it reloads when the
version changes."""
with domain_errors():
return MeshNodeConfig(**mesh.config_for(db, cfg, node))


@router.post("/node/renew")
def renew_node(body: MeshRenew, node: Node, db: Db, cfg: Cfg) -> MeshNodeConfig:
"""A new certificate for the calling node's public key, with the groups it has now."""
try:
return MeshNodeConfig(**mesh.renew(db, cfg, node, body.public_key))
except ValueError as e:
raise HTTPException(400, str(e)) from None
except mesh.MeshError as e:
raise HTTPException(503, str(e)) from None


@router.post("/node/beat")
def node_beat(body: MeshBeat, node: Node, db: Db) -> Ok:
"""The calling node's report, every minute: Nebula, its peers, its machine."""
mesh.beat(db, node, body.model_dump(exclude_none=True))
return Ok()
Loading
Loading