From 4f2f360b030bcfad03ac01aa007e5da046f2a607 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 09:11:52 +0000 Subject: [PATCH 1/3] feat(mesh): worker nodes run jobs as their own user of the database, with files over the mesh Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- README.md | 5 +- changelog.d/mesh-node.md | 5 +- changelog.d/mesh-workers.md | 7 + docker-compose.node.yml | 8 +- docs/configuration.md | 1 + docs/mesh.md | 46 ++++- docs/requirements.md | 1 + docs/scaling.md | 13 +- fastapi_backend/app/api/v1/routes/mesh.py | 47 ++++- fastapi_backend/app/domain/ingest.py | 5 + fastapi_backend/app/domain/jobs.py | 16 +- fastapi_backend/app/domain/keyring.py | 5 + fastapi_backend/app/domain/mesh.py | 84 +++++++- fastapi_backend/app/domain/mesh_node.py | 181 ++++++++++++++++- fastapi_backend/app/domain/scaling.py | 25 ++- fastapi_backend/app/domain/store.py | 2 + fastapi_backend/app/schemas/mesh.py | 9 + fastapi_backend/app/schemas/scaling.py | 4 + fastapi_backend/tests/api/test_mesh_node.py | 27 ++- .../tests/api/test_mesh_workers.py | 192 ++++++++++++++++++ .../tests/api/test_scaling_clouds.py | 31 ++- nextjs-frontend/app/openapi-client/index.ts | 8 + nextjs-frontend/app/openapi-client/sdk.gen.ts | 35 ++++ .../app/openapi-client/types.gen.ts | 87 ++++++++ nextjs-frontend/openapi.json | 118 +++++++++++ threat-model.json | 12 +- threat-model.md | 3 +- threat-model.yaml | 13 +- 28 files changed, 942 insertions(+), 48 deletions(-) create mode 100644 changelog.d/mesh-workers.md create mode 100644 fastapi_backend/tests/api/test_mesh_workers.py diff --git a/README.md b/README.md index ee136c488..56453f6ca 100644 --- a/README.md +++ b/README.md @@ -202,8 +202,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today; - AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching - once the mesh's install line lands), and Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); + AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, with + machines that join over the mesh to transcribe and tell voices apart, not yet tried on a real cloud), and Linux and + model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out)) - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine, diff --git a/changelog.d/mesh-node.md b/changelog.d/mesh-node.md index cab5f1786..48a16d665 100644 --- a/changelog.d/mesh-node.md +++ b/changelog.d/mesh-node.md @@ -1,5 +1,4 @@ - **Joining a node** (in progress). Another machine joins with the install line and `LENS_JOIN` and `LENS_JOIN_TOKEN`: it asks nothing, publishes no port, and runs Nebula and `lens node`, which joins once and then follows the primary's - config, renews its certificate and reports over the mesh. Adding a node token now returns that install line. Worker - nodes running jobs and the Settings pages follow in their own pull requests - ([On a node](docs/mesh.md#on-a-node)). + config, renews its certificate and reports over the mesh. Adding a node token now returns that install line. The + Settings pages follow in their own pull request ([On a node](docs/mesh.md#on-a-node)). diff --git a/changelog.d/mesh-workers.md b/changelog.d/mesh-workers.md new file mode 100644 index 000000000..f326a93a2 --- /dev/null +++ b/changelog.d/mesh-workers.md @@ -0,0 +1,7 @@ +- **Worker nodes run jobs** (in progress). On a worker node, `lens node` also runs `lens worker`, against this + install's database through the mesh, signed in as a user of that one database made for the node (removed with it; + `SURREAL_AUTH=database`). Its jobs fetch their files from the primary over the mesh, plain, never a vault's or + another namespace's, and it never gets the archive's secret, so it runs `transcribe` and `diarize` with the models + on that machine, skipping any other step `LENS_WORKER_STEPS` names. A worker that serves some namespaces takes only + their jobs, and an auto scaling group on machines elsewhere counts only the jobs a worker node takes, so it never + launches one for others ([Worker nodes](docs/mesh.md#worker-nodes)). diff --git a/docker-compose.node.yml b/docker-compose.node.yml index 52f8d84d2..871be6eb6 100644 --- a/docker-compose.node.yml +++ b/docker-compose.node.yml @@ -1,6 +1,7 @@ # A machine on another Lens install's mesh (docs/mesh.md, Nodes): what install.sh starts with LENS_JOIN and # LENS_JOIN_TOKEN. No database, no web app, nothing published: `mesh` runs Nebula, and `node` (`lens node`) joins once, -# then follows the primary's config over the mesh and reports every minute. +# then follows the primary's config over the mesh and reports every minute; on a worker node it runs `lens worker` too, +# against the primary's database over the mesh. # LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN=lensjoin1.… docker compose -p lens-node -f docker-compose.node.yml up -d --build name: lens-node @@ -36,9 +37,8 @@ services: depends_on: [mesh] security_opt: ["no-new-privileges:true"] cap_drop: [ALL] - read_only: true - tmpfs: [/tmp] - mem_limit: 512m + # not read-only and no memory cap: on a worker node it also runs `lens worker`, which fetches its speech model + # into the image's folders and needs the machine's memory, as the primary's worker does healthcheck: disable: true # the image's check is the API's, which a node doesn't run environment: diff --git a/docs/configuration.md b/docs/configuration.md index 05f6089d7..3e18db451 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -22,6 +22,7 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v | `PASSWORD_RESET_EXPIRE_MINUTES` | `60` | | | `SURREAL_URL` | embedded | `ws://host:8000` for a server; see [Database](database.md) | | `SURREAL_USER` / `SURREAL_PASS` / `SURREAL_NS` / `SURREAL_DB` / `SURREAL_POOL_SIZE` | `root` / `root` / `archive` / `main` / `8` | | +| `SURREAL_AUTH` | root | `database`: `SURREAL_USER` is a user of that one database, not the server's root (a [worker node](mesh.md#worker-nodes) signs in so) | | `ARCHIVE_SECRET_KEY` | generated in `data_dir/secret.key` | encrypts stored credentials; keep it stable and backed up | | `ARCHIVE_CONFIG` | `archive.yaml` | processing configuration file | | `ARCHIVE_ALLOWED_HOSTS` | | break-glass override of allowed Host headers if a bad setting locks everyone out | diff --git a/docs/mesh.md b/docs/mesh.md index e25bd337b..09d2862c1 100644 --- a/docs/mesh.md +++ b/docs/mesh.md @@ -3,8 +3,8 @@ Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API under `/api/v1/mesh`, Nebula running on the primary in the `mesh` container, relaying the database and the API, and -`lens node` with the install line that joins another machine. Coming in their own pull requests: worker nodes running -jobs, Settings › Network and Nodes in the web app, and running code on compute nodes. +`lens node` with the install line that joins another machine, and worker nodes running jobs. Coming in their own pull +requests: Settings › Network and Nodes in the web app, and running code on compute nodes. Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a @@ -21,7 +21,7 @@ install. | Role | What it does | What it reaches on the primary | | --- | --- | --- | | **primary** | This install: the lighthouse other machines find each other through, the relay for machines that can't reach each other directly, and the certificate authority | | -| **worker** | Runs jobs (transcription, OCR, the model steps it can) against this install's database, as `lens worker` does on this machine | the database port (8000) and the API (8080) | +| **worker** | Runs jobs against this install's database, as `lens worker` does on this machine: transcribing and telling voices apart with the models on that machine ([Worker nodes](#worker-nodes)) | the database port (8000) and the API (8080) | | **compute** | Runs code in sandboxes for the assistant. It gets nothing of Lens's: no database, no keys | the API (8080) | A worker node needs this install's database to be a SurrealDB server, which Docker installs have (`SURREAL_URL`); @@ -91,6 +91,31 @@ environment only, never in `.env`, and isn't needed again: running the line agai same node. A node that's removed keeps asking, so an undo in Settings › Nodes brings it back; `docker compose -f docker-compose.node.yml exec node lens node status` says what it is on the mesh, without its credentials. +## Worker nodes + +On a worker node, `lens node` also runs `lens worker` once the node is on the mesh, and keeps it running: it's started +again when it stops, stopped while the node is removed, and started with the new password after an undo. Its errors +and its last lines are in the node's report on the Nodes page. + +- **The database through the mesh.** It reaches this install's SurrealDB at the primary's mesh address (port 8000, + open to worker nodes only), signed in as a user of that one database made for the node (`lens_node_`, its + owner): never the server's root, and nothing on the server beyond Lens's database. The node asks for it + (`GET /api/v1/mesh/node/worker`, with its node token) and hands it to the worker's environment only; it isn't kept + on the node's disk. Removing the node removes the user, and an undo makes it again with a new password. + `SURREAL_AUTH=database` is what makes `lens worker` sign in so ([Configuration](configuration.md)). +- **Files from here.** A job's file is fetched from the primary over the mesh (`GET /api/v1/mesh/node/media/`), + plain, and kept on the node in `data_dir/cache/mesh` until it's unused for 6 hours; it's fetched again only when it + changed. A vault's files never leave this machine, nor those of a namespace the node doesn't serve, whose jobs it + doesn't take either. +- **No secrets of Lens's.** It never gets the archive's secret (`ARCHIVE_SECRET_KEY`): namespace keys and settings' + secrets (a cloud speech or chat model's key) stay sealed. So by default it runs the steps that need neither and + write only to the database: `transcribe` and `diarize`, with the models on that machine (fetched there on first + use). `LENS_WORKER_STEPS` on the install line picks among those; any other step it names is skipped (the node's + report says so), since its jobs would fail there, and stays with this machine's worker. An auto scaling group on + machines elsewhere counts only the jobs these steps run, so it never launches a machine for work it wouldn't take. +- **Its name** is the node's, or the auto scaling unit's (`LENS_SCALING_UNIT`), and it runs only the steps it's + given, never routines or the other services this machine's worker runs. + ## On this machine Lens's server can't create network devices, and nothing in Lens but Nebula needs to, so on the primary Nebula runs in @@ -169,22 +194,29 @@ removes it (undo with `POST /nodes/{id}/restore` while its certificate is good). - Only a machine with a token an admin made joins; the token is spent on use, expires, and never crosses the wire. - A machine's certificate names only its own address, role and namespaces; it can't widen them by renewing. - Nebula's firewall, rendered by Lens, opens only what each role needs. Lens itself never trusts the mesh's address: - a worker node signs in to the database with its credential, a node's calls carry its node token, and the primary's - calls to a node carry its agent key. + a worker node signs in to the database as its own user of Lens's database, a node's calls carry its node token, and + the primary's calls to a node carry its agent key. +- A worker node never gets the archive's secret, a vault's files or another namespace's: it reads the files of the + namespaces it serves, plain, over the mesh. - Removing a node blocks every certificate it was given, on every machine, and its credentials stop at once. - The CA's key and the tokens' secrets are sealed in the database like any setting's secret (`ARCHIVE_SECRET_KEY` or `secret.key`); node tokens are kept only as hashes. - Nebula's download is pinned to one version with its checksum kept in Lens, unlike programs fetched from a "latest" release. -What it doesn't do: a worker node holds the database credential, so it is a trusted peer as a worker on this machine -is; a compute node is not, and gets no database. The lighthouse must be reachable on UDP `mesh.port` from the nodes +What it doesn't do: a worker node owns Lens's database (its rows, which aren't encrypted by Lens) and keeps plain +copies of the files it worked on for a few hours, so it is a trusted peer as a worker on this machine is; a compute +node is not, and gets no database. The lighthouse must be reachable on UDP `mesh.port` from the nodes (a Raspberry Pi behind a carrier's NAT can be a node, not the primary a node reaches directly, unless `mesh.hosts` names an address that forwards to it). Nebula's own isolation is upstream's. ## Refine later - The upgrade agent on nodes: today a node is updated by running its install line again. +- Worker nodes running steps that write files (pages, frames, renditions), once those go through the file store, and + steps that need a provider's key, through calls to the primary rather than the key. +- A job a worker node can't do (a cloud speech provider set since it started) handed back to this machine. +- An auto scaling unit draining its worker on a spot instance's two-minute notice. - A second transport for networks that block UDP. - Rotating the CA. diff --git a/docs/requirements.md b/docs/requirements.md index 0f7fb8f72..d920df480 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -248,6 +248,7 @@ request, to another check that proves the same thing; loosening a requirement or | Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` | | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` | | DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` | +| A group of machines elsewhere counts only the jobs a worker node takes, and never launches one for others | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_group_elsewhere_counts_only_the_jobs_a_node_takes` | | Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` | | AWS EC2 units launch as spot or on-demand instances, hardened (IMDSv2, encrypted disk, terminate on shutdown), with only the mesh's one-time join line in their user-data | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_a_spot_unit_launches_hardened_and_priced_in_its_zone`, `test_the_driver_is_offered_and_its_settings_are_checked` | | A spot unit falls back to on demand when AWS has none; without a price or the mesh, nothing launches | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to`, `test_a_unit_without_a_price_or_the_mesh_launches_nothing` | diff --git a/docs/scaling.md b/docs/scaling.md index c59453358..0ddb280d6 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -10,8 +10,8 @@ Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the A Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that runs more `lens worker` processes where Lens runs. In progress: **AWS EC2** (spot or on demand), **DigitalOcean** and **Hetzner Cloud** providers are built (credentials checked, prices, launching, watching and deleting machines), -and launch once the mesh's install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is -**Proxmox VE**, clones of a VM template on your own server. Planned: providers for any Linux host over SSH; model hosts +and their machines join over the [mesh](mesh.md) as worker nodes that transcribe and tell voices apart; so is +**Proxmox VE**, clones of a VM template on your own server. They haven't been tried on a real cloud machine yet. Planned: providers for any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings. Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms @@ -67,8 +67,11 @@ it runs. A cloud provider's unit is a new machine. Its first boot runs a script (user-data) that sets `LENS_SCALING_UNIT` (its name) and `LENS_WORKER_STEPS` (the template's steps), then the mesh's install line with a join token made for that -unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) and starts `lens worker`; the -database credential travels inside the sealed join answer, so the user-data holds no long-lived secret. Each machine +unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) as a worker node and starts +`lens worker`; it asks for its own database user over the mesh once it has joined, so the user-data holds no +long-lived secret. A worker node runs only `transcribe` and `diarize` (it never gets the archive's key: +[Worker nodes](mesh.md#worker-nodes)), so a group on these providers counts only those steps' jobs in its metrics +(`unit_steps`), and one whose template names none of them waits and launches nothing. Each machine carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=`), so Lens's machines can be told apart. Letting a unit go deletes its machine; a machine switched off at the provider counts as gone and is deleted. @@ -80,7 +83,7 @@ deleted. Letting a unit go stops the clone and deletes it with its disks. The AP start, stop and delete VMs and use their guest agent (the `PVEVMAdmin` role on the template and the clones' pool is enough). -Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches +Until the mesh is on (Settings › Network), a group on these providers waits (`waiting_for` says so) and launches nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing. ## AWS EC2 diff --git a/fastapi_backend/app/api/v1/routes/mesh.py b/fastapi_backend/app/api/v1/routes/mesh.py index e938cc95f..e220bdc79 100644 --- a/fastapi_backend/app/api/v1/routes/mesh.py +++ b/fastapi_backend/app/api/v1/routes/mesh.py @@ -7,14 +7,16 @@ from __future__ import annotations +import os import urllib.parse from typing import Annotated, Any from fastapi import APIRouter, Depends, HTTPException, Request +from fastapi.responses import Response, StreamingResponse from app.api.deps import AdminReader, AdminWriter, Cfg, Db, client_ip, domain_errors, web_origin from app.api.v1.routes.jobs import _worker as worker_info -from app.domain import auth, jobs, mesh, settings +from app.domain import auth, jobs, keyring, mesh, settings from app.schemas.common import Ok from app.schemas.mesh import ( MeshBeat, @@ -28,6 +30,7 @@ MeshToken, MeshTokenCreate, MeshTokenCreated, + MeshWorkerAccess, ) router = APIRouter(prefix="/mesh", tags=["mesh"]) @@ -86,10 +89,10 @@ def remove_node(node_id: int, user: AdminWriter, db: Db) -> MeshNode: @router.post("/nodes/{node_id}/restore") -def restore_node(node_id: int, user: AdminWriter, db: Db) -> MeshNode: +def restore_node(node_id: int, user: AdminWriter, db: Db, cfg: Cfg) -> MeshNode: """Undo removing a node.""" with domain_errors(): - n = mesh.undo_remove(db, node_id) + n = mesh.undo_remove(db, node_id, cfg) auth.audit(db, user.as_audit(), "mesh.node.restore", f"mesh_node:{node_id}") return _with_workers(db, [n])[0] @@ -179,6 +182,44 @@ def renew_node(body: MeshRenew, node: Node, db: Db, cfg: Cfg) -> MeshNodeConfig: raise HTTPException(503, str(e)) from None +@router.get("/node/worker") +def node_worker(node: Node, db: Db, cfg: Cfg) -> MeshWorkerAccess: + """What the calling worker node's `lens worker` signs in to the database with: its own user of it. 403 for a node + of another role, 409 where the database is embedded.""" + try: + return MeshWorkerAccess(**mesh.worker_access(db, cfg, node)) + except PermissionError as e: + raise HTTPException(403, str(e)) from None + except mesh.MeshError as e: + raise HTTPException(409, str(e)) from None + + +@router.get("/node/media/{recording_id}", response_class=StreamingResponse) +def node_media(recording_id: int, node: Node, request: Request, db: Db, cfg: Cfg) -> Response: + """A recording's file, plain, for a job on the calling worker node (a vault's never). Its ETag is its size and + time: a node that has it sends If-None-Match and gets 304.""" + try: + path = mesh.media(db, cfg, node, recording_id) + st = os.stat(path) + tag = f'"{st.st_size}-{st.st_mtime_ns}"' + if request.headers.get("if-none-match") == tag: + return Response(status_code=304, headers={"ETag": tag}) + f = keyring.open_plain(db, cfg, path) + except PermissionError as e: + raise HTTPException(403, str(e)) from None + except FileNotFoundError as e: + raise HTTPException(404, str(e)) from None + except keyring.Locked: + raise HTTPException(423, "its namespace is a locked vault") from None + + def chunks(): + with f: + while block := f.read(1 << 20): + yield block + + return StreamingResponse(chunks(), media_type="application/octet-stream", headers={"ETag": tag}) + + @router.post("/node/beat") def node_beat(body: MeshBeat, node: Node, db: Db) -> Ok: """The calling node's report, every minute: Nebula, its peers, its machine.""" diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index 303257ef2..1efbc6594 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -484,6 +484,11 @@ def audio_path(db, cfg, rec, plain=True): encrypted file comes as a plain working copy for the tools to read, unless `plain` is False.""" from . import keyring + from . import mesh_node + + if mesh_node.on_worker_node(): # a worker node's job: the file comes from the primary over the mesh, plain + local = None if rec.get("remote") else store.resolve_path(cfg, rec.get("path")) + return local if local and pathlib.Path(local).is_file() else str(mesh_node.media(cfg, rec)) if rec.get("remote"): from . import sources diff --git a/fastapi_backend/app/domain/jobs.py b/fastapi_backend/app/domain/jobs.py index c0016071f..a1a718b95 100644 --- a/fastapi_backend/app/domain/jobs.py +++ b/fastapi_backend/app/domain/jobs.py @@ -365,13 +365,27 @@ def enqueue_pending(db, space=None, by=None): return out +def node_spaces(db): + """On a worker node that serves some namespaces only (docs/mesh.md), their ids; else None (every namespace).""" + if MESH_NODE is None: + return None + names = (db.one("SELECT namespaces FROM $r", r=R("mesh_node", MESH_NODE)) or {}).get("namespaces") or [] + if not names: + return None + return sorted(sid for sid, name in store.space_names(db).items() if name in names) + + def claim(db, worker, can): locked = keyring.locked_vaults(db) # a vault's work waits until someone unlocks it here + only = node_spaces(db) for r in db.rows( "SELECT record::id(id) AS id, space, priority, created_at, started_at FROM job WHERE status = 'queued' AND next_step IN $can " - "AND space NOTINSIDE $locked ORDER BY priority DESC, created_at ASC, id ASC LIMIT 10", + "AND space NOTINSIDE $locked" + + ("" if only is None else " AND space INSIDE $only") + + " ORDER BY priority DESC, created_at ASC, id ASC LIMIT 10", can=sorted(can), locked=sorted(locked), + only=only, ): t = store.now() try: diff --git a/fastapi_backend/app/domain/keyring.py b/fastapi_backend/app/domain/keyring.py index c3ee69328..a4c543ee7 100644 --- a/fastapi_backend/app/domain/keyring.py +++ b/fastapi_backend/app/domain/keyring.py @@ -246,6 +246,11 @@ def unlocked_until(db, sid): return _until(db).get(int(sid)) +def is_vault(db, sid): + """Whether namespace `sid` is a vault (its key opens with a passkey only).""" + return bool(db.values("SELECT VALUE space FROM data_key WHERE space = $s AND vault = true", s=int(sid))) + + def locked_vaults(db): """The vault namespaces nobody has unlocked in this process: their work waits.""" out = set() diff --git a/fastapi_backend/app/domain/mesh.py b/fastapi_backend/app/domain/mesh.py index ce4750415..4a6df1a5d 100644 --- a/fastapi_backend/app/domain/mesh.py +++ b/fastapi_backend/app/domain/mesh.py @@ -689,6 +689,8 @@ def join(db, cfg, token_id, nonce, name, public_key, sent_proof, joined_at=None, db.q("DELETE $r", r=R("mesh_node", nid)) raise _record_cert(db, nid, cert_info, crt) + if role == "worker": + grant_database(db, cfg, nid) row = db.one(f"SELECT {NODE_FIELDS} FROM $r", r=R("mesh_node", nid)) config = render(db, cfg, row, crt) payload = { @@ -749,6 +751,82 @@ def beat(db, row, status): ) +# ---------- worker nodes: jobs run there, against this install's database ---------- +# A worker node runs `lens worker` (started by its agent, mesh_node.Worker) against the database through the mesh +# container's relay (DB_PORT on the primary's mesh address, open to the lens-worker group only). It signs in as a user of +# this one database, its own (grant_database), never the server's root. It never gets the archive's secret: the files +# its jobs read come from here over the mesh, already plain (media()), a vault's never, and settings' secrets (a cloud +# speech or chat model's key) stay sealed, so a worker node runs the steps that need neither and write only rows: +# transcribing and telling voices apart with models on that machine (WORKER_STEPS). Removing the node removes its +# user; an undo makes it again with a new password. +WORKER_STEPS = ("transcribe", "diarize") + + +def _db_user(nid): + return f"lens_node_{int(nid)}" + + +def grant_database(db, cfg, nid): + """The node's own user of this database (its owner, and nothing else on the server), with a new password sealed + in its row.""" + pw = secrets.token_urlsafe(32) + if not re.fullmatch(r"[A-Za-z0-9_-]+", pw): # it goes into the statement as a literal: DEFINE USER takes no parameters + raise MeshError("couldn't make a password for the node's database user") + db.q(f"DEFINE USER OVERWRITE {_db_user(nid)} ON DATABASE PASSWORD '{pw}' ROLES OWNER") + db.q("UPDATE $r SET db_password = $p", r=R("mesh_node", int(nid)), p=settings.seal(cfg, pw, f"mesh:db.{int(nid)}")) + + +def revoke_database(db, nid): + db.q(f"REMOVE USER IF EXISTS {_db_user(nid)} ON DATABASE") + db.q("UPDATE $r SET db_password = NONE", r=R("mesh_node", int(nid))) + + +def worker_access(db, cfg, row): + """What a worker node's `lens worker` signs in to the database with. PermissionError for a node of another role; + MeshError where there's no database server for it to reach.""" + if row["role"] != "worker": + raise PermissionError("only a worker node runs jobs: this one is a " + row["role"]) + if not str(db.url).startswith(("ws://", "wss://", "http://", "https://")): + raise MeshError( + "this install's database is embedded: worker nodes need the SurrealDB server (database.url ws://…, as Docker installs have)" + ) + sealed = (db.one("SELECT db_password FROM $r", r=R("mesh_node", int(row["id"]))) or {}).get("db_password") + if not sealed: + grant_database(db, cfg, row["id"]) # a worker that joined before worker nodes ran jobs + sealed = db.one("SELECT db_password FROM $r", r=R("mesh_node", int(row["id"])))["db_password"] + ns, name = getattr(db, "_target", (cfg["database"]["namespace"], cfg["database"]["database"])) + return { + "url": f"ws://{primary_ip(cfg)}:{DB_PORT}", + "namespace": ns, + "database": name, + "user": _db_user(row["id"]), + "password": settings.unseal(cfg, sealed, f"mesh:db.{int(row['id'])}"), + "steps": list(WORKER_STEPS), + } + + +def media(db, cfg, row, rid): + """The path of recording `rid`'s file here, for a job on a worker node (read it with keyring.open_plain). + PermissionError for a node that isn't a worker, a namespace it doesn't serve or a vault's file; FileNotFoundError + when it's gone.""" + from . import ingest, keyring + + if row["role"] != "worker": + raise PermissionError("only a worker node reads the archive's files") + rec = db.one("SELECT record::id(id) AS id, path, remote, space FROM $r", r=R("recording", int(rid))) + if not rec: + raise FileNotFoundError(f"there's no recording {rid}") + served = row.get("namespaces") or [] + if served and store.space_names(db).get(rec.get("space")) not in served: + raise PermissionError("that recording is in a namespace this node doesn't serve") + if rec.get("space") is not None and keyring.is_vault(db, rec["space"]): + raise PermissionError("a vault's files never leave this machine") + path = ingest.audio_path(db, cfg, rec, plain=False) + if not path or not os.path.isfile(path): + raise FileNotFoundError(f"recording {rid}'s file isn't on this machine") + return path + + # ---------- the admin's changes ---------- def update(db, nid, name=None, namespaces=None): """Rename a node, or change the namespaces it serves (its certificate's groups follow when it next renews, which @@ -781,16 +859,20 @@ def remove(db, nid, by=None): if n["role"] == "primary": raise ValueError("the primary is this machine; turn the mesh off in Settings › Network instead") db.q("UPDATE $r SET revoked_at = $t, revoked_by = $b", r=R("mesh_node", int(nid)), t=store.now(), b=by) + if n["role"] == "worker": + revoke_database(db, nid) return node(db, nid) -def undo_remove(db, nid): +def undo_remove(db, nid, cfg): n = node(db, nid) if not n["revoked_at"]: return n if n["not_after"] and n["not_after"] < _ts(): raise ValueError("its certificate has run out since: add it again with a new token") db.q("UPDATE $r SET revoked_at = NONE, revoked_by = NONE", r=R("mesh_node", int(nid))) + if n["role"] == "worker": + grant_database(db, cfg, nid) # a new password: the node fetches it with its next worker_access return node(db, nid) diff --git a/fastapi_backend/app/domain/mesh_node.py b/fastapi_backend/app/domain/mesh_node.py index 8ceceb3ee..23d39b4f3 100644 --- a/fastapi_backend/app/domain/mesh_node.py +++ b/fastapi_backend/app/domain/mesh_node.py @@ -12,6 +12,10 @@ every minute asks the primary for its config as it is now (a removed node's certificate on the blocklist), renews its certificate when a third of its life is left, and reports how it runs (mesh.beat). A node the admin removed is refused; it keeps asking, so an undo brings it back. +3. **A worker node runs jobs.** On a worker node it also runs `lens worker` (Worker), signed in to the primary's + database through the mesh as the node's own user of it (mesh.worker_access), started again when it stops or that + user's password changes, and stopped while the node is removed. The worker's jobs read their files from the primary + over the mesh (media()). """ from __future__ import annotations @@ -23,7 +27,9 @@ import platform import re import secrets +import shutil import socket +import subprocess import time import httpx @@ -97,6 +103,7 @@ def __init__(self, state, shared, join_url=None, token=None, name=None, http=Non self.next_try = 0.0 self.last_beat = 0.0 self._reported = None + self.worker = Worker(self) # ---------- what's kept ---------- @property @@ -193,6 +200,7 @@ def follow(self, pub): if not self.removed: say("this machine was removed from the mesh in Settings › Nodes: waiting in case that's undone") self.removed = True + self.worker.stop() return r.raise_for_status() if self.removed: @@ -208,13 +216,16 @@ def follow(self, pub): if got["version"] != self.node.get("version") or got.get("not_after") != self.node.get("not_after"): self.node.update(config=got["config"], version=got["version"], not_after=got.get("not_after"), renew_days=got.get("renew_days")) self.save() + if self.node["node"]["role"] == "worker": + self.worker.sync() s = self.service() report = { "nebula": s.get("state") or ("running" if s.get("nebula") else "stopped"), "version": updates.current(), "machine": machine(), - "error": str(s["error"])[:500] if s.get("error") else None, # this call reached the primary: its own are past - "log": [str(x) for x in (s.get("log") or [])][-40:], + # this call reached the primary: its own errors are past; the mesh container's and the worker's aren't + "error": (str(s["error"]) if s.get("error") else self.worker.error or "")[:500] or None, + "log": ([str(x) for x in (s.get("log") or [])] + self.worker.log)[-40:], } self.http.post(self.api("/node/beat"), json={k: v for k, v in report.items() if v is not None}, headers=self._headers()) @@ -260,13 +271,16 @@ def report(self): self._reported = now def loop(self, stop): - while not stop.is_set(): - self.tick() - try: - self.report() - except OSError as e: - say(f"can't write agent.json: {e}") - stop.wait(TICK) + try: + while not stop.is_set(): + self.tick() + try: + self.report() + except OSError as e: + say(f"can't write agent.json: {e}") + stop.wait(TICK) + finally: + self.worker.stop() def view(self): """`lens node status`: what this machine is on the mesh, without its credentials.""" @@ -290,6 +304,155 @@ def view(self): } +class Worker: + """`lens worker` on a worker node (the module's docstring, 3): Agent.follow syncs it every minute.""" + + RESTART = (5, 30, 120) + + def __init__(self, agent, popen=subprocess.Popen): + self.agent, self.popen = agent, popen + self.proc = None + self.key = None + self.error = None + self.log = [] + self.starts = 0 + self.not_before = 0.0 + + def _say(self, msg): + say(msg) + self.log = (self.log + [f"{_now().strftime('%H:%M:%S')} {msg}"])[-10:] + + def running(self): + return self.proc is not None and self.proc.poll() is None + + def sync(self): + a = self.agent + r = a.http.get(a.api("/node/worker"), headers=a._headers()) + if r.status_code in (403, 409): + self.error = _detail(r) + self.stop() + return + r.raise_for_status() + access = r.json() + key = (access["url"], access["user"], access["password"]) + if self.running() and key == self.key: + self.error = None + return + if self.proc is not None and not self.running(): + code = self.proc.returncode + self.proc = None + self.error = f"lens worker stopped (exit {code})" + self._say(self.error) + self.not_before = time.monotonic() + self.RESTART[min(self.starts - 1, len(self.RESTART) - 1)] + self.stop() + if time.monotonic() < self.not_before: + return + self.start(access, key) + + def start(self, access, key): + n = self.agent.node["node"] + allowed = list(access["steps"]) + asked = [s.strip() for s in (os.environ.get("LENS_WORKER_STEPS") or "").split(",") if s.strip()] + # only the steps a node can run (it has no archive key): the rest would fail jobs the primary's worker takes + picked = [s for s in asked if s in allowed] if asked else allowed + if not picked: + msg = f"none of LENS_WORKER_STEPS run on a node; it runs {', '.join(allowed)}" + if msg != self.error: # said once, not every minute + self._say(msg) + self.error = msg + return + if len(picked) < len(asked): + self._say(f"skipping {', '.join(s for s in asked if s not in allowed)}: a node runs {', '.join(allowed)}") + steps = ",".join(picked) + name = machine_name(os.environ.get("LENS_SCALING_UNIT") or n["name"]) + env = {k: v for k, v in os.environ.items() if k not in ("LENS_JOIN_TOKEN", "ARCHIVE_SECRET_KEY")} + env.update( + SURREAL_URL=access["url"], + SURREAL_USER=access["user"], + SURREAL_PASS=access["password"], + SURREAL_NS=access["namespace"], + SURREAL_DB=access["database"], + SURREAL_AUTH="database", + LENS_MESH_NODE=str(n["id"]), + ) + cmd = [shutil.which("lens") or "lens", "worker", "--name", name, "--steps", steps, "--no-schedule"] + self.proc = self.popen(cmd, env=env) + self.key = key + self.starts += 1 + self.error = None + self._say(f"started lens worker {name} ({steps})") + + def stop(self): + if self.proc is None: + return + if self.proc.poll() is None: + self.proc.terminate() + try: + self.proc.wait(30) + except subprocess.TimeoutExpired: + self.proc.kill() + self._say("stopped lens worker") + self.proc = None + self.key = None + + +# ---------- in a worker node's `lens worker`: the files its jobs read ---------- +MEDIA_HOURS = 6 + + +def on_worker_node(): + """Whether this process is a worker node's `lens worker` (Worker started it).""" + return os.environ.get("LENS_MESH_NODE", "").isdigit() and bool(os.environ.get("LENS_NODE_DIR")) + + +def media(cfg, rec, http=None): + """A local copy of recording `rec`'s file, fetched from the primary over the mesh (mesh.media) and kept in + data_dir/cache/mesh, fetched again only when it changed there, and removed MEDIA_HOURS after it was last used.""" + node = json.loads((pathlib.Path(os.environ["LENS_NODE_DIR"]) / "node.json").read_text()) + folder = pathlib.Path(cfg["data_dir"]) / "cache" / "mesh" + folder.mkdir(parents=True, exist_ok=True, mode=0o700) + _sweep(folder) + rid = int(rec["id"]) + dest = folder / f"{rid}{pathlib.PurePosixPath(rec.get('path') or (rec.get('remote') or {}).get('path') or '').suffix.lower()}" + tag_file = dest.with_name(dest.name + ".etag") + headers = {"Authorization": f"Bearer {node['node_token']}"} + if dest.exists() and tag_file.exists(): + headers["If-None-Match"] = tag_file.read_text().strip() + url = f"http://{node['primary']['ip']}:{node['primary']['api_port']}/api/v1/mesh/node/media/{rid}" + client = http or httpx.Client(timeout=httpx.Timeout(60, read=600)) + try: + with client.stream("GET", url, headers=headers) as r: + if r.status_code == 304: + os.utime(dest) + return dest + if r.status_code in (403, 404, 423): + r.read() + raise FileNotFoundError(f"the primary won't give recording {rid}'s file: {_detail(r)}") + r.raise_for_status() + tmp = dest.with_name(dest.name + ".part") + with open(tmp, "wb") as f: + os.chmod(tmp, 0o600) + for block in r.iter_bytes(1 << 20): + f.write(block) + tmp.replace(dest) + tag_file.write_text(r.headers.get("etag") or "") + finally: + if http is None: + client.close() + return dest + + +def _sweep(folder): + old = time.time() - MEDIA_HOURS * 3600 + for p in folder.iterdir(): + try: + if p.suffix != ".etag" and p.stat().st_mtime < old: + p.unlink() + p.with_name(p.name + ".etag").unlink(missing_ok=True) + except OSError: + pass + + def _detail(r): try: return str(r.json().get("detail") or r.text) diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py index bfe2635bf..8aa335c7c 100644 --- a/fastapi_backend/app/domain/scaling.py +++ b/fastapi_backend/app/domain/scaling.py @@ -580,10 +580,22 @@ def group_view(db, g, now=None): "updated_at": g.get("updated_at"), "updated_by": g.get("updated_by"), } + out["unit_steps"] = unit_steps((g.get("template") or {}).get("steps"), out["needs_join"]) out["waiting_for"] = waiting_for(out) return out +def unit_steps(steps, needs_join): + """The steps a group's units run, which its queue metrics count: the template's (None: every step). A machine + elsewhere joins as a worker node, which runs only the steps a node can, with no archive key (mesh.WORKER_STEPS): + counting jobs it never takes would launch machines for nothing. [] when none of the template's are such steps.""" + mesh = mesh_module() if needs_join else None + allowed = getattr(mesh, "WORKER_STEPS", None) + if not allowed: + return list(steps) if steps else None + return [s for s in steps if s in allowed] if steps else list(allowed) + + def waiting_for(view, cfg=None): """Why a group launches nothing now, in words, or None.""" if cfg is not None and not enabled(cfg): @@ -597,7 +609,9 @@ def waiting_for(view, cfg=None): if view.get("capped"): return f"its budget: {view['capped']}" if cfg is not None and view.get("needs_join") and not can_join(cfg): - return "the mesh's install line, which other machines join Lens with (in progress, docs/mesh.md)" + return "the mesh, which other machines join Lens over: turn it on in Settings › Network" + if view.get("unit_steps") == []: + return "steps its machines can run: a machine elsewhere runs only transcribe and diarize (docs/mesh.md)" return None @@ -975,8 +989,9 @@ def can_join(cfg): def bootstrap(db, cfg, g, name): """How a unit joins Lens: a worker started here gets its name and inherits the rest; a machine elsewhere gets the mesh's install line (`LENS_JOIN=… LENS_JOIN_TOKEN=… sh install.sh`) with a one-time join token good for an hour, - named for the unit, so the provider's user-data carries no long-lived secret: the database credential travels - inside the sealed join answer. {env, line, token_id, tag} (line and token_id None without the mesh).""" + named for the unit, so the provider's user-data carries no long-lived secret: the node asks for its database + user over the mesh once it has joined (docs/mesh.md, Worker nodes). {env, line, token_id, tag} (line and token_id + None without the mesh).""" boot = {"env": {"LENS_SCALING_UNIT": name}, "line": None, "token_id": None, "tag": conf(cfg)["tag"]} if not can_join(cfg): return boot @@ -1046,11 +1061,11 @@ def _group(self, cfg, g, now, did): launching = [n for n in units if n.get("state") == "launching"] view = group_view(db, g, now) may_launch = enabled(cfg) and view["enabled"] and not view["paused"] and view["approved"] and not capped - joinable = not drv.needs_join or can_join(cfg) + joinable = not drv.needs_join or (can_join(cfg) and view["unit_steps"] != []) lo, hi, desired = g.get("min") or 0, g.get("max") or 0, g.get("desired") or 0 if may_launch and g.get("rules"): want, state, said, new_min, new_max = evaluate( - g, metrics(db, cfg, (g.get("template") or {}).get("steps"), now), len(running), now, g.get("rule_state") + g, metrics(db, cfg, view["unit_steps"], now), len(running), now, g.get("rule_state") ) sets = {"rule_state": state, "last_eval": {"at": _iso(now), "said": said}} if new_min is not None or new_max is not None: diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 783df8996..e6382938d 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -994,6 +994,8 @@ def __init__(self, cfg, pool_size=None): "password": os.environ.get("SURREAL_PASS") or d.get("password") or "root", } self._target = (os.environ.get("SURREAL_NS") or d["namespace"], os.environ.get("SURREAL_DB") or d["database"]) + if os.environ.get("SURREAL_AUTH") == "database": # a user of this one database (a worker node's: docs/mesh.md) + self._creds.update(namespace=self._target[0], database=self._target[1]) size = 1 if self.embedded else max(1, int(pool_size or os.environ.get("SURREAL_POOL_SIZE") or d.get("pool_size") or 8)) self.fulltext = None # FULLTEXT (3.x), SEARCH (2.x) or None; set by connect() self.textindex = None # the SQLite full-text index (textindex.py) when search.engine is sqlite diff --git a/fastapi_backend/app/schemas/mesh.py b/fastapi_backend/app/schemas/mesh.py index cd1617bb5..541777ac6 100644 --- a/fastapi_backend/app/schemas/mesh.py +++ b/fastapi_backend/app/schemas/mesh.py @@ -129,6 +129,15 @@ class MeshNodeConfig(ResponseModel): crt: str | None = Field(None, description="the new certificate (renew)") +class MeshWorkerAccess(ResponseModel): + url: str = Field(description="the database through the mesh: ws://:8000") + namespace: str + database: str + user: str = Field(description="the node's own user of this database (removed with the node)") + password: str + steps: list[str] = Field(description="the steps a worker node runs unless LENS_WORKER_STEPS says otherwise") + + class MeshRenew(RequestModel): public_key: str = Field(max_length=500, description="its Nebula public key, the same or a new one") diff --git a/fastapi_backend/app/schemas/scaling.py b/fastapi_backend/app/schemas/scaling.py index db31ce956..380b17b36 100644 --- a/fastapi_backend/app/schemas/scaling.py +++ b/fastapi_backend/app/schemas/scaling.py @@ -174,6 +174,10 @@ class Group(ResponseModel): budget: GroupBudget | None = None units: UnitCounts waiting_for: str | None = Field(None, description="why it launches nothing now, when it doesn't") + unit_steps: list[str] | None = Field( + None, + description="the steps its units run, which its queue metrics count (None: every step); on a machine elsewhere, only a worker node's", + ) last_out_at: str | None = None last_in_at: str | None = None last_eval: dict[str, Any] | None = Field(None, description="what the rules said last: {at, said}") diff --git a/fastapi_backend/tests/api/test_mesh_node.py b/fastapi_backend/tests/api/test_mesh_node.py index 483a60ca1..847a3b05e 100644 --- a/fastapi_backend/tests/api/test_mesh_node.py +++ b/fastapi_backend/tests/api/test_mesh_node.py @@ -20,8 +20,31 @@ def folders(tmp_path): return tmp_path / "node", tmp_path / "mesh" +class Proc: + """`lens worker` as a worker node's agent starts it, without running it.""" + + started: list = [] + + def __init__(self, cmd, env): + self.cmd, self.env, self.returncode, self.stopped = cmd, env, None, False + Proc.started.append(self) + + def poll(self): + return self.returncode + + def terminate(self): + self.stopped, self.returncode = True, -15 + + def wait(self, timeout=None): + return self.returncode + + def kill(self): + self.terminate() + + def agent(client, folders, token, name="pi"): a = mesh_node.Agent(folders[0], folders[1], "http://testserver/", token, name, http=client) + a.worker.popen = Proc return a @@ -66,7 +89,8 @@ def test_a_machine_joins_and_follows_the_primary(client, db, cfg, nebula, server a.tick() me = client.get("/api/v1/mesh/nodes", headers=h).json()[1] assert me["last_seen"] and me["status"]["machine"]["cpus"] and me["status"]["version"] and a.error is None - assert me["status"]["nebula"] == "running" and me["status"]["error"] == "a peer is slow" and me["status"]["log"] == ["up"] + assert me["status"]["nebula"] == "running" and me["status"]["error"] == "a peer is slow" and me["status"]["log"][0] == "up" + assert "started lens worker Studio Mac -M2--" in me["status"]["log"][1] # a worker node's worker (test_mesh_workers.py) # a removed node is refused, and keeps asking: an undo brings it back client.delete(f"/api/v1/mesh/nodes/{me['id']}", headers=h) @@ -112,6 +136,7 @@ def test_a_machine_joins_and_follows_the_primary(client, db, cfg, nebula, server # started again (a restart of the machine), it carries on as the same node without its token again = mesh_node.Agent(node_dir, shared, None, None, "whatever", http=client) + again.worker.popen = Proc again.tick() assert again.node["node"]["id"] == me["id"] and not again.refused and again.error is None view = again.view() diff --git a/fastapi_backend/tests/api/test_mesh_workers.py b/fastapi_backend/tests/api/test_mesh_workers.py new file mode 100644 index 000000000..e9969410b --- /dev/null +++ b/fastapi_backend/tests/api/test_mesh_workers.py @@ -0,0 +1,192 @@ +"""Worker nodes (docs/mesh.md, Worker nodes): a machine that joined as a worker runs `lens worker` against this +install's database through the mesh, signed in as its own user of that one database, never the server's root. It +never gets the archive's secret: its jobs read their files from here over the mesh, plain, and a vault's never leave. +Removing the node removes its user; an undo makes it again with a new password. `lens worker` is played by Proc.""" + +from __future__ import annotations + +import json + +import pytest + +from app.domain import ingest, keyring, mesh, mesh_node, store +from tests.api.test_mesh import admin, nebula, server_db # noqa: F401 - fixtures +from tests.api.test_mesh_node import Proc, agent, due, running + + +def quiet(*a, **k): + pass + + +def users(db): + return set((db.one("INFO FOR DB") or {}).get("users") or {}) + + +def on_mesh(client, cfg, h, folders, role="worker", namespaces=()): + made = client.post("/api/v1/mesh/tokens", json={"role": role, "namespaces": list(namespaces)}, headers=h).json() + a = agent(client, folders, made["secret"], name="gpu box") + a.tick() + (folders[1] / "host.pub").write_text(mesh.keygen(cfg)[1]) + running(folders[1]) + a.tick() + return a + + +@pytest.fixture +def folders(tmp_path): + return tmp_path / "node", tmp_path / "mesh" + + +@pytest.fixture(autouse=True) +def no_procs(): + Proc.started.clear() + + +def test_a_worker_node_runs_lens_worker_as_its_own_user_of_the_database(client, db, cfg, nebula, server_db, folders, monkeypatch): # noqa: F811 + monkeypatch.setenv("LENS_JOIN_TOKEN", "lensjoin1.1.spent") + monkeypatch.setenv("ARCHIVE_SECRET_KEY", "the archive's own") + h = admin(client, db) + a = on_mesh(client, cfg, h, folders) + nid = a.node["node"]["id"] + assert f"lens_node_{nid}" in users(db) + + (w,) = Proc.started + assert w.cmd[1:] == ["worker", "--name", "gpu box", "--steps", "transcribe,diarize", "--no-schedule"] + assert w.env["SURREAL_URL"] == "ws://10.73.0.1:8000" and w.env["SURREAL_AUTH"] == "database" + assert w.env["SURREAL_USER"] == f"lens_node_{nid}" and len(w.env["SURREAL_PASS"]) > 30 + assert (w.env["SURREAL_NS"], w.env["SURREAL_DB"]) == tuple(db._target) and w.env["LENS_MESH_NODE"] == str(nid) + assert "LENS_JOIN_TOKEN" not in w.env and "ARCHIVE_SECRET_KEY" not in w.env # nothing it doesn't need + assert w.env["SURREAL_PASS"] not in json.dumps(a.node) # asked for, never kept on the node's disk + + # a minute later nothing changed: it keeps running + due(a) + a.tick() + assert Proc.started == [w] and not w.stopped + + # it stopped by itself: started again (after a short wait), and the Nodes page says so meanwhile + w.returncode = 1 + due(a) + a.tick() + assert len(Proc.started) == 1 and "exit 1" in a.worker.error + me = [n for n in client.get("/api/v1/mesh/nodes", headers=h).json() if n["id"] == nid][0] + assert "exit 1" in me["status"]["error"] and any("stopped (exit 1)" in x for x in me["status"]["log"]) + a.worker.not_before = 0.0 + due(a) + a.tick() + w2 = Proc.started[-1] + assert len(Proc.started) == 2 and w2.env["SURREAL_PASS"] == w.env["SURREAL_PASS"] and a.worker.error is None + + # removed: its worker stops and its user is gone; undone: a new password, and the worker starts with it + client.delete(f"/api/v1/mesh/nodes/{nid}", headers=h) + assert f"lens_node_{nid}" not in users(db) + due(a) + a.tick() + assert w2.stopped and a.removed + client.post(f"/api/v1/mesh/nodes/{nid}/restore", headers=h) + assert f"lens_node_{nid}" in users(db) + due(a) + a.tick() + w3 = Proc.started[-1] + assert len(Proc.started) == 3 and w3.env["SURREAL_PASS"] != w2.env["SURREAL_PASS"] + + # the agent stopping stops it too + import threading + + stop = threading.Event() + stop.set() + a.loop(stop) + assert w3.stopped + + +def test_a_scaling_unit_names_its_worker_and_its_steps(client, db, cfg, nebula, server_db, folders, monkeypatch): # noqa: F811 + monkeypatch.setenv("LENS_SCALING_UNIT", "hetzner-3") + monkeypatch.setenv("LENS_WORKER_STEPS", "transcribe") + on_mesh(client, cfg, admin(client, db), folders) + assert Proc.started[-1].cmd[1:6] == ["worker", "--name", "hetzner-3", "--steps", "transcribe"] + + +def test_a_node_runs_only_the_steps_a_node_can(client, db, cfg, nebula, server_db, folders, monkeypatch): # noqa: F811 + monkeypatch.setenv("LENS_WORKER_STEPS", "summarize,diarize") + a = on_mesh(client, cfg, admin(client, db), folders) + assert Proc.started[-1].cmd[5] == "diarize" # summarize needs the archive key, which a node never has + assert any("skipping summarize" in line for line in a.worker.log) + monkeypatch.setenv("LENS_WORKER_STEPS", "summarize") + a.worker.stop() + a.worker.proc, a.worker.key = None, None + n = len(Proc.started) + a.worker.sync() + assert len(Proc.started) == n and "none of LENS_WORKER_STEPS" in a.worker.error + + +def test_only_a_worker_node_gets_the_database(client, db, cfg, nebula, server_db, folders): # noqa: F811 + a = on_mesh(client, cfg, admin(client, db), folders, role="compute") + assert not Proc.started and a.worker.error is None # a compute node runs no worker, and never asks + r = client.get("/api/v1/mesh/node/worker", headers=a._headers()) + assert r.status_code == 403 + assert client.get("/api/v1/mesh/node/worker").status_code == 401 + + +def test_a_worker_on_an_embedded_database_is_told_why_it_waits(client, db, cfg, nebula, server_db, folders, monkeypatch): # noqa: F811 + a = on_mesh(client, cfg, admin(client, db), folders) + Proc.started[-1].returncode = 0 + monkeypatch.setattr(db, "url", "surrealkv:///data/surrealdb") + r = client.get("/api/v1/mesh/node/worker", headers=a._headers()) + assert r.status_code == 409 and "SurrealDB server" in r.json()["detail"] + + +def test_its_jobs_read_files_from_here_over_the_mesh(client, db, cfg, nebula, server_db, folders, folder, monkeypatch): # noqa: F811 + h = admin(client, db) + transcript = folder / "talk.txt" + transcript.write_text("Alice: hello there.\n") + audio = folder / "talk.mp3" + audio.write_bytes(b"ID3 not really audio" * 100) + rid = ingest.import_transcript(db, cfg, "pods", transcript, audio=str(audio), log=quiet) + a = on_mesh(client, cfg, h, folders) + + r = client.get(f"/api/v1/mesh/node/media/{rid}", headers=a._headers()) + assert r.status_code == 200 and r.content == audio.read_bytes() + assert client.get(f"/api/v1/mesh/node/media/{rid}", headers={**a._headers(), "If-None-Match": r.headers["etag"]}).status_code == 304 + assert client.get("/api/v1/mesh/node/media/999999", headers=a._headers()).status_code == 404 + + # in the node's `lens worker`, a job's file is that, kept until it changes + node_cfg = {**cfg, "data_dir": str(folder / "node-data")} + monkeypatch.setenv("LENS_NODE_DIR", str(folders[0])) + monkeypatch.setenv("LENS_MESH_NODE", str(a.node["node"]["id"])) + monkeypatch.setattr(mesh_node.httpx, "Client", lambda **k: client) + monkeypatch.setattr(client, "close", lambda: None, raising=False) + rec = db.one("SELECT record::id(id) AS id, path, remote, space FROM $r", r=store.R("recording", rid)) + rec["path"] = "/srv/lens/uploads/pods/1/talk.mp3" # where it is on the primary, not here + got = ingest.audio_path(db, node_cfg, rec) + assert got.endswith(f"/cache/mesh/{rid}.mp3") and open(got, "rb").read() == audio.read_bytes() + assert mesh_node.media(node_cfg, rec) == mesh_node.media(node_cfg, rec) + + # encrypted here, it comes plain; a vault's never comes; nor one in a namespace the node doesn't serve + keyring.encrypt_file(db, cfg, store.ns_id(db, "pods"), str(audio), force=True) + assert keyring.is_encrypted(str(audio)) + r = client.get(f"/api/v1/mesh/node/media/{rid}", headers=a._headers()) + assert r.status_code == 200 and r.content == b"ID3 not really audio" * 100 + db.q("UPSERT data_key SET vault = true WHERE space = $s", s=store.ns_id(db, "pods")) + assert client.get(f"/api/v1/mesh/node/media/{rid}", headers=a._headers()).status_code == 403 + db.q("UPSERT data_key SET vault = false WHERE space = $s", s=store.ns_id(db, "pods")) + store.ns_id(db, "calls") + client.patch(f"/api/v1/mesh/nodes/{a.node['node']['id']}", json={"namespaces": ["calls"]}, headers=h) + assert client.get(f"/api/v1/mesh/node/media/{rid}", headers=a._headers()).status_code == 403 + + +def test_a_node_serving_some_namespaces_takes_only_their_jobs(client, db, cfg, nebula, server_db, folders, folder, monkeypatch): # noqa: F811 + from app.domain import jobs + + h = admin(client, db) + rids = {} + for ns in ("pods", "calls"): + t = folder / f"{ns}.txt" + t.write_text("Alice: hello there.\n") + rids[ns] = ingest.import_transcript(db, cfg, ns, t, log=quiet) + jobs.enqueue(db, rids[ns], steps=[{"type": "analyze"}]) + a = on_mesh(client, cfg, h, folders, namespaces=["calls"]) + monkeypatch.setattr(jobs, "MESH_NODE", a.node["node"]["id"]) + got = jobs.claim(db, "gpu box", {"analyze"}) + assert got and got["space"] == store.ns_id(db, "calls") + assert jobs.claim(db, "gpu box", {"analyze"}) is None # the other is pods', which this node doesn't serve + monkeypatch.setattr(jobs, "MESH_NODE", None) + assert jobs.claim(db, "here", {"analyze"}) # this machine's worker takes it diff --git a/fastapi_backend/tests/api/test_scaling_clouds.py b/fastapi_backend/tests/api/test_scaling_clouds.py index 0a85ff84e..5cc969cc0 100644 --- a/fastapi_backend/tests/api/test_scaling_clouds.py +++ b/fastapi_backend/tests/api/test_scaling_clouds.py @@ -1,6 +1,6 @@ """Auto scaling on DigitalOcean and Hetzner Cloud (docs/scaling.md "Providers"): the drivers against their REST APIs -(an httpx MockTransport for each cloud), and a group on another machine waiting for the mesh's install line before it -launches anything.""" +(an httpx MockTransport for each cloud), and a group on another machine waiting for the mesh before it +launches anything, and counting only the jobs a worker node takes.""" from __future__ import annotations @@ -12,6 +12,7 @@ from app.domain import scaling, settings, store from app.domain import scaling_drivers as drivers +from app.domain.mesh import WORKER_STEPS from tests.helpers import login, make_user R = store.R @@ -136,7 +137,7 @@ def test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets( assert g["template"] == {"size": "s-1vcpu-2gb", "image": "ubuntu-24-04-x64", "steps": ["transcribe"]} assert tick(db, cfg)["launched"] == 0 and not do.machines # no way for a droplet to join yet: none is started (view,) = client.get("/api/v1/scaling", headers=h).json()["groups"] - assert "install line" in view["waiting_for"] + assert "Settings › Network" in view["waiting_for"] # the mesh is off mesh = Mesh() monkeypatch.setattr(scaling, "mesh_module", lambda: mesh) @@ -194,3 +195,27 @@ def test_the_cloud_drivers_are_listed_and_need_the_mesh(client, h): assert ds["digitalocean"]["needs_join"] is True and ds["hetzner"]["priced"] is True assert ds["local"]["needs_join"] is False assert ds["hetzner"]["fields"]["token"]["secret"] is True and ds["digitalocean"]["extra"] is None # nothing to install + + +def test_a_group_elsewhere_counts_only_the_jobs_a_node_takes(client, db, cfg, h, do, monkeypatch): + """A machine elsewhere is a worker node, which runs only mesh.WORKER_STEPS: a group never launches one for jobs + it would never take (docs/mesh.md, Worker nodes).""" + settings.save(db, cfg, "scaling", {"enabled": True}, "root@x.io") + cfg = settings.effective(db, cfg) + fake = Mesh() + fake.WORKER_STEPS = WORKER_STEPS + monkeypatch.setattr(scaling, "mesh_module", lambda: fake) + body = {"name": "DO", "driver": "digitalocean", "settings": {"region": "fra1"}, "secrets": {"token": "dop_v1_abc"}} + p = client.post("/api/v1/scaling/providers", headers=h, json=body).json() + rule = {"type": "step", "metric": "jobs_waiting", "above": 0, "minutes": 0, "add": 1} + made = {"provider": p["id"], "min": 0, "max": 2, "budget": 30, "rules": [rule], "cooldown_seconds": 0, "warmup_seconds": 0} + g = client.post("/api/v1/scaling/groups", headers=h, json={"name": "Any", **made}).json() + assert g["unit_steps"] == ["transcribe", "diarize"] and g["waiting_for"] is None + odd = client.post("/api/v1/scaling/groups", headers=h, json={"name": "Odd", **made, "template": {"steps": ["summarize"]}}).json() + assert odd["unit_steps"] == [] and "transcribe and diarize" in odd["waiting_for"] + + db.q("CREATE job CONTENT {status: 'queued', next_step: 'summarize', created_at: $t}", t=NOW.isoformat()) + assert tick(db, cfg)["launched"] == 0 and not do.machines # a summarize job: no node would take it + db.q("CREATE job CONTENT {status: 'queued', next_step: 'transcribe', created_at: $t}", t=NOW.isoformat()) + assert tick(db, cfg)["launched"] == 1 # the Any group, for the transcription; Odd still launches nothing + assert [len(scaling.nodes(db, x["id"])) for x in (g, odd)] == [1, 0] diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index d9fda920e..9684f7cb0 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -2461,6 +2461,7 @@ export type { MeshTokensData, MeshTokensResponse, MeshTokensResponses, + MeshWorkerAccess, MessageCreate, MessageDetail, MessageRecord, @@ -2535,9 +2536,16 @@ export type { NodeConfigData, NodeConfigResponse, NodeConfigResponses, + NodeMediaData, + NodeMediaError, + NodeMediaErrors, + NodeMediaResponses, NodeParam, NodeTrace, NodeType, + NodeWorkerData, + NodeWorkerResponse, + NodeWorkerResponses, Note, NoteBacklink, NoteCopy, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 8e8f158ed..bcd487f86 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -1258,6 +1258,11 @@ import type { NodeBeatResponses, NodeConfigData, NodeConfigResponses, + NodeMediaData, + NodeMediaErrors, + NodeMediaResponses, + NodeWorkerData, + NodeWorkerResponses, NotSameSpeakerData, NotSameSpeakerErrors, NotSameSpeakerResponses, @@ -13639,6 +13644,36 @@ export class Mesh { }); } + /** + * Node Worker + * + * What the calling worker node's `lens worker` signs in to the database with: its own user of it. 403 for a node + * of another role, 409 where the database is embedded. + */ + public static nodeWorker( + options?: Options, + ): RequestResult { + return (options?.client ?? client).get({ + url: "/api/v1/mesh/node/worker", + ...options, + }); + } + + /** + * Node Media + * + * A recording's file, plain, for a job on the calling worker node (a vault's never). Its ETag is its size and + * time: a node that has it sends If-None-Match and gets 304. + */ + public static nodeMedia( + options: Options, + ): RequestResult { + return (options.client ?? client).get({ + url: "/api/v1/mesh/node/media/{recording_id}", + ...options, + }); + } + /** * Node Beat * diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 19f8849ed..f407fe088 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -9442,6 +9442,12 @@ export type Group = { * why it launches nothing now, when it doesn't */ waiting_for?: string | null; + /** + * Unit Steps + * + * the steps its units run, which its queue metrics count (None: every step); on a machine elsewhere, only a worker node's + */ + unit_steps?: Array | null; /** * Last Out At */ @@ -13539,6 +13545,43 @@ export type MeshTokenCreated = { [key: string]: unknown; }; +/** + * MeshWorkerAccess + */ +export type MeshWorkerAccess = { + /** + * Url + * + * the database through the mesh: ws://:8000 + */ + url: string; + /** + * Namespace + */ + namespace: string; + /** + * Database + */ + database: string; + /** + * User + * + * the node's own user of this database (removed with the node) + */ + user: string; + /** + * Password + */ + password: string; + /** + * Steps + * + * the steps a worker node runs unless LENS_WORKER_STEPS says otherwise + */ + steps: Array; + [key: string]: unknown; +}; + /** * MessageCreate */ @@ -49980,6 +50023,50 @@ export type RenewNodeResponses = { export type RenewNodeResponse = RenewNodeResponses[keyof RenewNodeResponses]; +export type NodeWorkerData = { + body?: never; + path?: never; + query?: never; + url: "/api/v1/mesh/node/worker"; +}; + +export type NodeWorkerResponses = { + /** + * Successful Response + */ + 200: MeshWorkerAccess; +}; + +export type NodeWorkerResponse = NodeWorkerResponses[keyof NodeWorkerResponses]; + +export type NodeMediaData = { + body?: never; + path: { + /** + * Recording Id + */ + recording_id: number; + }; + query?: never; + url: "/api/v1/mesh/node/media/{recording_id}"; +}; + +export type NodeMediaErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type NodeMediaError = NodeMediaErrors[keyof NodeMediaErrors]; + +export type NodeMediaResponses = { + /** + * Successful Response + */ + 200: unknown; +}; + export type NodeBeatData = { body: MeshBeat; path?: never; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 6863d1fa2..6334a7b75 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -34902,6 +34902,64 @@ } } }, + "/api/v1/mesh/node/worker": { + "get": { + "tags": [ + "mesh" + ], + "summary": "Node Worker", + "description": "What the calling worker node's `lens worker` signs in to the database with: its own user of it. 403 for a node\nof another role, 409 where the database is embedded.", + "operationId": "node_worker", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MeshWorkerAccess" + } + } + } + } + } + } + }, + "/api/v1/mesh/node/media/{recording_id}": { + "get": { + "tags": [ + "mesh" + ], + "summary": "Node Media", + "description": "A recording's file, plain, for a job on the calling worker node (a vault's never). Its ETag is its size and\ntime: a node that has it sends If-None-Match and gets 304.", + "operationId": "node_media", + "parameters": [ + { + "name": "recording_id", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Recording Id" + } + } + ], + "responses": { + "200": { + "description": "Successful Response" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/mesh/node/beat": { "post": { "tags": [ @@ -51398,6 +51456,21 @@ "title": "Waiting For", "description": "why it launches nothing now, when it doesn't" }, + "unit_steps": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Unit Steps", + "description": "the steps its units run, which its queue metrics count (None: every step); on a machine elsewhere, only a worker node's" + }, "last_out_at": { "anyOf": [ { @@ -58242,6 +58315,51 @@ ], "title": "MeshTokenCreated" }, + "MeshWorkerAccess": { + "properties": { + "url": { + "type": "string", + "title": "Url", + "description": "the database through the mesh: ws://:8000" + }, + "namespace": { + "type": "string", + "title": "Namespace" + }, + "database": { + "type": "string", + "title": "Database" + }, + "user": { + "type": "string", + "title": "User", + "description": "the node's own user of this database (removed with the node)" + }, + "password": { + "type": "string", + "title": "Password" + }, + "steps": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Steps", + "description": "the steps a worker node runs unless LENS_WORKER_STEPS says otherwise" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "url", + "namespace", + "database", + "user", + "password", + "steps" + ], + "title": "MeshWorkerAccess" + }, "MessageCreate": { "properties": { "content": { diff --git a/threat-model.json b/threat-model.json index d10f1a4b5..3d89285b2 100644 --- a/threat-model.json +++ b/threat-model.json @@ -146,6 +146,7 @@ "SMB share (port 1445 on the host, 4445 in the worker)", "POST /api/v1/mesh/join (a machine joining the mesh)", "/api/v1/mesh/node/config, /node/renew, /node/beat (a node's own calls, with its node token)", + "GET /api/v1/mesh/node/worker, /node/media/{id} (a worker node's database user and its jobs' files, with its node token)", "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh", "UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the mesh container, while the mesh is on)" ], @@ -772,6 +773,15 @@ "provenance": "documented", "source": "docs/mesh.md \"After joining\"" }, + { + "entry_point": "GET /api/v1/mesh/node/worker, /node/media/{id} (a worker node's database user and its jobs' files, with its node token)", + "parameter": "a recording id; the node's role and namespaces", + "attacker_controllable": "yes", + "component": "optional", + "caller_must_enforce": "worker nodes only (403 for others); the database user is the node's own, of Lens's database only, never the server's root, removed with the node and made again with a new password on an undo; files are those of the namespaces the node serves, never a vault's, sent plain over the mesh; the archive's secret never leaves the primary", + "provenance": "documented", + "source": "docs/mesh.md \"Worker nodes\"" + }, { "entry_point": "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh", "parameter": "a node's role, name, namespaces, token hours and uses; the mesh's network, port, hosts, cipher", @@ -1531,7 +1541,7 @@ "property": "P-MESH", "violation_symptom": "info-leak, integrity-bypass", "severity_tier": "security", - "conditions": "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only", + "conditions": "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only; a worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads its namespaces' files over the mesh, never a vault's, without the archive's secret", "provenance": "documented", "source": "docs/mesh.md \"What keeps it safe\"" } diff --git a/threat-model.md b/threat-model.md index 1019d7795..12bca5689 100644 --- a/threat-model.md +++ b/threat-model.md @@ -335,6 +335,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `ask/NAME.want` in a microsandbox's `/data/.lens/programs` (LibreOffice and Chromium on first use) | a program's name | writable by the Lens user inside the sandbox | x-env-config | the start script, as root, installs only `libreoffice` or `chromium`, each a fixed list of signed Debian packages, and answers in `got/`, which only root writes; it reads nothing from the file but its name, and only deletes any other name or link | *(documented, docs/microsandbox.md "LibreOffice and Chromium")* | | `POST /api/v1/mesh/join` (a machine joining the mesh) | a join token's id, a nonce, the machine's name and Nebula public key, an HMAC of those keyed with the token's secret, facts about the machine | yes, anyone reaching the web app, once a token exists | x-credential, data | no sign-in: the token's secret never crosses the wire, the HMAC is checked in constant time, wrong proofs count towards a throttle per address (8 in 15 minutes), a token is spent on use (or after `uses`) and expires, and the answer (certificate, config, node token) is sealed with a key derived from the secret, so an eavesdropper on plain http:// learns nothing | *(documented, docs/mesh.md "Joining")* | | `/api/v1/mesh/node/config`, `/node/renew`, `/node/beat` (a node's own calls, with its node token) | a public key to certify; a status report: whether Nebula runs, its peers, its machine, its log lines | yes, a machine that joined (a removed one is refused) | x-credential, data, x-peer | the node token is 256 random bits kept only as a hash; a node gets its own config and a certificate for its own address and groups only, and can't widen them; reports are capped and kept as its status for admins; they arrive at the primary's mesh address, which the API's host check admits for these paths alone while the mesh is on | *(documented, docs/mesh.md "After joining")* | +| `GET /api/v1/mesh/node/worker`, `/node/media/{id}` (a worker node's database user and its jobs' files, with its node token) | a recording id; the node's role and namespaces | yes, a worker node that joined (a removed one is refused) | x-credential, data, x-peer | worker nodes only (403 for others); the database user (`lens_node_`) owns Lens's database and nothing else on the server, never root, is removed with the node and made again with a new password on an undo, and isn't kept on the node's disk; files are those of the namespaces the node serves, never a vault's, sent plain over the mesh and cached on the node for 6 hours; the archive's secret never leaves the primary | *(documented, docs/mesh.md "Worker nodes")* | | `POST /api/v1/mesh/tokens`, `PATCH` and `DELETE /api/v1/mesh/nodes/{id}`, `PUT /api/v1/settings/mesh` | a node's role, name, namespaces, token hours and uses; the mesh's network, port, hosts, cipher | admin only | x-env-config | admin is trusted; audited (`mesh.token.create`, `mesh.join`, `mesh.node.*`); the network can't change once machines have addresses in it | *(documented, docs/mesh.md "Settings")* | | UDP 4242 on the primary and ports 8000 and 8080 on its mesh address (the `mesh` container, while the mesh is on) | Nebula handshakes and tunnelled packets from anyone reaching the port; on the mesh, TCP to the database and the API | yes, anyone reaching UDP 4242; on the mesh, a machine with a certificate the mesh's CA signed | data, x-peer | Nebula drops what no certificate of the mesh's CA signs and anything on the blocklist; its firewall, rendered by Lens, lets only `lens-worker` reach 8000 and `lens-node` 8080; the relays listen on the mesh address only; the container is the only one holding NET_ADMIN, makes its own key pair (the private key stays in its own volume) and runs only Nebula's release with the checksum kept in Lens | *(documented, docs/mesh.md "On this machine")* | @@ -593,7 +594,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-BROWSER-GUARD | The Lens extension's side panel reads and acts only on sites the person allowed in it (and the browser granted), and opens an address only on such a site. It asks the person before a click, submit or Enter that the model marks consequential or whose control's words or posting form look so, and, once the answer has read text from outside Lens (`ai.untrusted_asks`), before opening or downloading an address or typing text the person's own message doesn't contain. It never types into password, card or file fields, never reports their values, and shows each action as it happens. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Letting it use your browser")* | Voided by: The consequential check is the model's flag plus a word list (`extensions/browser/src/core/act.js`): a control with no telling words that a misled model doesn't mark runs without asking; synthetic events reach only the page's own handlers. Removing a site from the options stops the side panel acting there, but leaves the browser's access to it for the clipper. Lens sends the ask (`confirm`) and the extension honours it: an extension older than this doesn't ask. | | P-SANDBOX | Code the assistant runs (`run_python`, `run_bash`; `sandbox.enabled`, off by default) runs only for people `sandbox.who` allows (editors and admins, or admins; never viewers), in a sandbox of that conversation's own on the OpenShell gateway, never in Lens's processes. The sandbox gets none of Lens's data, keys or database address: only the code and the files a run copies in, which must be attached to that conversation by that person (25 MB each, 100 MB a run). Its policy allows no egress with `sandbox.network: off`, only pypi.org, files.pythonhosted.org and registry.npmjs.org on 443 with `packages`, and those and the names in `sandbox.hosts` on 80 and 443 with `hosts`; every endpoint is held to public addresses (no private, loopback, link-local or reserved range), so a listed name that resolves to one is refused. A file leaves it only as one of the person's held uploads (25 MB) placed in a namespace by an `import_files` approval, or as a download (`share_sandbox_file`, 25 MB, 20 a conversation) kept encrypted with the conversation, served only to its person as an attachment with `nosniff`, and deleted with the conversation or their account. A sandbox stops after `sandbox.idle_minutes` idle and is deleted after `sandbox.keep_hours` unused or with its conversation. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Code sandbox")* | Voided by: isolation inside the sandbox is OpenShell's (§1.9); `landlock.compatibility: best_effort` runs without Landlock on a kernel that lacks it — `domain/sandbox.py`; whoever the gateway lets in can run sandboxes of their own (in the `sandbox` profile, a holder of its client certificate); a file shared to download is readable by whoever holds `secret.key` or `ARCHIVE_SECRET_KEY` (Q14); what the code prints reaches the model as third-party text (N-PROMPT). | | P-RESEARCH | A research plan (`domain/research.py`) runs only once the person who asked for it approves it: signed in, or with a "yes" in their chat room from someone in `bridge.users`; an API key, an app or an agent (MCP, the assistant) can only decline it, unless an admin turns on `research.agents_skip_review`. The run acts as that person with their access at run time and stops if their account is disabled. It is offered only the read tools of the sources its plan names (never a tool that changes anything), makes at most the plan's tool calls (`research.max_tool_calls`, 40 by default), stops looking things up after `research.max_minutes` and writes up what it found, and checks the budgets of its namespaces and `all` before each stage, waiting or stopping as they say. Its web searches and page reads, and code runs with the network when the plan names the code sandbox, don't ask again after outside text, since the plan approved them. A person has one run going at a time. One process carries a run out at a time (a claim on `research_run:` with a heartbeat every 10 seconds, taken over only after two minutes of silence). Every call it makes is in the activity ledger for the conversation and `research:`. | integrity-bypass | correctness-only | *(documented, docs/assistant.md "Deep research")* | Voided by: `research.agents_skip_review: true` lets an agent's plan run without review — `domain/research.py` `agents_may_approve`; a run's web queries can carry archive text it read, within its plan's tool calls (the misled model of N-PROMPT); a "yes" in a room is the room's people's (N-BRIDGE). | -| P-MESH | The mesh between Lens's machines (`mesh.enabled`, off until an admin adds a node) admits only machines that joined with a token an admin made, which is spent on use and expires; the token never crosses the wire (an HMAC proves it, the answer is sealed with a key derived from it), each machine's certificate names only its own address, role and namespaces, and Nebula's firewall, rendered by Lens, lets worker nodes reach the database port and every node the API port on the primary, and the primary a node's agent port, and nothing else. Removing a node puts every certificate it was given on the blocklist every machine's config carries, and its node token stops working at once. Nodes still sign in to Lens as any client does: the mesh is a transport, not an identity. On the primary Nebula runs in the `mesh` container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the checksum kept in Lens; its relays listen on the mesh address only. A node runs `lens node` and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only. | info-leak, integrity-bypass | security-critical | *(documented, docs/mesh.md "What keeps it safe")* | Voided by: the join request goes to whatever address is put in the install line, so a wrong address gives a stranger the token's id (not its secret); a worker node that reaches the database port holds the database credential (a trusted peer, Q9); the lighthouse must be reachable on UDP `mesh.port`; Nebula's own isolation is upstream's (§1.9) — `domain/mesh.py`. | +| P-MESH | The mesh between Lens's machines (`mesh.enabled`, off until an admin adds a node) admits only machines that joined with a token an admin made, which is spent on use and expires; the token never crosses the wire (an HMAC proves it, the answer is sealed with a key derived from it), each machine's certificate names only its own address, role and namespaces, and Nebula's firewall, rendered by Lens, lets worker nodes reach the database port and every node the API port on the primary, and the primary a node's agent port, and nothing else. Removing a node puts every certificate it was given on the blocklist every machine's config carries, and its node token stops working at once. Nodes still sign in to Lens as any client does: the mesh is a transport, not an identity. On the primary Nebula runs in the `mesh` container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the checksum kept in Lens; its relays listen on the mesh address only. A node runs `lens node` and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only. A worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads the files of the namespaces it serves over the mesh, never a vault's, without the archive's secret. | info-leak, integrity-bypass | security-critical | *(documented, docs/mesh.md "What keeps it safe")* | Voided by: the join request goes to whatever address is put in the install line, so a wrong address gives a stranger the token's id (not its secret); a worker node owns Lens's database and keeps plain copies of the files it worked on for a few hours (a trusted peer, Q9); the lighthouse must be reachable on UDP `mesh.port`; Nebula's own isolation is upstream's (§1.9) — `domain/mesh.py`. | ### Worked routing examples diff --git a/threat-model.yaml b/threat-model.yaml index 1cfd9744d..80778ec12 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -572,6 +572,15 @@ entry_points: control_kinds: [x-credential, data, x-peer] caller_must_enforce: "the node token is 256 random bits kept only as a hash; a node gets its own config and a certificate for its own address and groups only; reports are capped; they arrive at the primary's mesh address, which the API's host check admits for these paths alone while the mesh is on" provenance: {kind: documented, source: 'docs/mesh.md "After joining"'} + - id: "GET /api/v1/mesh/node/worker, /node/media/{id} (a worker node's database user and its jobs' files, with its node token)" + component: optional + parameters: + - name: "a recording id; the node's role and namespaces" + attacker_controllable: true + x-condition: "a worker node that joined (a removed one is refused)" + control_kinds: [x-credential, data, x-peer] + caller_must_enforce: "worker nodes only (403 for others); the database user is the node's own, of Lens's database only, never the server's root, removed with the node and made again with a new password on an undo; files are those of the namespaces the node serves, never a vault's, sent plain over the mesh; the archive's secret never leaves the primary" + provenance: {kind: documented, source: 'docs/mesh.md "Worker nodes"'} - id: "POST /api/v1/mesh/tokens, PATCH and DELETE /api/v1/mesh/nodes/{id}, PUT /api/v1/settings/mesh" component: optional parameters: @@ -1974,9 +1983,9 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only" + conditions: "the mesh (mesh.enabled, off until an admin adds a node) admits only machines that joined with a token an admin made, spent on use and expiring, which never crosses the wire (HMAC proof; answer sealed with a key derived from it); each machine's certificate names only its own address, role and namespaces; the rendered firewall opens the database port to worker nodes and the API port to every node on the primary, a node's agent port to the primary only; removing a node blocklists every certificate it was given on every machine and stops its node token; nodes still sign in to Lens as any client does; on the primary Nebula runs in the mesh container, the only one holding NET_ADMIN, with a key pair it makes itself and Nebula's release checked against the kept checksum; its relays listen on the mesh address only; a node runs lens node and Nebula in two containers with nothing published, its node token and agent key in its own volume (0600), the join token in the first start's environment only; a worker node signs in to the database as its own user of Lens's database (never root, removed with the node) and reads its namespaces' files over the mesh, never a vault's, without the archive's secret" violation_symptoms: [info-leak, integrity-bypass] - x-voided-by: "a wrong address in the install line gives a stranger the token's id (not its secret); a worker node holds the database credential (trusted peer, Q9); the lighthouse must be reachable on UDP mesh.port; Nebula's own isolation is upstream's (domain/mesh.py)" + x-voided-by: "a wrong address in the install line gives a stranger the token's id (not its secret); a worker node owns Lens's database and keeps plain copies of its files for a few hours (trusted peer, Q9); the lighthouse must be reachable on UDP mesh.port; Nebula's own isolation is upstream's (domain/mesh.py)" provenance: {kind: documented, source: 'docs/mesh.md "What keeps it safe"'} - id: P-SANDBOX kind: integrity From 453211eeef4415c09766decdfdcd4b9fd09e7fec Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 12:25:54 +0000 Subject: [PATCH 2/3] docs(readme): the mesh's In progress line links its open pull request Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 56453f6ca..64185bf5c 100644 --- a/README.md +++ b/README.md @@ -269,7 +269,7 @@ These are open pull requests, not yet part of Lens: ([#406](https://github.com/adeelahmad/lense/pull/406)). - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)). - **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates - for ([#591](https://github.com/adeelahmad/lense/pull/591)). + for ([#595](https://github.com/adeelahmad/lense/pull/595)). ## Planned From 33a5bd4b78c7327a85f8a51760d83447c09c704c Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 12:32:48 +0000 Subject: [PATCH 3/3] feat(mesh): auto scaling may launch onto worker nodes now that they run lens worker Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GTrHPwWHtEvFJFq3M1YpH4 --- fastapi_backend/app/domain/mesh.py | 1 + 1 file changed, 1 insertion(+) diff --git a/fastapi_backend/app/domain/mesh.py b/fastapi_backend/app/domain/mesh.py index 4a6df1a5d..d767bbfe9 100644 --- a/fastapi_backend/app/domain/mesh.py +++ b/fastapi_backend/app/domain/mesh.py @@ -760,6 +760,7 @@ def beat(db, row, status): # transcribing and telling voices apart with models on that machine (WORKER_STEPS). Removing the node removes its # user; an undo makes it again with a new password. WORKER_STEPS = ("transcribe", "diarize") +WORKER_NODES = True # a worker node runs `lens worker`, named for its scaling unit: auto scaling launches on it def _db_user(nid):