diff --git a/README.md b/README.md index d21f9b448..4778bd810 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today; - cloud, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); + DigitalOcean and Hetzner Cloud providers are in progress (built, launching once the mesh's install line lands), and + AWS, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out)) - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine, @@ -253,9 +254,9 @@ Next, and not in a pull request yet; each links to where it is tracked: topics, recordings and the other things that have pages ([notes](docs/notes.md)). - **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models ([local models](docs/local-models.md)). -- **Auto scaling:** providers for AWS, DigitalOcean, Hetzner, Proxmox and Linux hosts over SSH, joining over the - Nebula mesh, model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in - Settings ([auto scaling](docs/scaling.md)). +- **Auto scaling:** providers for AWS, Proxmox and Linux hosts over SSH, joining over the Nebula mesh, model hosts + (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings + ([auto scaling](docs/scaling.md)). - **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the activity ledger ([speech providers](docs/speech-providers.md)). - **Storage:** uploads and assets' media in the file store, and a storage key to download diff --git a/docs/requirements.md b/docs/requirements.md index 77b3b7c57..c2be8084d 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -216,6 +216,7 @@ request, to another check that proves the same thing; loosening a requirement or | Each unit's cost goes into the activity ledger as an estimate, and a group stops at its budget's cap | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_at_its_budgets_cap_a_group_stops` | | Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` | | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` | +| DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` | | Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` | ## Keep it up to date and backed up diff --git a/docs/scaling.md b/docs/scaling.md index 4ada0de54..39759320f 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -8,9 +8,11 @@ launched. Everything it does is in the [activity ledger](activity.md), and its c Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools, Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that -runs more `lens worker` processes where Lens runs. Planned: providers for AWS, DigitalOcean, Hetzner, Proxmox and any -Linux host over SSH, joining Lens over its Nebula mesh network; model hosts (RunPod, Lambda, Hugging Face, SageMaker, -Bedrock) for models; and making providers and groups in Settings. +runs more `lens worker` processes where Lens runs. In progress: **DigitalOcean** and **Hetzner Cloud** providers are +built (credentials checked, list prices, launching, watching and deleting machines), and launch once the mesh's +install line, which another machine joins Lens with, lands ([the mesh](mesh.md)). Planned: providers for AWS, Proxmox +and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making +providers and groups in Settings. Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms signed in to the web app. Agents can propose and manage groups, but never approve one. @@ -53,10 +55,24 @@ carries none of them. | Driver | Units | Costs | Template | | --- | --- | --- | --- | | `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) | +| `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` | +| `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` | A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where it runs. +### Machines elsewhere + +A cloud provider's unit is a new machine. Its first boot runs a script (user-data) that sets `LENS_SCALING_UNIT` (its +name) and `LENS_WORKER_STEPS` (the template's steps), then the mesh's install line with a join token made for that +unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) and starts `lens worker`; the +database credential travels inside the sealed join answer, so the user-data holds no long-lived secret. Each machine +carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=`), so Lens's machines can be told +apart. Letting a unit go deletes its machine; a machine switched off at the provider counts as gone and is deleted. + +Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches +nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing. + ## Groups ```http @@ -145,5 +161,7 @@ Turning scaling on (`change_settings scaling`) always asks. ## Refine later -Providers beyond This machine; spot and preemptible instances; warm pools; predictive scaling; health checks beyond +Providers for Proxmox, SSH hosts and model hosts; the AWS provider (in its own pull request); revoking a unit's mesh +certificate when it goes; reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro +rate; spot and preemptible instances; warm pools; predictive scaling; health checks beyond joining; prices fetched from providers' pricing APIs. diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py index 03aac825e..25da0e33a 100644 --- a/fastapi_backend/app/domain/scaling.py +++ b/fastapi_backend/app/domain/scaling.py @@ -44,6 +44,7 @@ import time from . import activity, budgets, jobs, schedule, settings, store +from . import scaling_clouds # noqa: F401 - registers the cloud drivers from . import scaling_drivers as drivers from .store import R @@ -539,6 +540,7 @@ def group_view(db, g, now=None): st = budgets.status(db, _budget_ref(g["id"]), b, now) if b else None approval = g.get("approval") t = terms({**g, "budget": b.get("usd") if b else None}) # the budget lives in the budget table, not on the row + drv = drivers.DRIVERS.get((db.one("SELECT driver FROM $r", r=R("scale_provider", int(g["provider"]))) or {}).get("driver")) out = { "id": g["id"], "name": g.get("name"), @@ -560,6 +562,7 @@ def group_view(db, g, now=None): "approval": approval, "approved": bool(approval) and not widens(approval.get("terms") or {}, t), "terms": t, + "needs_join": bool(drv and drv.needs_join), "budget": {"usd": b.get("usd"), "period": b.get("period"), "state": st["state"], "spent": st["spent"], "left": st["left"]} if b else None, @@ -593,6 +596,8 @@ def waiting_for(view, cfg=None): return "a person's approval of its terms, in Settings › Auto scaling" if view.get("capped"): return f"its budget: {view['capped']}" + if cfg is not None and view.get("needs_join") and not can_join(cfg): + return "the mesh's install line, which other machines join Lens with (in progress, docs/mesh.md)" return None @@ -951,7 +956,7 @@ def evaluate(g, m, units, now, state=None): # ---------- the scaler ---------- def mesh_module(): - """The Nebula mesh (mesh.py, docs/network.md) when this Lens has it, else None.""" + """The Nebula mesh (mesh.py, docs/mesh.md) when this Lens has it, else None.""" try: from . import mesh except ImportError: @@ -959,18 +964,24 @@ def mesh_module(): return mesh +def can_join(cfg): + """Whether another machine can join Lens now: the mesh is on and has the install line machines join with.""" + mesh = mesh_module() + if mesh is None or not (getattr(mesh, "conf", lambda c: {})(cfg) or {}).get("enabled"): + return False + return bool(getattr(mesh, "make_token", None) and getattr(mesh, "join_line", None)) + + def bootstrap(db, cfg, g, name): """How a unit joins Lens: a worker started here gets its name and inherits the rest; a machine elsewhere gets the mesh's install line (`LENS_JOIN=… LENS_JOIN_TOKEN=… sh install.sh`) with a one-time join token good for an hour, named for the unit, so the provider's user-data carries no long-lived secret: the database credential travels - inside the sealed join answer. {env, line, token_id} (line and token_id None without the mesh).""" - boot = {"env": {"LENS_SCALING_UNIT": name}, "line": None, "token_id": None} - mesh = mesh_module() - if mesh is None or not (getattr(mesh, "conf", lambda c: {})(cfg) or {}).get("enabled"): - return boot - make, line = getattr(mesh, "make_token", None), getattr(mesh, "join_line", None) - if not (make and line): + inside the sealed join answer. {env, line, token_id, tag} (line and token_id None without the mesh).""" + boot = {"env": {"LENS_SCALING_UNIT": name}, "line": None, "token_id": None, "tag": conf(cfg)["tag"]} + if not can_join(cfg): return boot + mesh = mesh_module() + make, line = mesh.make_token, mesh.join_line row, token = make(db, cfg, by="scaling", role="worker", name=name, hours=1, uses=1) boot["line"], boot["token_id"] = line(cfg, token), (row or {}).get("id") return boot @@ -1035,6 +1046,7 @@ def _group(self, cfg, g, now, did): launching = [n for n in units if n.get("state") == "launching"] view = group_view(db, g, now) may_launch = enabled(cfg) and view["enabled"] and not view["paused"] and view["approved"] and not capped + joinable = not drv.needs_join or can_join(cfg) lo, hi, desired = g.get("min") or 0, g.get("max") or 0, g.get("desired") or 0 if may_launch and g.get("rules"): want, state, said, new_min, new_max = evaluate( @@ -1057,7 +1069,7 @@ def _group(self, cfg, g, now, did): target = 0 elif capped: target = lo if g.get("at_cap") == "min" else 0 - elif not view["approved"] or view["paused"]: + elif not view["approved"] or view["paused"] or not joinable: target = min(target, len(running) + len(launching)) # nothing new; what runs stays have = len(running) + len(launching) if target > have: diff --git a/fastapi_backend/app/domain/scaling_clouds.py b/fastapi_backend/app/domain/scaling_clouds.py new file mode 100644 index 000000000..f655b9c02 --- /dev/null +++ b/fastapi_backend/app/domain/scaling_clouds.py @@ -0,0 +1,215 @@ +"""Cloud providers auto scaling launches machines on over their REST APIs (scaling_drivers.py, docs/scaling.md): +DigitalOcean and Hetzner Cloud. Each unit is a new machine whose first boot runs the mesh's install line +(scaling_drivers.user_data), which joins it to Lens and starts `lens worker`; terminating a unit deletes the machine. +Every machine carries Lens's tag (`scaling.tag`), so `listing` finds the ones Lens made. + +They call the APIs with httpx, which Lens already has, so a provider of these kinds installs nothing. +""" + +from __future__ import annotations + +import httpx + +from . import jobs +from .scaling_drivers import Driver, DriverError, _field, register, user_data + +TIMEOUT = 30.0 + + +def _steps(): + return _field( + "Steps", + "list", + help="the steps these workers run (transcribe, analyze, embed...); none: every step", + options=sorted(jobs.STEPS), + ) + + +class Rest(Driver): + """A cloud with a JSON API and a bearer token.""" + + base = "" + needs_join = True + transport: httpx.BaseTransport | None = None # tests hand in an httpx.MockTransport + + def _client(self, p): + return httpx.Client( + base_url=self.base, + headers={"Authorization": f"Bearer {p['secrets'].get('token') or ''}", "Accept": "application/json"}, + timeout=TIMEOUT, + transport=self.transport, + ) + + def _error(self, r): + """What the cloud said, in its own words.""" + return f"{self.label} said {r.status_code}" + + def call(self, p, method, path, ok404=False, **kw): + """The JSON answer; None for a 404 when `ok404` (the thing is already gone). DriverError otherwise.""" + try: + with self._client(p) as c: + r = c.request(method, path, **kw) + except httpx.HTTPError as e: + raise DriverError(f"couldn't reach {self.label}: {type(e).__name__}") from None + if r.status_code == 404 and ok404: + return None + if r.status_code >= 400: + raise DriverError(self._error(r)) + return r.json() if r.content else {} + + +# ---------- DigitalOcean ---------- +class DigitalOcean(Rest): + id = "digitalocean" + label = "DigitalOcean" + about = "Droplets that join Lens over the mesh and run Lens workers. Billed by the hour by DigitalOcean." + base = "https://api.digitalocean.com/v2" + fields = { + "region": _field("Region", required=True, default="nyc3", help="where droplets start: nyc3, fra1, sgp1..."), + "token": _field("API token", required=True, secret=True, help="a personal access token with write scope"), + "ssh_keys": _field("SSH keys", "list", help="the ids or fingerprints of SSH keys to put on each droplet"), + "vpc_uuid": _field("VPC", help="the VPC droplets join; none: the region's default"), + } + template = { + "size": _field("Size", required=True, default="s-1vcpu-2gb", help="a droplet size slug"), + "image": _field("Image", required=True, default="ubuntu-24-04-x64", help="an image slug or id"), + "steps": _steps(), + } + + def _error(self, r): + try: + said = r.json().get("message") + except ValueError: + said = None + return f"DigitalOcean said {r.status_code}" + (f": {said}" if said else "") + + def check(self, p): + regions = self.call(p, "GET", "/regions", params={"per_page": 200}).get("regions") or [] + names = {x.get("slug") for x in regions if x.get("available")} + region = p["settings"].get("region") + if region and names and region not in names: + raise DriverError(f"DigitalOcean has no region {region} open; it has {', '.join(sorted(names))}") + return {"regions": len(names)} + + def price(self, p, template): + for s in self.call(p, "GET", "/sizes", params={"per_page": 200}).get("sizes") or []: + if s.get("slug") == template.get("size"): + return float(s["price_hourly"]) if s.get("price_hourly") is not None else None + return None + + def launch(self, p, template, name, boot): + body = { + "name": name, + "region": p["settings"]["region"], + "size": template["size"], + "image": template["image"], + "user_data": user_data(boot, template), + "tags": [boot["tag"]], + } + if p["settings"].get("ssh_keys"): + body["ssh_keys"] = p["settings"]["ssh_keys"] + if p["settings"].get("vpc_uuid"): + body["vpc_uuid"] = p["settings"]["vpc_uuid"] + d = self.call(p, "POST", "/droplets", json=body).get("droplet") or {} + if d.get("id") is None: + raise DriverError("DigitalOcean made no droplet") + return {"id": d["id"]} + + def describe(self, p, node): + d = (self.call(p, "GET", f"/droplets/{node['external_id']}", ok404=True) or {}).get("droplet") + if not d or d.get("status") in ("off", "archive"): + return {"state": "gone"} + ip = next((n.get("ip_address") for n in (d.get("networks") or {}).get("v4") or [] if n.get("type") == "public"), None) + return {"state": "running" if d.get("status") == "active" else "pending", "address": ip} + + def terminate(self, p, node): + if node.get("external_id"): + self.call(p, "DELETE", f"/droplets/{node['external_id']}", ok404=True) + + def listing(self, p, tag): + got = self.call(p, "GET", "/droplets", params={"tag_name": tag, "per_page": 200}).get("droplets") or [] + return [{"id": str(d["id"]), "name": d.get("name")} for d in got] + + +# ---------- Hetzner Cloud ---------- +class Hetzner(Rest): + id = "hetzner" + label = "Hetzner Cloud" + about = "Cloud servers that join Lens over the mesh and run Lens workers. Billed by the hour by Hetzner, in euros." + base = "https://api.hetzner.cloud/v1" + fields = { + "location": _field("Location", required=True, default="fsn1", help="where servers start: fsn1, nbg1, hel1, ash..."), + "token": _field("API token", required=True, secret=True, help="a project's API token with read and write"), + "ssh_keys": _field("SSH keys", "list", help="the names or ids of the project's SSH keys to put on each server"), + "usd_per_eur": _field( + "Dollars per euro", + "number", + default=1.1, + help="turns Hetzner's euro prices into the dollars budgets count; an estimate, so set your rate", + ), + } + template = { + "server_type": _field("Server type", required=True, default="cx22", help="cx22, cpx31, cax21..."), + "image": _field("Image", required=True, default="ubuntu-24.04"), + "steps": _steps(), + } + + def _error(self, r): + try: + said = (r.json().get("error") or {}).get("message") + except (ValueError, AttributeError): + said = None + return f"Hetzner said {r.status_code}" + (f": {said}" if said else "") + + def check(self, p): + names = {x.get("name") for x in self.call(p, "GET", "/locations").get("locations") or []} + loc = p["settings"].get("location") + if loc and names and loc not in names: + raise DriverError(f"Hetzner has no location {loc}; it has {', '.join(sorted(names))}") + return {"locations": len(names)} + + def price(self, p, template): + types = self.call(p, "GET", "/server_types", params={"name": template.get("server_type")}).get("server_types") or [] + loc = p["settings"].get("location") + for t in types: + for pr in t.get("prices") or []: + if pr.get("location") == loc: + eur = (pr.get("price_hourly") or {}).get("gross") + return round(float(eur) * float(p["settings"].get("usd_per_eur") or 1.1), 6) if eur is not None else None + return None + + def launch(self, p, template, name, boot): + body = { + "name": name, + "server_type": template["server_type"], + "image": template["image"], + "location": p["settings"]["location"], + "user_data": user_data(boot, template), + "labels": {"lens": boot["tag"]}, + "start_after_create": True, + } + if p["settings"].get("ssh_keys"): + body["ssh_keys"] = p["settings"]["ssh_keys"] + s = self.call(p, "POST", "/servers", json=body).get("server") or {} + if s.get("id") is None: + raise DriverError("Hetzner made no server") + return {"id": s["id"], "address": ((s.get("public_net") or {}).get("ipv4") or {}).get("ip")} + + def describe(self, p, node): + s = (self.call(p, "GET", f"/servers/{node['external_id']}", ok404=True) or {}).get("server") + if not s or s.get("status") in ("off", "stopping", "deleting"): + return {"state": "gone"} + ip = ((s.get("public_net") or {}).get("ipv4") or {}).get("ip") + return {"state": "running" if s.get("status") == "running" else "pending", "address": ip} + + def terminate(self, p, node): + if node.get("external_id"): + self.call(p, "DELETE", f"/servers/{node['external_id']}", ok404=True) + + def listing(self, p, tag): + got = self.call(p, "GET", "/servers", params={"label_selector": f"lens=={tag}", "per_page": 50}).get("servers") or [] + return [{"id": str(s["id"]), "name": s.get("name")} for s in got] + + +register(DigitalOcean()) +register(Hetzner()) diff --git a/fastapi_backend/app/domain/scaling_drivers.py b/fastapi_backend/app/domain/scaling_drivers.py index 65209530d..e9e1835ed 100644 --- a/fastapi_backend/app/domain/scaling_drivers.py +++ b/fastapi_backend/app/domain/scaling_drivers.py @@ -23,6 +23,7 @@ from __future__ import annotations import os +import shlex import signal import socket import subprocess @@ -57,6 +58,7 @@ class Driver: label = "" about = "" priced = True # it charges money: its groups need a budget + needs_join = False # its units are other machines, which join over the mesh's install line (scaling.bootstrap) extra = None # the optional dependency (pyproject.toml) it needs, fetched when a provider of it is saved modules = () # what that extra installs, to see whether it's here license = None @@ -79,6 +81,7 @@ def spec(self): "label": self.label, "about": self.about, "priced": self.priced, + "needs_join": self.needs_join, "extra": self.extra, "license": self.license, "ready": self.ready(), @@ -153,6 +156,16 @@ def check_template(drv, given): return _values(drv.template, given, f"a {drv.label} unit") +def user_data(boot, template): + """The script a new machine runs at its first boot (cloud-init runs a user-data that starts with `#!`): the unit's + name and steps as environment, then the mesh's install line, which joins Lens and starts `lens worker`.""" + env = {**(boot.get("env") or {})} + if template.get("steps"): + env["LENS_WORKER_STEPS"] = ",".join(template["steps"]) + lines = ["#!/bin/sh", "set -e", *(f"export {k}={shlex.quote(str(v))}" for k, v in sorted(env.items())), boot["line"]] + return "\n".join(lines) + "\n" + + # ---------- this machine ---------- def _die_with_parent(): """In the child, before it runs: be stopped when the process that started it ends (Linux), so a worker this diff --git a/fastapi_backend/tests/api/test_scaling_clouds.py b/fastapi_backend/tests/api/test_scaling_clouds.py new file mode 100644 index 000000000..0a85ff84e --- /dev/null +++ b/fastapi_backend/tests/api/test_scaling_clouds.py @@ -0,0 +1,196 @@ +"""Auto scaling on DigitalOcean and Hetzner Cloud (docs/scaling.md "Providers"): the drivers against their REST APIs +(an httpx MockTransport for each cloud), and a group on another machine waiting for the mesh's install line before it +launches anything.""" + +from __future__ import annotations + +import datetime as dt +import json + +import httpx +import pytest + +from app.domain import scaling, settings, store +from app.domain import scaling_drivers as drivers +from tests.helpers import login, make_user + +R = store.R +NOW = dt.datetime(2026, 10, 18, 12, 0, tzinfo=dt.UTC) + + +class Cloud: + """One cloud's API in memory: what it was asked, and its machines.""" + + def __init__(self): + self.calls, self.machines, self.next = [], {}, 100 + + def __call__(self, request): + body = json.loads(request.content) if request.content else None + self.calls.append((request.method, request.url.path, dict(request.url.params), body, request.headers.get("authorization"))) + return self.answer(request.method, request.url.path, dict(request.url.params), body) + + +class DO(Cloud): + def answer(self, method, path, params, body): + if path == "/v2/regions": + return httpx.Response(200, json={"regions": [{"slug": "nyc3", "available": True}, {"slug": "fra1", "available": True}]}) + if path == "/v2/sizes": + return httpx.Response(200, json={"sizes": [{"slug": "s-1vcpu-2gb", "price_hourly": 0.01786}]}) + if path == "/v2/droplets" and method == "POST": + self.next += 1 + self.machines[self.next] = {"id": self.next, "name": body["name"], "status": "new", "tags": body["tags"], "body": body} + return httpx.Response(202, json={"droplet": self.machines[self.next]}) + if path == "/v2/droplets": + tag = params.get("tag_name") + return httpx.Response(200, json={"droplets": [m for m in self.machines.values() if tag in m["tags"]]}) + did = int(path.rsplit("/", 1)[1]) + if did not in self.machines: + return httpx.Response(404, json={"id": "not_found", "message": "The resource you requested could not be found."}) + if method == "DELETE": + del self.machines[did] + return httpx.Response(204) + return httpx.Response(200, json={"droplet": self.machines[did]}) + + +class Hz(Cloud): + def answer(self, method, path, params, body): + if path == "/v1/locations": + return httpx.Response(200, json={"locations": [{"name": "fsn1"}, {"name": "hel1"}]}) + if path == "/v1/server_types": + prices = [{"location": "fsn1", "price_hourly": {"net": "0.0060", "gross": "0.0071"}}] + return httpx.Response(200, json={"server_types": [{"name": params.get("name"), "prices": prices}]}) + if path == "/v1/servers" and method == "POST": + self.next += 1 + self.machines[self.next] = {"id": self.next, "name": body["name"], "status": "initializing", "labels": body["labels"]} + self.machines[self.next]["public_net"] = {"ipv4": {"ip": "203.0.113.7"}} + return httpx.Response(201, json={"server": self.machines[self.next]}) + if path == "/v1/servers": + return httpx.Response(200, json={"servers": list(self.machines.values())}) + sid = int(path.rsplit("/", 1)[1]) + if sid not in self.machines: + return httpx.Response(404, json={"error": {"code": "not_found", "message": "server not found"}}) + if method == "DELETE": + del self.machines[sid] + return httpx.Response(200, json={"action": {}}) + return httpx.Response(200, json={"server": self.machines[sid]}) + + +@pytest.fixture +def do(monkeypatch): + cloud = DO() + monkeypatch.setattr(drivers.get("digitalocean"), "transport", httpx.MockTransport(cloud)) + return cloud + + +@pytest.fixture +def hz(monkeypatch): + cloud = Hz() + monkeypatch.setattr(drivers.get("hetzner"), "transport", httpx.MockTransport(cloud)) + return cloud + + +@pytest.fixture +def h(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +class Mesh: + """The mesh with its install line, as mesh.py will have it.""" + + def __init__(self): + self.tokens = [] + + def conf(self, cfg): + return {"enabled": True} + + def make_token(self, db, cfg, by=None, role="worker", name=None, namespaces=None, hours=24, uses=1): + self.tokens.append({"by": by, "role": role, "name": name, "hours": hours, "uses": uses}) + return {"id": len(self.tokens)}, f"lensjoin1.{len(self.tokens)}.secret" + + def join_line(self, cfg, token): + return f"curl -fsSL https://lens.example/install.sh | LENS_JOIN=http://lens.home:3000 LENS_JOIN_TOKEN={token} sh" + + +def tick(db, cfg, now=NOW): + return scaling.Scaler(db, lambda: cfg, "test").tick(now=now, lease=False) + + +def test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets(client, db, cfg, h, do, monkeypatch): + settings.save(db, cfg, "scaling", {"enabled": True}, "root@x.io") + cfg = settings.effective(db, cfg) + body = {"name": "DO", "driver": "digitalocean", "settings": {"region": "fra1"}, "secrets": {"token": "dop_v1_abc"}} + p = client.post("/api/v1/scaling/providers", headers=h, json=body).json() + assert p["check"] == {"ok": True, "regions": 2} and p["secrets"] == {"token": True} + assert do.calls[0][4] == "Bearer dop_v1_abc" # opened only for the call + assert "dop_v1_abc" not in str(db.one("SELECT sealed FROM $r", r=R("scale_provider", p["id"]))) + bad = {**body, "settings": {"region": "mars1"}} + assert "no region mars1" in client.post("/api/v1/scaling/providers", headers=h, json=bad).json()["check"]["error"] + + g = client.post( + "/api/v1/scaling/groups", + headers=h, + json={"name": "Workers", "provider": p["id"], "min": 1, "max": 2, "budget": 30, "template": {"steps": ["transcribe"]}}, + ).json() + assert g["approved"] is True and g["needs_join"] is True + assert g["template"] == {"size": "s-1vcpu-2gb", "image": "ubuntu-24-04-x64", "steps": ["transcribe"]} + assert tick(db, cfg)["launched"] == 0 and not do.machines # no way for a droplet to join yet: none is started + (view,) = client.get("/api/v1/scaling", headers=h).json()["groups"] + assert "install line" in view["waiting_for"] + + mesh = Mesh() + monkeypatch.setattr(scaling, "mesh_module", lambda: mesh) + assert client.get("/api/v1/scaling", headers=h).json()["groups"][0]["waiting_for"] is None + assert tick(db, cfg)["launched"] == 1 + ((did, m),) = do.machines.items() + (n,) = scaling.nodes(db, g["id"]) + assert n["external_id"] == str(did) and n["price"] == pytest.approx(0.01786) # DigitalOcean's list price + sent = m["body"] + assert sent["region"] == "fra1" and sent["size"] == "s-1vcpu-2gb" and sent["tags"] == ["lens"] and sent["name"] == n["name"] + assert sent["user_data"].startswith("#!/bin/sh\n") + assert "LENS_JOIN_TOKEN=lensjoin1.1.secret" in sent["user_data"] and "export LENS_WORKER_STEPS=transcribe" in sent["user_data"] + assert f"export LENS_SCALING_UNIT={n['name']}" in sent["user_data"] + assert mesh.tokens == [{"by": "scaling", "role": "worker", "name": n["name"], "hours": 1, "uses": 1}] # one use, one hour + + drv = drivers.get("digitalocean") + opened = scaling.opened(db, cfg, p["id"]) + assert drv.describe(opened, n)["state"] == "pending" + do.machines[did]["status"] = "active" + do.machines[did]["networks"] = {"v4": [{"type": "private", "ip_address": "10.0.0.2"}, {"type": "public", "ip_address": "198.51.100.4"}]} + assert drv.describe(opened, n) == {"state": "running", "address": "198.51.100.4"} + assert drv.listing(opened, "lens") == [{"id": str(did), "name": n["name"]}] + drv.terminate(opened, n) + assert not do.machines and drv.describe(opened, n) == {"state": "gone"} + drv.terminate(opened, n) # already gone: no error + + +def test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners(client, db, cfg, h, hz): + body = {"name": "Hetzner", "driver": "hetzner", "settings": {"usd_per_eur": 1.2}, "secrets": {"token": "hz"}} + p = client.post("/api/v1/scaling/providers", headers=h, json=body).json() + assert p["check"] == {"ok": True, "locations": 2} and p["settings"]["location"] == "fsn1" + drv = drivers.get("hetzner") + opened = scaling.opened(db, cfg, p["id"]) + assert drv.price(opened, {"server_type": "cx22"}) == pytest.approx(0.0071 * 1.2) + + boot = {"env": {"LENS_SCALING_UNIT": "lens-w-1"}, "line": "echo join", "tag": "lens"} + got = drv.launch(opened, {"server_type": "cx22", "image": "ubuntu-24.04"}, "lens-w-1", boot) + assert got["address"] == "203.0.113.7" + sent = hz.calls[-1][3] + assert sent["labels"] == {"lens": "lens"} and sent["location"] == "fsn1" and sent["user_data"].endswith("echo join\n") + node = {"external_id": str(got["id"])} + assert drv.describe(opened, node)["state"] == "pending" + hz.machines[got["id"]]["status"] = "running" + assert drv.describe(opened, node)["state"] == "running" + hz.machines[got["id"]]["status"] = "off" + assert drv.describe(opened, node)["state"] == "gone" # switched off: the scaler lets it go + assert drv.listing(opened, "lens") == [{"id": str(got["id"]), "name": "lens-w-1"}] + drv.terminate(opened, node) + with pytest.raises(drivers.DriverError, match="Hetzner said 404: server not found"): + drv.call(opened, "GET", f"/servers/{got['id']}") + + +def test_the_cloud_drivers_are_listed_and_need_the_mesh(client, h): + ds = {d["id"]: d for d in client.get("/api/v1/scaling", headers=h).json()["drivers"]} + assert ds["digitalocean"]["needs_join"] is True and ds["hetzner"]["priced"] is True + assert ds["local"]["needs_join"] is False + assert ds["hetzner"]["fields"]["token"]["secret"] is True and ds["digitalocean"]["extra"] is None # nothing to install diff --git a/threat-model.md b/threat-model.md index 53610f9e7..a8414cbd2 100644 --- a/threat-model.md +++ b/threat-model.md @@ -535,7 +535,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-MEDIA-LINK | Media, frames, documents and reports open without a session only through a link the server signed for that path, or a live share link. Text people or models write is never signed. | info-leak | security-critical | *(documented, docs/architecture.md "is never signed")* | Voided by: Knowing `ACCESS_SECRET_KEY` signs any path — `core/security.py:43`; dev compose key — `docker-compose.yml:24`. Lifetime `MEDIA_URL_EXPIRE_SECONDS` — `config.py:30`. Report pages sign every asset of the namespace — `api/pages.py:140-141`. Signing is limited to allow-listed fields — `api/media.py:25-30`. | | P-SHARE | A share link opens one asset, read-only. It expires and can be revoked. Only hashes are stored. A dead link answers the same 410 whatever went wrong. | integrity-bypass | security-critical | *(documented, docs/authentication.md "give read-only access to one asset's player and embed")* | Voided by: Liveness plus asset match only — `domain/auth.py:441-444`; `?s=` routes — `api/deps.py:252` (Q7). | | P-FEEDS | A published feed's address opens that feed alone, read-only, with no session: the feed's XML and the audio of items it lists. Only an editor of the namespace makes a feed, and what it lists is bounded by who already reads that namespace: an item's resource or activity is in the namespace, or in one every reader of the feed's namespace reads too, never a vault's, and a namespace made a vault stops its feeds. The token is found by its hash; rotating or deleting the feed stops the old address at once. Text in an item cannot add markup to a reader. | info-leak | security-critical | *(documented, docs/feeds.md "Private by default")* | Voided by: the token is sealed with the server key, not only hashed, so the namespace's readers (and whoever holds `secret.key`) see the address again — `domain/syndication.py` `token_of`; anyone given the address reads the feed until it is rotated; editors choose what goes in, as does a routine (admins make them) whose workflow nodes publish what they pick, bounded the same way; a resource's title and summary show as they are now, so an edit after publishing shows; the fit is by namespace roles (`domain/syndication.py` `Fit`), not per-asset grants, so an asset later closed by access level still shows while its namespace fits. | -| P-SCALING | Auto scaling launches nothing that costs money unless a person, signed in to the web app, approved the group's terms: its provider, what each unit is, its max, its budget and what happens at its cap. An API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away. A provider's credentials are sealed per provider and field, opened only for a call to it, and never shown. A unit is handed no long-lived Lens secret to start. | integrity-bypass | security-critical | *(documented, docs/scaling.md "The person approves")* | Voided by: an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (`domain/scaling.py` `accrue`), not bills, so a provider can charge more than the budget counts. | +| P-SCALING | Auto scaling launches nothing that costs money unless a person, signed in to the web app, approved the group's terms: its provider, what each unit is, its max, its budget and what happens at its cap. An API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away. A provider's credentials are sealed per provider and field, opened only for a call to it, and never shown. A unit is handed no long-lived Lens secret to start. | integrity-bypass | security-critical | *(documented, docs/scaling.md "The person approves")* | Voided by: an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (`domain/scaling.py` `accrue`), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds a join token for that unit, usable once within an hour, which anyone with the cloud account or a process on that machine reading its metadata can see until it is spent (`domain/scaling_drivers.py` `user_data`); a cloud provider's API token can do whatever its account can, and Lens calls only the provider's fixed API host with it (`domain/scaling_clouds.py`). | | P-AUTH-REQ | A request is authenticated only if its bearer verifies and the account and session are live. Sign-out and disabling take effect at once, for every token kind. Revoking an app grant or API token ends its tokens at once. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Every request checks that the account is still active")* | Voided by: Liveness checked per request — `api/deps.py:92`; JWT algorithm from `ALGORITHM` — `config.py:27`; dev secrets — `docker-compose.yml:24`. No bypass flag: `grep -rniE "disable_auth\|skip_auth\|no_auth\|auth_bypass\|AUTH_DISABLED\|ALLOW_ANON\|DEV_MODE" fastapi_backend/app nextjs-frontend/lib nextjs-frontend/auth.ts nextjs-frontend/proxy.ts docker-compose*.yml install.sh` — 0 hits. | | P-REMOVE | Removing an account ends every way it signs in at once (sessions, passkeys, sign-in links, connected accounts, API tokens, app grants, IIIF tokens) and takes every role, collection role and recording permission. Its email can make a new account, which inherits nothing. Admins only; never yourself, never the Matterbridge account, never the last passkey into a vault unless the admin says its files may be lost. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Removing someone")* | Voided by: the account row stays as a marker with an `.invalid` email — `domain/auth.py` `remove_account`; shared notes, views and comments keep its id. | | P-BEARER-ONLY | `/api/v1` takes bearer tokens only, never cookies. The IIIF authorization flow and the web app's NextAuth handlers are the cookie surfaces. | integrity-bypass | security-critical | *(documented, docs/architecture.md "so there is no CSRF surface")* | Voided by: IIIF auth cookies — `api/iiif.py:232`, `api/iiif.py:333`; external-login state cookie — `api/v1/routes/external.py:34`. | diff --git a/threat-model.yaml b/threat-model.yaml index 04d05c44d..d08d60fc1 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -1547,7 +1547,7 @@ properties_claimed: tier: security-critical conditions: "auto scaling launches nothing unless a person signed in to the web app approved the group's terms (its provider, what each unit is, its max, its budget, what happens at its cap); an API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away; a provider's credentials are sealed per provider and field, opened only for a call to it and never shown; a unit is handed no long-lived Lens secret" violation_symptoms: [integrity-bypass] - x-voided-by: "an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (domain/scaling.py accrue), not bills, so a provider can charge more than the budget counts" + x-voided-by: "an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (domain/scaling.py accrue), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds a join token for that unit, usable once within an hour, which anyone with the cloud account or a process on that machine reading its metadata can see until it is spent (domain/scaling_drivers.py user_data); a cloud provider's API token can do whatever its account can, and Lens calls only the provider's fixed API host with it (domain/scaling_clouds.py)" provenance: {kind: documented, source: 'docs/scaling.md "The person approves"'} - id: P-AUTH-REQ kind: authentication