From a5a30b8dca023857d367d3d105172ba19eab0da3 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 07:33:00 +0000 Subject: [PATCH 1/2] feat(scaling): Proxmox VE provider Units are linked (or full) clones of a VM template, tagged for Lens and started; once the guest agent answers, the first-boot script runs in the clone through it, so nothing lands on the server's storage. Letting a unit go stops the clone and deletes it with its disks. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01ALjNbLobvPe7VwvDL8j7V3 --- README.md | 6 +- docs/requirements.md | 1 + docs/scaling.md | 15 +- fastapi_backend/app/domain/scaling.py | 2 +- fastapi_backend/app/domain/scaling_proxmox.py | 207 ++++++++++++++++++ .../tests/api/test_scaling_proxmox.py | 142 ++++++++++++ threat-model.md | 2 +- threat-model.yaml | 2 +- 8 files changed, 368 insertions(+), 9 deletions(-) create mode 100644 fastapi_backend/app/domain/scaling_proxmox.py create mode 100644 fastapi_backend/tests/api/test_scaling_proxmox.py diff --git a/README.md b/README.md index 8ea54f243..765a56782 100644 --- a/README.md +++ b/README.md @@ -182,8 +182,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today; - DigitalOcean and Hetzner Cloud providers are in progress (built, launching once the mesh's install line lands), and - AWS, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); + DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching once the mesh's install + line lands), and AWS, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out)) - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine, @@ -261,7 +261,7 @@ Next, and not in a pull request yet; each links to where it is tracked: topics, recordings and the other things that have pages ([notes](docs/notes.md)). - **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models ([local models](docs/local-models.md)). -- **Auto scaling:** providers for AWS, Proxmox and Linux hosts over SSH, joining over the Nebula mesh, model hosts +- **Auto scaling:** providers for AWS and Linux hosts over SSH, joining over the Nebula mesh, model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings ([auto scaling](docs/scaling.md)). - **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the diff --git a/docs/requirements.md b/docs/requirements.md index 18ce5ad34..5fdb7663c 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -222,6 +222,7 @@ request, to another check that proves the same thing; loosening a requirement or | Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` | | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` | | DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` | +| Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` | | Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` | ## Keep it up to date and backed up diff --git a/docs/scaling.md b/docs/scaling.md index 39759320f..ed8072449 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -10,8 +10,8 @@ Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the A Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that runs more `lens worker` processes where Lens runs. In progress: **DigitalOcean** and **Hetzner Cloud** providers are built (credentials checked, list prices, launching, watching and deleting machines), and launch once the mesh's -install line, which another machine joins Lens with, lands ([the mesh](mesh.md)). Planned: providers for AWS, Proxmox -and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making +install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is **Proxmox VE**, clones of a +VM template on your own server. Planned: providers for AWS and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings. Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms @@ -56,6 +56,7 @@ carries none of them. | --- | --- | --- | --- | | `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) | | `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` | +| `proxmox` (Proxmox VE) | clones of a VM template on `node`, at `url` with an API token (`token_id`, `token_secret` sealed); `verify_tls` (on) and `ca_cert` for a private CA | nothing, unless you give it a price | `vm_template` (its id), `full_clone` (off: linked), `storage`, `pool`, `steps` | | `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` | A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where @@ -70,6 +71,14 @@ database credential travels inside the sealed join answer, so the user-data hold carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=`), so Lens's machines can be told apart. Letting a unit go deletes its machine; a machine switched off at the provider counts as gone and is deleted. +A Proxmox unit is a clone of `vm_template`, tagged with Lens's tag and started. Its template needs the QEMU guest +agent (`qemu-guest-agent` installed, and the VM's Guest Agent option on): once the agent answers, Lens runs the same +first-boot script in the clone through it, then tags it `lens-booted`, so nothing is put on the server's storage. The +script stays in Lens's memory until then; a clone whose boot was cut short, by Lens restarting, never joins and is +deleted. Letting a unit go stops the clone and deletes it with its disks. The API token needs to clone, configure, +start, stop and delete VMs and use their guest agent (the `PVEVMAdmin` role on the template and the clones' pool is +enough). + Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing. @@ -161,7 +170,7 @@ Turning scaling on (`change_settings scaling`) always asks. ## Refine later -Providers for Proxmox, SSH hosts and model hosts; the AWS provider (in its own pull request); revoking a unit's mesh +Providers for SSH hosts and model hosts; Proxmox LXC containers; the AWS provider (in its own pull request); revoking a unit's mesh certificate when it goes; reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro rate; spot and preemptible instances; warm pools; predictive scaling; health checks beyond joining; prices fetched from providers' pricing APIs. diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py index 25da0e33a..dd4928626 100644 --- a/fastapi_backend/app/domain/scaling.py +++ b/fastapi_backend/app/domain/scaling.py @@ -44,7 +44,7 @@ import time from . import activity, budgets, jobs, schedule, settings, store -from . import scaling_clouds # noqa: F401 - registers the cloud drivers +from . import scaling_clouds, scaling_proxmox # noqa: F401 - registers their drivers from . import scaling_drivers as drivers from .store import R diff --git a/fastapi_backend/app/domain/scaling_proxmox.py b/fastapi_backend/app/domain/scaling_proxmox.py new file mode 100644 index 000000000..251c0cba1 --- /dev/null +++ b/fastapi_backend/app/domain/scaling_proxmox.py @@ -0,0 +1,207 @@ +"""Proxmox VE as an auto scaling provider (scaling_drivers.py, docs/scaling.md): each unit is a clone of a VM template +on your own Proxmox server, which joins Lens over the mesh and runs `lens worker`; letting the unit go stops and +deletes the clone. It costs nothing unless you give the provider a price. + +The template needs the QEMU guest agent (`qemu-guest-agent`, with the VM's Guest Agent option on): once the clone is +up, Lens runs the first-boot script (scaling_drivers.user_data) in it through the agent, so nothing has to be put on +the Proxmox server's storage. Lens calls the Proxmox API with an API token (`user@realm!name` and its secret), over +httpx, so a provider of this kind installs nothing. +""" + +from __future__ import annotations + +import ssl +import threading +import time +from urllib.parse import quote + +import httpx + +from . import jobs +from .scaling_drivers import Driver, DriverError, _field, register, user_data + +TIMEOUT = 30.0 +CLONE_SECONDS = 600 # a full clone of a big disk takes a while; a linked clone, seconds +AGENT_SECONDS = 600 # how long the clone has to boot far enough for its guest agent to answer +BOOTED = "lens-booted" # the tag a clone gets once its first-boot script ran + + +class Proxmox(Driver): + id = "proxmox" + label = "Proxmox VE" + about = "Clones of a VM template on your Proxmox server, which join Lens over the mesh and run Lens workers." + priced = False + needs_join = True + fields = { + "url": _field("Address", required=True, help="the Proxmox API, like https://pve.home:8006"), + "node": _field("Node", required=True, help="the Proxmox node clones run on, like pve"), + "token_id": _field("API token id", required=True, help="user@realm!name, like lens@pve!scaling"), + "token_secret": _field("API token secret", required=True, secret=True), + "verify_tls": _field("Check the certificate", "boolean", default=True, help="off only for a test server"), + "ca_cert": _field("CA certificate", help="the PEM of the CA that signed the server's certificate, when it isn't public"), + } + template = { + "vm_template": _field("VM template", "integer", required=True, help="the id of the template to clone"), + "full_clone": _field("Full clone", "boolean", default=False, help="a full copy of the disk; off: a linked clone"), + "storage": _field("Storage", help="where a full clone's disk goes; none: the template's"), + "pool": _field("Pool", help="the resource pool clones join"), + "steps": _field( + "Steps", + "list", + help="the steps these workers run (transcribe, analyze, embed...); none: every step", + options=sorted(jobs.STEPS), + ), + } + transport: httpx.BaseTransport | None = None # tests hand in an httpx.MockTransport + background = True # tests run the boot inline + pause = staticmethod(time.sleep) + + def __init__(self): + self.booting: dict[str, str | None] = {} # clone name -> None while it boots, or why its boot failed + self.lock = threading.Lock() + + # ---------- the API ---------- + def _verify(self, p): + s = p["settings"] + if not s.get("verify_tls", True): + return False + if s.get("ca_cert"): + try: + return ssl.create_default_context(cadata=s["ca_cert"]) + except ssl.SSLError as e: + raise DriverError(f"the CA certificate can't be read: {e.reason}") from None + return True + + def _client(self, p): + s = p["settings"] + return httpx.Client( + base_url=s["url"].rstrip("/") + "/api2/json", + headers={"Authorization": f"PVEAPIToken={s['token_id']}={p['secrets'].get('token_secret') or ''}"}, + timeout=TIMEOUT, + verify=self._verify(p) if self.transport is None else True, + transport=self.transport, + ) + + def call(self, p, method, path, gone_ok=False, **kw): + """The answer's `data`; None when `gone_ok` and the VM isn't there. DriverError otherwise.""" + try: + with self._client(p) as c: + r = c.request(method, path, **kw) + except httpx.HTTPError as e: + raise DriverError(f"couldn't reach Proxmox: {type(e).__name__}") from None + said = (r.reason_phrase or "").strip() + if gone_ok and (r.status_code == 404 or (r.status_code == 500 and "does not exist" in (said + r.text))): + return None + if r.status_code >= 400: + raise DriverError(f"Proxmox said {r.status_code}" + (f": {said}" if said else "")) + return (r.json() if r.content else {}).get("data") + + def _vm(self, node): + where, _, vmid = (node.get("external_id") or "").partition("/") + return where, vmid + + def _wait(self, p, where, upid, seconds): + """Until the task ends; DriverError if it failed or took too long.""" + end = time.monotonic() + seconds + while True: + st = self.call(p, "GET", f"/nodes/{where}/tasks/{quote(upid, safe='')}/status") or {} + if st.get("status") == "stopped": + if st.get("exitstatus") != "OK": + raise DriverError(f"Proxmox's task failed: {st.get('exitstatus')}") + return + if time.monotonic() > end: + raise DriverError(f"Proxmox's task didn't finish within {seconds} seconds") + self.pause(2) + + # ---------- the driver ---------- + def check(self, p): + version = (self.call(p, "GET", "/version") or {}).get("version") + nodes = {n.get("node") for n in self.call(p, "GET", "/nodes") or []} + if p["settings"]["node"] not in nodes: + raise DriverError(f"Proxmox has no node {p['settings']['node']}; it has {', '.join(sorted(nodes)) or 'none'}") + return {"version": version} + + def launch(self, p, template, name, boot): + where = p["settings"]["node"] + vmid = int(self.call(p, "GET", "/cluster/nextid")) + body = {"newid": vmid, "name": name, "full": 1 if template.get("full_clone") else 0} + if template.get("full_clone") and template.get("storage"): + body["storage"] = template["storage"] + if template.get("pool"): + body["pool"] = template["pool"] + upid = self.call(p, "POST", f"/nodes/{where}/qemu/{template['vm_template']}/clone", data=body) + with self.lock: + self.booting[name] = None + script = user_data(boot, template) + if self.background: + threading.Thread(target=self._boot, args=(p, where, vmid, upid, name, boot["tag"], script), daemon=True).start() + else: + self._boot(p, where, vmid, upid, name, boot["tag"], script) + return {"id": f"{where}/{vmid}"} + + def _boot(self, p, where, vmid, upid, name, tag, script): + """Wait for the clone, tag it, start it, and once its guest agent answers, run the first-boot script in it. The + script lives only here, never on the Proxmox server: a clone whose boot was cut short (Lens restarted) never + joins, and the scaler deletes it.""" + vm = f"/nodes/{where}/qemu/{vmid}" + try: + if upid: + self._wait(p, where, upid, CLONE_SECONDS) + self.call(p, "PUT", f"{vm}/config", data={"tags": tag}) + self._wait(p, where, self.call(p, "POST", f"{vm}/status/start"), 120) + end = time.monotonic() + AGENT_SECONDS + while True: + try: + self.call(p, "POST", f"{vm}/agent/ping") + break + except DriverError: + if time.monotonic() > end: + raise DriverError(f"the clone's guest agent didn't answer within {AGENT_SECONDS} seconds") from None + self.pause(5) + self.call(p, "POST", f"{vm}/agent/exec", data={"command": ["sh", "-c", script]}) + self.call(p, "PUT", f"{vm}/config", data={"tags": f"{tag};{BOOTED}"}) + with self.lock: + self.booting.pop(name, None) + except Exception as e: # noqa: BLE001 - shown on the unit when the scaler next looks + with self.lock: + self.booting[name] = str(e)[:300] if isinstance(e, DriverError) else f"{type(e).__name__}: {e}"[:300] + + def describe(self, p, node): + with self.lock: + boot = self.booting.get(node.get("name"), ...) + if boot is None: + return {"state": "pending"} # still cloning or booting here + if boot is not ...: + raise DriverError(boot) # the boot failed: shown on the unit, which fails when it doesn't join + where, vmid = self._vm(node) + st = self.call(p, "GET", f"/nodes/{where}/qemu/{vmid}/status/current", gone_ok=True) + if st is None or st.get("status") != "running": + return {"state": "gone"} # stopped, or a clone whose boot this process never finished + return {"state": "running"} + + def terminate(self, p, node): + with self.lock: + self.booting.pop(node.get("name"), None) + where, vmid = self._vm(node) + if not vmid: + return + vm = f"/nodes/{where}/qemu/{vmid}" + st = self.call(p, "GET", f"{vm}/status/current", gone_ok=True) + if st is None: + return + if st.get("status") == "running": + self._wait(p, where, self.call(p, "POST", f"{vm}/status/stop"), 120) + upid = self.call(p, "DELETE", vm, gone_ok=True, params={"purge": 1, "destroy-unreferenced-disks": 1}) + if upid: + self._wait(p, where, upid, 300) + + def listing(self, p, tag): + vms = self.call(p, "GET", "/cluster/resources", params={"type": "vm"}) or [] + return [ + {"id": f"{v.get('node')}/{v.get('vmid')}", "name": v.get("name")} + for v in vms + if tag in (v.get("tags") or "").split(";") and not v.get("template") + ] + + +register(Proxmox()) diff --git a/fastapi_backend/tests/api/test_scaling_proxmox.py b/fastapi_backend/tests/api/test_scaling_proxmox.py new file mode 100644 index 000000000..a5c3b5baa --- /dev/null +++ b/fastapi_backend/tests/api/test_scaling_proxmox.py @@ -0,0 +1,142 @@ +"""Auto scaling on Proxmox VE (docs/scaling.md "Providers"): clones of a VM template, booted and joined through the +guest agent, against an httpx MockTransport standing in for the Proxmox API.""" + +from __future__ import annotations + +from urllib.parse import parse_qs + +import httpx +import pytest + +from app.domain import scaling +from app.domain import scaling_drivers as drivers +from tests.helpers import login, make_user + + +class PVE: + """A Proxmox node in memory: VMs by id, tasks that finish at once, a guest agent that answers once booted.""" + + def __init__(self): + self.calls, self.vms, self.next = [], {9000: {"name": "debian-lens", "template": 1, "status": "stopped"}}, 105 + self.agent_after = 0 # pings that fail before the agent answers + + def __call__(self, req): + path = req.url.path.removeprefix("/api2/json") + form = {k: v if len(v) > 1 else v[0] for k, v in parse_qs(req.content.decode()).items()} if req.content else {} + self.calls.append((req.method, path, form, req.headers.get("authorization"))) + ok = lambda data: httpx.Response(200, json={"data": data}) # noqa: E731 + if path == "/version": + return ok({"version": "8.2.4"}) + if path == "/nodes": + return ok([{"node": "pve"}]) + if path == "/cluster/nextid": + return ok(str(self.next)) + if path == "/cluster/resources": + return ok([{"node": "pve", "vmid": i, **v} for i, v in self.vms.items()]) + if path.startswith("/nodes/pve/tasks/"): + return ok({"status": "stopped", "exitstatus": "OK"}) + parts = path.split("/") # ['', 'nodes', 'pve', 'qemu', id, ...] + vmid, rest = int(parts[4]), "/".join(parts[5:]) + if rest == "clone": + self.vms[int(form["newid"])] = {"name": form["name"], "status": "stopped", "tags": ""} + self.next += 1 + return ok("UPID:pve:clone") + vm = self.vms.get(vmid) + if vm is None: + return httpx.Response( + 500, json={"data": None}, extensions={"reason_phrase": f"Configuration file '{vmid}.conf' does not exist".encode()} + ) + if req.method == "DELETE": + del self.vms[vmid] + return ok("UPID:pve:destroy") + if rest == "config": + vm["tags"] = form["tags"] + return ok(None) + if rest == "status/start": + vm["status"] = "running" + return ok("UPID:pve:start") + if rest == "status/stop": + vm["status"] = "stopped" + return ok("UPID:pve:stop") + if rest == "status/current": + return ok({"status": vm["status"]}) + if rest == "agent/ping": + if self.agent_after > 0: + self.agent_after -= 1 + return httpx.Response(500, json={"data": None}, extensions={"reason_phrase": b"QEMU guest agent is not running"}) + return ok({}) + if rest == "agent/exec": + vm["ran"] = form["command"] + return ok({"pid": 1}) + return httpx.Response(501) + + +@pytest.fixture +def pve(monkeypatch): + api = PVE() + drv = drivers.get("proxmox") + monkeypatch.setattr(drv, "transport", httpx.MockTransport(api)) + monkeypatch.setattr(drv, "background", False) + monkeypatch.setattr(drv, "pause", lambda s: None) + return api + + +@pytest.fixture +def h(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +SETTINGS = {"url": "https://pve.home:8006", "node": "pve", "token_id": "lens@pve!scaling"} + + +def test_proxmox_clones_boots_through_the_guest_agent_and_deletes(client, db, cfg, h, pve): + body = {"name": "Home lab", "driver": "proxmox", "settings": SETTINGS, "secrets": {"token_secret": "s3cret"}} + p = client.post("/api/v1/scaling/providers", headers=h, json=body).json() + assert p["check"] == {"ok": True, "version": "8.2.4"} and p["priced"] is False + assert pve.calls[0][3] == "PVEAPIToken=lens@pve!scaling=s3cret" + bad = {**body, "settings": {**SETTINGS, "node": "pve2"}} + assert "no node pve2" in client.post("/api/v1/scaling/providers", headers=h, json=bad).json()["check"]["error"] + + drv = drivers.get("proxmox") + opened = scaling.opened(db, cfg, p["id"]) + pve.agent_after = 2 # the clone takes a while to boot + boot = {"env": {"LENS_SCALING_UNIT": "lens-lab-1"}, "line": "echo join", "tag": "lens"} + got = drv.launch(opened, {"vm_template": 9000, "steps": ["ocr"]}, "lens-lab-1", boot) + assert got == {"id": "pve/105"} + clone = next(c for c in pve.calls if c[1].endswith("/clone")) + assert clone[1] == "/nodes/pve/qemu/9000/clone" and clone[2] == {"newid": "105", "name": "lens-lab-1", "full": "0"} + vm = pve.vms[105] + assert vm["status"] == "running" and vm["tags"] == "lens;lens-booted" + assert vm["ran"][:2] == ["sh", "-c"] and "export LENS_WORKER_STEPS=ocr" in vm["ran"][2] and vm["ran"][2].endswith("echo join\n") + + node = {"name": "lens-lab-1", "external_id": got["id"]} + assert drv.describe(opened, node) == {"state": "running"} + assert drv.listing(opened, "lens") == [{"id": "pve/105", "name": "lens-lab-1"}] # the template isn't Lens's + drv.terminate(opened, node) + assert 105 not in pve.vms and drv.describe(opened, node) == {"state": "gone"} + drv.terminate(opened, node) # already gone: no error + + +def test_a_clone_whose_agent_never_answers_says_so(client, db, cfg, h, pve, monkeypatch): + body = {"name": "Lab", "driver": "proxmox", "settings": SETTINGS, "secrets": {"token_secret": "s"}} + p = client.post("/api/v1/scaling/providers", headers=h, json=body).json() + drv = drivers.get("proxmox") + monkeypatch.setattr("app.domain.scaling_proxmox.AGENT_SECONDS", 0) + pve.agent_after = 10**6 + opened = scaling.opened(db, cfg, p["id"]) + got = drv.launch(opened, {"vm_template": 9000}, "lens-lab-2", {"env": {}, "line": "echo join", "tag": "lens"}) + with pytest.raises(drivers.DriverError, match="guest agent didn't answer"): + drv.describe(opened, {"name": "lens-lab-2", "external_id": got["id"]}) + assert "ran" not in pve.vms[105] + drv.terminate(opened, {"name": "lens-lab-2", "external_id": got["id"]}) + assert 105 not in pve.vms + + +def test_a_clone_still_booting_is_pending(pve): + drv = drivers.get("proxmox") + drv.booting["lens-lab-3"] = None # its boot is still running in the background + try: + assert drv.describe({"settings": SETTINGS, "secrets": {}}, {"name": "lens-lab-3", "external_id": "pve/200"}) == {"state": "pending"} + finally: + drv.booting.pop("lens-lab-3", None) diff --git a/threat-model.md b/threat-model.md index dcbe3504f..cd2f6ea6d 100644 --- a/threat-model.md +++ b/threat-model.md @@ -538,7 +538,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-MEDIA-LINK | Media, frames, documents and reports open without a session only through a link the server signed for that path, or a live share link. Text people or models write is never signed. | info-leak | security-critical | *(documented, docs/architecture.md "is never signed")* | Voided by: Knowing `ACCESS_SECRET_KEY` signs any path — `core/security.py:43`; dev compose key — `docker-compose.yml:24`. Lifetime `MEDIA_URL_EXPIRE_SECONDS` — `config.py:30`. Report pages sign every asset of the namespace — `api/pages.py:140-141`. Signing is limited to allow-listed fields — `api/media.py:25-30`. | | P-SHARE | A share link opens one asset, read-only. It expires and can be revoked. Only hashes are stored. A dead link answers the same 410 whatever went wrong. | integrity-bypass | security-critical | *(documented, docs/authentication.md "give read-only access to one asset's player and embed")* | Voided by: Liveness plus asset match only — `domain/auth.py:441-444`; `?s=` routes — `api/deps.py:252` (Q7). | | P-FEEDS | A published feed's address opens that feed alone, read-only, with no session: the feed's XML and the audio of items it lists. Only an editor of the namespace makes a feed, and what it lists is bounded by who already reads that namespace: an item's resource or activity is in the namespace, or in one every reader of the feed's namespace reads too, never a vault's, and a namespace made a vault stops its feeds. The token is found by its hash; rotating or deleting the feed stops the old address at once. Text in an item cannot add markup to a reader. | info-leak | security-critical | *(documented, docs/feeds.md "Private by default")* | Voided by: the token is sealed with the server key, not only hashed, so the namespace's readers (and whoever holds `secret.key`) see the address again — `domain/syndication.py` `token_of`; anyone given the address reads the feed until it is rotated; editors choose what goes in, as does a routine (admins make them) whose workflow nodes publish what they pick, bounded the same way; a resource's title and summary show as they are now, so an edit after publishing shows; the fit is by namespace roles (`domain/syndication.py` `Fit`), not per-asset grants, so an asset later closed by access level still shows while its namespace fits. | -| P-SCALING | Auto scaling launches nothing that costs money unless a person, signed in to the web app, approved the group's terms: its provider, what each unit is, its max, its budget and what happens at its cap. An API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away. A provider's credentials are sealed per provider and field, opened only for a call to it, and never shown. A unit is handed no long-lived Lens secret to start. | integrity-bypass | security-critical | *(documented, docs/scaling.md "The person approves")* | Voided by: an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (`domain/scaling.py` `accrue`), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds a join token for that unit, usable once within an hour, which anyone with the cloud account or a process on that machine reading its metadata can see until it is spent (`domain/scaling_drivers.py` `user_data`); a cloud provider's API token can do whatever its account can, and Lens calls only the provider's fixed API host with it (`domain/scaling_clouds.py`). | +| P-SCALING | Auto scaling launches nothing that costs money unless a person, signed in to the web app, approved the group's terms: its provider, what each unit is, its max, its budget and what happens at its cap. An API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away. A provider's credentials are sealed per provider and field, opened only for a call to it, and never shown. A unit is handed no long-lived Lens secret to start. | integrity-bypass | security-critical | *(documented, docs/scaling.md "The person approves")* | Voided by: an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (`domain/scaling.py` `accrue`), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds a join token for that unit, usable once within an hour, which anyone with the cloud account or a process on that machine reading its metadata can see until it is spent (`domain/scaling_drivers.py` `user_data`); a cloud provider's API token can do whatever its account can, and Lens calls only the provider's fixed API host with it (`domain/scaling_clouds.py`); a Proxmox provider's address is the admin's to give, and turning `verify_tls` off trusts any server answering there with the API token; a Proxmox unit's first-boot script, with its join token, is passed to the clone through the guest agent (`domain/scaling_proxmox.py`). | | P-AUTH-REQ | A request is authenticated only if its bearer verifies and the account and session are live. Sign-out and disabling take effect at once, for every token kind. Revoking an app grant or API token ends its tokens at once. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Every request checks that the account is still active")* | Voided by: Liveness checked per request — `api/deps.py:92`; JWT algorithm from `ALGORITHM` — `config.py:27`; dev secrets — `docker-compose.yml:24`. No bypass flag: `grep -rniE "disable_auth\|skip_auth\|no_auth\|auth_bypass\|AUTH_DISABLED\|ALLOW_ANON\|DEV_MODE" fastapi_backend/app nextjs-frontend/lib nextjs-frontend/auth.ts nextjs-frontend/proxy.ts docker-compose*.yml install.sh` — 0 hits. | | P-REMOVE | Removing an account ends every way it signs in at once (sessions, passkeys, sign-in links, connected accounts, API tokens, app grants, IIIF tokens) and takes every role, collection role and recording permission. Its email can make a new account, which inherits nothing. Admins only; never yourself, never the Matterbridge account, never the last passkey into a vault unless the admin says its files may be lost. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Removing someone")* | Voided by: the account row stays as a marker with an `.invalid` email — `domain/auth.py` `remove_account`; shared notes, views and comments keep its id. | | P-BEARER-ONLY | `/api/v1` takes bearer tokens only, never cookies. The IIIF authorization flow and the web app's NextAuth handlers are the cookie surfaces. | integrity-bypass | security-critical | *(documented, docs/architecture.md "so there is no CSRF surface")* | Voided by: IIIF auth cookies — `api/iiif.py:232`, `api/iiif.py:333`; external-login state cookie — `api/v1/routes/external.py:34`. | diff --git a/threat-model.yaml b/threat-model.yaml index 119f5af42..24c111fca 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -1547,7 +1547,7 @@ properties_claimed: tier: security-critical conditions: "auto scaling launches nothing unless a person signed in to the web app approved the group's terms (its provider, what each unit is, its max, its budget, what happens at its cap); an API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away; a provider's credentials are sealed per provider and field, opened only for a call to it and never shown; a unit is handed no long-lived Lens secret" violation_symptoms: [integrity-bypass] - x-voided-by: "an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (domain/scaling.py accrue), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds a join token for that unit, usable once within an hour, which anyone with the cloud account or a process on that machine reading its metadata can see until it is spent (domain/scaling_drivers.py user_data); a cloud provider's API token can do whatever its account can, and Lens calls only the provider's fixed API host with it (domain/scaling_clouds.py)" + x-voided-by: "an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (domain/scaling.py accrue), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds a join token for that unit, usable once within an hour, which anyone with the cloud account or a process on that machine reading its metadata can see until it is spent (domain/scaling_drivers.py user_data); a cloud provider's API token can do whatever its account can, and Lens calls only the provider's fixed API host with it (domain/scaling_clouds.py); a Proxmox provider's address is the admin's to give, and turning `verify_tls` off trusts any server answering there with the API token; a Proxmox unit's first-boot script, with its join token, is passed to the clone through the guest agent (`domain/scaling_proxmox.py`)" provenance: {kind: documented, source: 'docs/scaling.md "The person approves"'} - id: P-AUTH-REQ kind: authentication From 08e9d4ad987589ca63c1ffab629d3d24fdd3e62b Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 08:11:46 +0000 Subject: [PATCH 2/2] docs(readme): list answering questions about Lens as built, now that v0.15.0 released it The requirements check fails every pull request once a release's changelog names an In progress item. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01ALjNbLobvPe7VwvDL8j7V3 --- README.md | 7 +++---- docs/requirements.md | 1 + 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 765a56782..f57f5aeaf 100644 --- a/README.md +++ b/README.md @@ -88,11 +88,12 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine (opt-in), any message in one click. Long conversations are summarised before they fill the model's context, and `/clear` and `/compact` start afresh or summarise now. Conversations title themselves after their first answer, unless - you name them. Each namespace can have its own assistant with + you name them. It answers questions about Lens itself from its docs and this server. Each namespace can have its + own assistant with its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages), - [mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records), + [mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [questions about Lens](docs/assistant.md#questions-about-lens-itself), [answer records](docs/assistant.md#what-each-answer-records), [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md); [checks](docs/requirements.md#ask-and-get-cited-answers)) - **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python), @@ -241,8 +242,6 @@ still loading. ([Components](docs/components.md), These are open pull requests, not yet part of Lens: -- **The assistant** answering questions about Lens itself, from its docs and this server - ([#400](https://github.com/adeelahmad/lense/pull/400)). - **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)). - **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)). - **Web clipper:** signing in, or using your API token, not only an import webhook's diff --git a/docs/requirements.md b/docs/requirements.md index 5fdb7663c..07c7d47c5 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -88,6 +88,7 @@ request, to another check that proves the same thing; loosening a requirement or | Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` | | Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` | | Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` | +| It answers questions about Lens itself from its docs and this server, and agents ask the same over MCP | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_docs_are_found_by_words`, `test_the_assistant_answers_about_lens_itself`, `test_admins_see_where_each_setting_is`, `test_agents_ask_lens_over_mcp` | | Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` | | Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again`, `test_what_tools_returned_is_cut_to_fit_the_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app`; [chat-summary-mark.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-summary-mark.test.tsx) `says how many messages the summary stands for, and shows it on request` | | A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` |