From 928e0bc234372711278b88ab9446822281212bf9 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 06:08:05 +0000 Subject: [PATCH 1/2] feat(scaling): AWS EC2 units, as spot instances or on demand Lens can grow from one machine onto AWS when work waits: a scaling group on an AWS provider launches EC2 instances that join over the mesh and run its worker. Spot units fall back to on demand when AWS has no spot capacity; when AWS takes one back the scaler says why and launches another. Prices are estimates (zone spot price or the Price List API); with no price nothing launches. Instances require IMDSv2, encrypt their disk and terminate on shutdown. botocore is the new `aws` extra, fetched when the first AWS provider is saved. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SETHpH6NUGKT1Bwd6xyPCA --- README.md | 7 +- docs/requirements.md | 3 + docs/scaling.md | 71 ++- fastapi_backend/app/domain/scaling.py | 2 +- fastapi_backend/app/domain/scaling_aws.py | 456 ++++++++++++++++++ fastapi_backend/app/domain/scaling_drivers.py | 2 + fastapi_backend/pyproject.toml | 3 + .../tests/domain/test_scaling_aws.py | 317 ++++++++++++ fastapi_backend/uv.lock | 6 +- threat-model.md | 2 +- threat-model.yaml | 2 +- 11 files changed, 861 insertions(+), 10 deletions(-) create mode 100644 fastapi_backend/app/domain/scaling_aws.py create mode 100644 fastapi_backend/tests/domain/test_scaling_aws.py diff --git a/README.md b/README.md index d21f9b448..3484aaa65 100644 --- a/README.md +++ b/README.md @@ -176,8 +176,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p - **Scale out.** Scaling groups add workers when work waits and let them go when it doesn't, by target, step and schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage - groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today; - cloud, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); + groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine, and on AWS EC2 + as spot or on-demand instances (they launch once the mesh joins machines); other clouds, Proxmox, Linux and model + hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out)) - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine, @@ -253,7 +254,7 @@ Next, and not in a pull request yet; each links to where it is tracked: topics, recordings and the other things that have pages ([notes](docs/notes.md)). - **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models ([local models](docs/local-models.md)). -- **Auto scaling:** providers for AWS, DigitalOcean, Hetzner, Proxmox and Linux hosts over SSH, joining over the +- **Auto scaling:** providers for DigitalOcean, Hetzner, Proxmox and Linux hosts over SSH, joining over the Nebula mesh, model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings ([auto scaling](docs/scaling.md)). - **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the diff --git a/docs/requirements.md b/docs/requirements.md index 77b3b7c57..92ac483a2 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -216,6 +216,9 @@ request, to another check that proves the same thing; loosening a requirement or | Each unit's cost goes into the activity ledger as an estimate, and a group stops at its budget's cap | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_at_its_budgets_cap_a_group_stops` | | Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` | | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` | +| AWS EC2 units launch as spot or on-demand instances, hardened (IMDSv2, encrypted disk, terminate on shutdown), with only the mesh's one-time join line in their user-data | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_a_spot_unit_launches_hardened_and_priced_in_its_zone`, `test_the_driver_is_offered_and_its_settings_are_checked` | +| A spot unit falls back to on demand when AWS has none; without a price or the mesh, nothing launches | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to`, `test_a_unit_without_a_price_or_the_mesh_launches_nothing` | +| When AWS takes a spot instance back, the scaler says why and launches another | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_describe_says_when_aws_took_a_spot_instance_back`, `test_the_scaler_replaces_a_spot_unit_aws_took_back`, `test_terminate_check_and_listing` | | Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` | ## Keep it up to date and backed up diff --git a/docs/scaling.md b/docs/scaling.md index 4ada0de54..790e54565 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -8,8 +8,9 @@ launched. Everything it does is in the [activity ledger](activity.md), and its c Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools, Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that -runs more `lens worker` processes where Lens runs. Planned: providers for AWS, DigitalOcean, Hetzner, Proxmox and any -Linux host over SSH, joining Lens over its Nebula mesh network; model hosts (RunPod, Lambda, Hugging Face, SageMaker, +runs more `lens worker` processes where Lens runs, and **AWS EC2**, spot or on demand (its units launch once the +[mesh](mesh.md) can join machines: until then a launch says so and costs nothing). Planned: providers for +DigitalOcean, Hetzner, Proxmox and any Linux host over SSH, joining Lens over its Nebula mesh network; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings. Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms @@ -53,10 +54,71 @@ carries none of them. | Driver | Units | Costs | Template | | --- | --- | --- | --- | | `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) | +| `aws` (AWS EC2) | EC2 instances that join over the mesh and run `lens worker` | the spot or on-demand price, an estimate | `instance_type`, `market` (`spot` or `on-demand`), `max_price`, `fallback`, `image`, `disk_gb`, `steps` | A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where it runs. +## AWS EC2 + +Lens grows onto AWS when work waits: from a Raspberry Pi at home to as many EC2 instances as the group's max and +budget allow, and back to none. Each unit is an instance that joins Lens over the [mesh](mesh.md) and runs its worker. + +```http +POST /api/v1/scaling/providers +{"name": "AWS", "driver": "aws", "settings": {"region": "eu-west-1"}, + "secrets": {"access_key_id": "AKIA...", "secret_access_key": "..."}} +``` + +| Provider setting | Default | | +| --- | --- | --- | +| `region` | `us-east-1` | the region units launch in | +| `access_key_id`, `secret_access_key` | | an IAM user's keys, sealed | +| `subnet_id` | the default VPC's | where units launch | +| `security_group_ids` | the VPC's default group | units only reach out (to the mesh's lighthouse), so they need no inbound rule | +| `key_name` | none | an EC2 key pair, to sign in to units yourself | + +| Unit (template) | Default | | +| --- | --- | --- | +| `instance_type` | `t4g.medium` | any EC2 type; Arm (Graviton) and x86 both work | +| `market` | `spot` | `spot`: AWS's spare capacity, often far cheaper, which AWS may take back with two minutes' notice; `on-demand` | +| `max_price` | the on-demand price | the most a spot unit may cost per hour | +| `fallback` | `true` | when AWS has no spot capacity of that type, or none under the max price, launch on demand instead | +| `image` | the latest Ubuntu 24.04 | an AMI; by default Canonical's for the type's architecture, found in the region | +| `disk_gb` | `30` | the unit's disk (gp3, encrypted) | +| `steps` | every step | the steps its worker runs | + +**Spot instances.** When AWS takes a spot instance back, the scaler sees it gone, records why ("AWS took the spot +instance back") and launches another, on demand if spot has run out and `fallback` allows. The job its worker was +running goes back in the queue once its heartbeat stops, so nothing is lost. + +**What it costs.** Each unit's price is an estimate: the spot price in the zone it landed in (from +`DescribeSpotPriceHistory`), or the on-demand list price (from the AWS Price List API). A price you give the provider +wins over both. Without any price Lens launches nothing, because a unit with no price would never count against the +budget. AWS bills separately; the ledger's figures are estimates. + +**Hardened by default.** Every instance requires IMDSv2, has an encrypted disk that goes with it, terminates itself +when it shuts down (a stopped one is terminated too, so its disk doesn't keep costing), and is tagged `lens-unit` with +its name. Its user-data is the mesh's install line with a join token that works once, within the hour, so it carries +no long-lived secret. + +The keys need these permissions (an IAM policy for a user only Lens uses): + +```json +{ + "Version": "2012-10-17", + "Statement": [ + {"Effect": "Allow", "Resource": "*", "Action": [ + "ec2:DescribeAvailabilityZones", "ec2:DescribeInstanceTypes", "ec2:DescribeImages", + "ec2:DescribeSpotPriceHistory", "ec2:DescribeInstances", "ec2:RunInstances", + "ec2:CreateTags", "ec2:TerminateInstances", "pricing:GetProducts"]} + ] +} +``` + +`pricing:GetProducts` is only needed for on-demand prices you don't give yourself. Lens talks to AWS through botocore +(Apache-2.0), fetched the first time an AWS provider is saved. + ## Groups ```http @@ -145,5 +207,8 @@ Turning scaling on (`change_settings scaling`) always asks. ## Refine later -Providers beyond This machine; spot and preemptible instances; warm pools; predictive scaling; health checks beyond +Providers beyond This machine and AWS; preemptible and spot instances at other clouds; draining an AWS spot unit on +its two-minute notice (the node agent reading the instance's metadata) and handing its job back at once; following +the spot price as it changes; AWS sign-in with an instance role or IAM Identity Center instead of keys; mixed instance +types per group; warm pools; predictive scaling; health checks beyond joining; prices fetched from providers' pricing APIs. diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py index 03aac825e..1a5ca9b6c 100644 --- a/fastapi_backend/app/domain/scaling.py +++ b/fastapi_backend/app/domain/scaling.py @@ -1176,7 +1176,7 @@ def _reconcile(self, cfg, g, drv, p, n, now, did): db.q("UPDATE $r SET error = $e", r=R("scale_node", n["id"]), e=f"{type(e).__name__}: {e}"[:300]) return True if d.get("state") == "gone": - why = "stopped at the provider" if state != "launching" else "never came up at the provider" + why = d.get("why") or ("stopped at the provider" if state != "launching" else "never came up at the provider") _terminate(db, cfg, drv, p, {**n, "external_id": None}, why, state="failed" if state == "launching" else "terminated", now=now) did["terminated"] += 1 return False diff --git a/fastapi_backend/app/domain/scaling_aws.py b/fastapi_backend/app/domain/scaling_aws.py new file mode 100644 index 000000000..27b6407ea --- /dev/null +++ b/fastapi_backend/app/domain/scaling_aws.py @@ -0,0 +1,456 @@ +"""AWS EC2 for auto scaling (scaling.py, docs/scaling.md): Lens launches EC2 instances that join it over the mesh and +run `lens worker`, as spot instances (AWS's spare capacity, a fraction of the on-demand price, which AWS may take back +with two minutes' notice) or on demand. + +A spot unit falls back to on demand when AWS has no spot capacity of that type, or none under the group's max price, +unless the group says not to. When AWS takes a spot instance back, `describe` says so and the scaler launches another; +the job its worker was running goes back in the queue (jobs.reap), so nothing is lost. + +What each unit costs is an estimate: the spot price in the instance's zone when it launched, or the on-demand list +price from the AWS Price List API; an admin's price on the provider wins over both. Without any of them Lens doesn't +launch, because a unit with no price would never count against its group's budget. + +Each instance is tagged `lens-unit` with its unit's name (`listing` finds the ones Lens lost by it), asks for +IMDSv2, encrypts its disk and terminates itself when it shuts down, so a machine that stops never keeps costing. Its +user-data is the mesh's install line, whose join token works once, within the hour: no long-lived secret is in it. + +It talks to AWS through botocore (Apache-2.0), the `aws` extra, fetched when the first AWS provider is saved. +""" + +from __future__ import annotations + +import datetime as dt +import json +import shlex +import threading +import time + +from . import jobs +from .scaling_drivers import Driver, DriverError, _field, register + +TAG = "lens-unit" +UBUNTU_OWNER = "099720109477" # Canonical +UBUNTU_IMAGE = "ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-{arch}-server-*" +ARCH = {"x86_64": "amd64", "arm64": "arm64"} +# What run_instances says when there's no spot to have: on demand may still be there +NO_SPOT = { + "InsufficientInstanceCapacity", + "InsufficientCapacity", + "SpotMaxPriceTooLow", + "MaxSpotInstanceCountExceeded", + "Unsupported", +} +GONE_STATES = {"shutting-down", "terminated", "stopping", "stopped"} +WHY = { + "Server.SpotInstanceTermination": "AWS took the spot instance back", + "Server.SpotInstanceShutdown": "AWS took the spot instance back", + "Server.InsufficientInstanceCapacity": "AWS had no capacity left for it", + "Server.InternalError": "AWS stopped it with an internal error", + "Client.InstanceInitiatedShutdown": "it shut itself down", + "Client.UserInitiatedShutdown": "someone stopped it at AWS", + "Client.VolumeLimitExceeded": "the account's disk limit was reached", +} +PRICE_SECONDS = 600 # prices and images are looked up again after this long +PRICING_REGION = "us-east-1" # the Price List API answers for every region from here + + +def _now(): + return dt.datetime.now(dt.timezone.utc) + + +class Ec2(Driver): + id = "aws" + kind = "node" + label = "AWS EC2" + about = ( + "Launches EC2 instances that join Lens over the mesh and run its workers: spot instances for a fraction of the " + "price, falling back to on demand when AWS has no spot capacity, or on demand only." + ) + priced = True + extra = "aws" + modules = ("botocore",) + license = "Apache-2.0" + fields = { + "region": _field("Region", required=True, default="us-east-1", help="the AWS region units launch in, such as eu-west-1"), + "access_key_id": _field( + "Access key ID", required=True, secret=True, help="an IAM user's key; docs/scaling.md has the policy it needs" + ), + "secret_access_key": _field("Secret access key", required=True, secret=True), + "subnet_id": _field("Subnet", help="the subnet units launch in; none: the region's default VPC"), + "security_group_ids": _field("Security groups", "list", help="none: the VPC's default group (units only need to reach out)"), + "key_name": _field("SSH key pair", help="a key pair's name, to sign in to units yourself; none: no SSH key"), + } + template = { + "instance_type": _field("Instance type", required=True, default="t4g.medium", help="such as t4g.medium (Arm) or m7i.large"), + "market": _field( + "Market", default="spot", options=["spot", "on-demand"], help="spot: AWS's spare capacity, which it may take back" + ), + "max_price": _field("Spot max price (USD per hour)", "number", help="none: up to the on-demand price"), + "fallback": _field("On demand when there's no spot", "boolean", default=True), + "image": _field("AMI", help="none: the latest Ubuntu 24.04 for the instance type's architecture"), + "disk_gb": _field("Disk (GB)", "integer", default=30), + "steps": _field( + "Steps", + "list", + help="the steps these workers run (transcribe, analyze, embed...); none: every step", + options=sorted(jobs.STEPS), + ), + } + + def __init__(self): + self.lock = threading.Lock() + self.clients: dict = {} + self.cache: dict = {} + + # ---------- talking to AWS ---------- + def client(self, p, service="ec2", region=None): + """A botocore client for the provider's account (one per account, region and service, reused).""" + s = p.get("secrets") or {} + region = region or (p.get("settings") or {}).get("region") or "us-east-1" + key = (service, region, s.get("access_key_id"), hash(s.get("secret_access_key"))) + with self.lock: + c = self.clients.get(key) + if c is not None: + return c + import botocore.session + from botocore.config import Config + + c = botocore.session.Session().create_client( + service, + region_name=region, + aws_access_key_id=s.get("access_key_id"), + aws_secret_access_key=s.get("secret_access_key"), + config=Config(connect_timeout=10, read_timeout=30, retries={"max_attempts": 3, "mode": "standard"}, user_agent_extra="lens"), + ) + with self.lock: + self.clients[key] = c + return c + + def _call(self, client, op, **params): + """One AWS call; what AWS said, as a DriverError, when it fails.""" + from botocore.exceptions import BotoCoreError, ClientError + + try: + return getattr(client, op)(**params) + except ClientError as e: + err = e.response.get("Error") or {} + raise AwsError(err.get("Code") or "Error", err.get("Message") or str(e), op) from None + except BotoCoreError as e: + raise DriverError(f"AWS {op}: {e}") from None + + def _cached(self, key, fn): + now = time.monotonic() + with self.lock: + hit = self.cache.get(key) + if hit and now - hit[0] < PRICE_SECONDS: + return hit[1] + value = fn() + with self.lock: + self.cache[key] = (now, value) + return value + + def _region(self, p): + return (p.get("settings") or {}).get("region") or "us-east-1" + + # ---------- what a unit is ---------- + def arch(self, p, itype): + """The instance type's architecture (x86_64 or arm64); DriverError when AWS has no such type here.""" + + def look(): + out = self._call(self.client(p), "describe_instance_types", InstanceTypes=[itype]) + types = out.get("InstanceTypes") or [] + if not types: + raise DriverError(f"AWS has no instance type {itype} in {self._region(p)}") + archs = (types[0].get("ProcessorInfo") or {}).get("SupportedArchitectures") or [] + return "arm64" if "arm64" in archs else "x86_64" + + try: + return self._cached(("arch", self._region(p), itype), look) + except AwsError as e: + if e.code == "InvalidInstanceType": + raise DriverError(f"AWS has no instance type {itype} in {self._region(p)}") from None + raise + + def image(self, p, arch): + """The newest Ubuntu 24.04 server image for this architecture in the provider's region: (id, root device).""" + + def look(): + out = self._call( + self.client(p), + "describe_images", + Owners=[UBUNTU_OWNER], + Filters=[ + {"Name": "name", "Values": [UBUNTU_IMAGE.format(arch=ARCH[arch])]}, + {"Name": "state", "Values": ["available"]}, + {"Name": "architecture", "Values": [arch]}, + ], + ) + images = sorted(out.get("Images") or [], key=lambda i: i.get("CreationDate") or "", reverse=True) + if not images: + raise DriverError(f"there's no Ubuntu 24.04 image for {arch} in {self._region(p)}: give the group an AMI") + return images[0]["ImageId"], images[0].get("RootDeviceName") or "/dev/sda1" + + return self._cached(("image", self._region(p), arch), look) + + def root_device(self, p, image_id): + out = self._call(self.client(p), "describe_images", ImageIds=[image_id]) + images = out.get("Images") or [] + if not images: + raise DriverError(f"there's no AMI {image_id} in {self._region(p)}") + return images[0].get("RootDeviceName") or "/dev/sda1" + + # ---------- prices ---------- + def spot_prices(self, p, itype, zone=None): + """The spot price now in each zone of the region (or in one zone): {zone: USD per hour}.""" + + def look(): + params = {"InstanceTypes": [itype], "ProductDescriptions": ["Linux/UNIX"], "StartTime": _now()} + if zone: + params["AvailabilityZone"] = zone + out = self._call(self.client(p), "describe_spot_price_history", **params) + prices: dict = {} + for h in out.get("SpotPriceHistory") or []: + z = h.get("AvailabilityZone") + if z and z not in prices: # the newest comes first + try: + prices[z] = float(h["SpotPrice"]) + except (KeyError, TypeError, ValueError): + continue + return prices + + return self._cached(("spot", self._region(p), itype, zone), look) + + def on_demand_price(self, p, itype): + """The on-demand list price of a Linux instance of this type in the provider's region, from the AWS Price + List API; None when it doesn't say.""" + + def look(): + pricing = self.client(p, "pricing", PRICING_REGION) + match = { + "instanceType": itype, + "regionCode": self._region(p), + "operatingSystem": "Linux", + "tenancy": "Shared", + "preInstalledSw": "NA", + "capacitystatus": "Used", + "licenseModel": "No License required", + } + out = self._call( + pricing, + "get_products", + ServiceCode="AmazonEC2", + FormatVersion="aws_v1", + Filters=[{"Type": "TERM_MATCH", "Field": k, "Value": v} for k, v in match.items()], + MaxResults=10, + ) + for item in out.get("PriceList") or []: + try: + doc = json.loads(item) if isinstance(item, str) else item + except ValueError: + continue + for term in ((doc.get("terms") or {}).get("OnDemand") or {}).values(): + for dim in (term.get("priceDimensions") or {}).values(): + try: + usd = float((dim.get("pricePerUnit") or {}).get("USD")) + except (TypeError, ValueError): + continue + if usd > 0: + return usd + return None + + try: + return self._cached(("demand", self._region(p), itype), look) + except DriverError: + return None # no pricing:GetProducts permission, say: the admin's price, or none + + def _spot_estimate(self, p, template, zone=None): + try: + prices = self.spot_prices(p, template["instance_type"], zone) + except DriverError: + return None + if not prices: + return None + price = max(prices.values()) # before launch the zone isn't known: the dearest, so budgets err high + cap = template.get("max_price") + return min(price, float(cap)) if cap else price + + def price(self, p, template): + """USD per hour of one unit, an estimate: the spot price (the dearest zone's, under the max price) or the + on-demand list price.""" + if template.get("market", "spot") == "spot": + spot = self._spot_estimate(p, template) + if spot is not None: + return spot + return self.on_demand_price(p, template["instance_type"]) + + # ---------- the calls the scaler makes ---------- + def check(self, p): + """Prove the keys work in the region with a read-only call.""" + out = self._call(self.client(p), "describe_availability_zones") + zones = sorted(z["ZoneName"] for z in out.get("AvailabilityZones") or [] if z.get("State") == "available") + return {"region": self._region(p), "zones": zones} + + def user_data(self, name, template, boot): + """The script an instance runs on its first boot: the mesh's install line, with the unit's name and the + steps its worker runs.""" + env = {**(boot.get("env") or {}), "LENS_SCALING_UNIT": name} + if template.get("steps"): + env["LENS_WORKER_STEPS"] = ",".join(template["steps"]) + lines = ["#!/bin/sh", f"# Lens scaling unit {name}: joins Lens over the mesh and runs a worker", "set -eu"] + lines += [f"export {k}={shlex.quote(str(v))}" for k, v in sorted(env.items())] + lines.append(boot["line"]) + return "\n".join(lines) + "\n" + + def launch(self, p, template, name, boot): + if not boot.get("line"): + raise DriverError("an EC2 unit joins Lens over the mesh: turn the mesh on in Settings › Network first") + st = p.get("settings") or {} + itype = template["instance_type"] + spot = template.get("market", "spot") == "spot" + own_price = p.get("price") + estimate = None if own_price is not None else self.price(p, template) + if own_price is None and estimate is None: + raise DriverError( + f"AWS didn't say what {itype} costs in {self._region(p)}: allow pricing:GetProducts for these keys, " + "or give the provider a price per hour" + ) + arch = self.arch(p, itype) + if template.get("image"): + image_id, root = template["image"], self.root_device(p, template["image"]) + else: + image_id, root = self.image(p, arch) + tags = [{"Key": "Name", "Value": name}, {"Key": TAG, "Value": name}] + params = { + "ImageId": image_id, + "InstanceType": itype, + "MinCount": 1, + "MaxCount": 1, + "UserData": self.user_data(name, template, boot), + "InstanceInitiatedShutdownBehavior": "terminate", + "MetadataOptions": {"HttpTokens": "required", "HttpEndpoint": "enabled"}, + "BlockDeviceMappings": [ + { + "DeviceName": root, + "Ebs": { + "VolumeSize": int(template.get("disk_gb") or 30), + "VolumeType": "gp3", + "Encrypted": True, + "DeleteOnTermination": True, + }, + } + ], + "TagSpecifications": [{"ResourceType": "instance", "Tags": tags}, {"ResourceType": "volume", "Tags": tags}], + } + if st.get("subnet_id"): + params["SubnetId"] = st["subnet_id"] + if st.get("security_group_ids"): + params["SecurityGroupIds"] = list(st["security_group_ids"]) + if st.get("key_name"): + params["KeyName"] = st["key_name"] + market = "on-demand" + if spot: + options: dict = {"SpotInstanceType": "one-time", "InstanceInterruptionBehavior": "terminate"} + if template.get("max_price"): + options["MaxPrice"] = f"{float(template['max_price']):.4f}" + try: + out = self._call( + self.client(p), "run_instances", **params, InstanceMarketOptions={"MarketType": "spot", "SpotOptions": options} + ) + market = "spot" + except AwsError as e: + if e.code not in NO_SPOT or template.get("fallback") is False: + raise + out = None + fell_back = e.message + if out is None: + if own_price is None: + estimate = self.on_demand_price(p, itype) + if estimate is None: + raise DriverError(f"there was no spot {itype} ({fell_back}), and AWS didn't say what on demand costs") + out = self._call(self.client(p), "run_instances", **params) + else: + out = self._call(self.client(p), "run_instances", **params) + inst = (out.get("Instances") or [{}])[0] + if not inst.get("InstanceId"): + raise DriverError("AWS launched nothing") + price = own_price + if price is None: + price = estimate + zone = (inst.get("Placement") or {}).get("AvailabilityZone") + if market == "spot" and zone: + exact = self._spot_estimate(p, template, zone) + price = exact if exact is not None else price + got = {"id": inst["InstanceId"], "address": inst.get("PrivateIpAddress"), "market": market} + if price is not None: + got["price"] = round(float(price), 6) + return got + + def describe(self, p, node): + iid = node.get("external_id") + if not iid: + return {"state": "gone"} + try: + out = self._call(self.client(p), "describe_instances", InstanceIds=[iid]) + except AwsError as e: + if e.code == "InvalidInstanceID.NotFound": + return {"state": "gone", "why": "AWS has no such instance any more"} + raise + insts = [i for r in out.get("Reservations") or [] for i in r.get("Instances") or []] + if not insts: + return {"state": "gone", "why": "AWS has no such instance any more"} + inst = insts[0] + state = (inst.get("State") or {}).get("Name") + if state in GONE_STATES: + code = (inst.get("StateReason") or {}).get("Code") + if state in ("stopping", "stopped"): + self.terminate(p, node) # it takes no jobs, and its disk would keep costing; the scaler won't call again + return {"state": "gone", "why": WHY.get(code) or f"it's {state} at AWS", "reason": code} + out_ = {"state": "running" if state == "running" else "pending"} + if inst.get("PrivateIpAddress"): + out_["address"] = inst["PrivateIpAddress"] + if inst.get("InstanceLifecycle") == "spot": + out_["market"] = "spot" + return out_ + + def terminate(self, p, node): + iid = node.get("external_id") + if not iid: + return + try: + self._call(self.client(p), "terminate_instances", InstanceIds=[iid]) + except AwsError as e: + if e.code != "InvalidInstanceID.NotFound": + raise + + def listing(self, p, tag): + client = self.client(p) + found, token = [], None + while True: + params = { + "Filters": [ + {"Name": f"tag:{TAG}", "Values": [f"{tag}-*"]}, + {"Name": "instance-state-name", "Values": ["pending", "running", "stopping", "stopped"]}, + ], + "MaxResults": 100, + } + if token: + params["NextToken"] = token + out = self._call(client, "describe_instances", **params) + for r in out.get("Reservations") or []: + for i in r.get("Instances") or []: + name = next((t.get("Value") for t in i.get("Tags") or [] if t.get("Key") == TAG), None) + found.append({"id": i["InstanceId"], "name": name}) + token = out.get("NextToken") + if not token: + return found + + +class AwsError(DriverError): + """What AWS said when a call failed: its error code and message.""" + + def __init__(self, code, message, op): + self.code, self.message, self.op = code, message, op + super().__init__(f"AWS {op}: {message} ({code})") + + +register(Ec2()) diff --git a/fastapi_backend/app/domain/scaling_drivers.py b/fastapi_backend/app/domain/scaling_drivers.py index 65209530d..4bc926660 100644 --- a/fastapi_backend/app/domain/scaling_drivers.py +++ b/fastapi_backend/app/domain/scaling_drivers.py @@ -268,3 +268,5 @@ def get(driver_id): register(Local()) + +from . import scaling_aws # noqa: E402,F401 - AWS EC2 registers itself diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml index a7675df8f..2664db492 100644 --- a/fastapi_backend/pyproject.toml +++ b/fastapi_backend/pyproject.toml @@ -69,6 +69,9 @@ sandbox = ["openshell>=0.1.4,<0.2"] # the assistant's web search through DuckDuckGo (web.py): ddgs (MIT), with its Rust HTTP client (primp) and lxml; # fetched when DuckDuckGo is picked in Settings › AI assistant › Web search web = ["ddgs>=9.16,<10"] +# auto scaling onto AWS EC2 (scaling_aws.py): botocore (Apache-2.0); fetched when the first AWS provider is saved in +# Settings › Auto scaling +aws = ["botocore>=1.43.113"] # Outlook .msg emails (extract-msg is GPL-3.0; .eml needs nothing) msg = ["extract-msg"] diff --git a/fastapi_backend/tests/domain/test_scaling_aws.py b/fastapi_backend/tests/domain/test_scaling_aws.py new file mode 100644 index 000000000..33abe8327 --- /dev/null +++ b/fastapi_backend/tests/domain/test_scaling_aws.py @@ -0,0 +1,317 @@ +"""AWS EC2 for auto scaling (scaling_aws.py, docs/scaling.md): spot or on-demand instances that join over the mesh, +with prices as estimates, a fall back to on demand when there's no spot, and AWS taking a spot instance back. AWS is +botocore's Stubber: every call is checked against EC2's own API model, and answered from here.""" + +from __future__ import annotations + +import base64 +import datetime as dt +import json + +import pytest +from botocore.session import Session +from botocore.stub import Stubber + +from app.domain import scaling, settings, store +from app.domain import scaling_drivers as drivers +from app.domain.scaling_aws import TAG, Ec2 +from tests.helpers import login, make_user + +R = store.R +NOW = dt.datetime(2026, 10, 18, 12, 0, tzinfo=dt.UTC) +LINE = "curl -fsSL https://lens.example/install.sh | LENS_JOIN=https://lens.example LENS_JOIN_TOKEN=lensjoin1.7.x sh" +P = { + "id": 1, + "name": "AWS", + "settings": {"region": "eu-west-1"}, + "secrets": {"access_key_id": "AKIA", "secret_access_key": "s"}, + "price": None, +} +SPOT = {"instance_type": "t4g.medium", "market": "spot", "fallback": True, "disk_gb": 30} + + +def _client(service, region): + return Session().create_client(service, region_name=region, aws_access_key_id="AKIA", aws_secret_access_key="s") + + +class Aws: + """An Ec2 driver whose clients are stubbed; `calls` is every call it made, with its parameters.""" + + def __init__(self, monkeypatch): + self.drv = Ec2() + self.ec2, self.pricing = _client("ec2", "eu-west-1"), _client("pricing", "us-east-1") + self.stubs = {"ec2": Stubber(self.ec2), "pricing": Stubber(self.pricing)} + self.calls = [] + clients = {"ec2": self.ec2, "pricing": self.pricing} + monkeypatch.setattr(self.drv, "client", lambda p, service="ec2", region=None: clients[service]) + real = self.drv._call + + def call(client, op, **params): + self.calls.append((op, params)) + return real(client, op, **params) + + monkeypatch.setattr(self.drv, "_call", call) + for s in self.stubs.values(): + s.activate() + + def add(self, op, response=None, service="ec2"): + self.stubs[service].add_response(op, response or {}) + + def error(self, op, code, message="no", service="ec2"): + self.stubs[service].add_client_error(op, service_error_code=code, service_message=message, http_status_code=400) + + def ops(self): + return [op for op, _ in self.calls] + + def params(self, op): + return [p for o, p in self.calls if o == op] + + def done(self): + for s in self.stubs.values(): + s.assert_no_pending_responses() + + # the answers a launch reads + def spot_prices(self, **zones): + self.add( + "describe_spot_price_history", + {"SpotPriceHistory": [{"AvailabilityZone": z.replace("_", "-"), "SpotPrice": str(v)} for z, v in zones.items()]}, + ) + + def arch(self, arch="arm64"): + self.add( + "describe_instance_types", + {"InstanceTypes": [{"InstanceType": "t4g.medium", "ProcessorInfo": {"SupportedArchitectures": [arch]}}]}, + ) + + def ubuntu(self): + self.add( + "describe_images", + { + "Images": [ + {"ImageId": "ami-old", "CreationDate": "2026-01-01T00:00:00.000Z", "RootDeviceName": "/dev/sda1"}, + {"ImageId": "ami-new", "CreationDate": "2026-09-01T00:00:00.000Z", "RootDeviceName": "/dev/sda1"}, + ] + }, + ) + + def launched(self, iid="i-1", zone="eu-west-1b", spot=True): + inst = {"InstanceId": iid, "PrivateIpAddress": "172.31.0.5", "Placement": {"AvailabilityZone": zone}} + if spot: + inst["InstanceLifecycle"] = "spot" + self.add("run_instances", {"Instances": [inst]}) + + def on_demand(self, usd="0.0368"): + doc = {"terms": {"OnDemand": {"x": {"priceDimensions": {"y": {"pricePerUnit": {"USD": usd}}}}}}} + self.add("get_products", {"PriceList": [json.dumps(doc)]}, service="pricing") + + +@pytest.fixture +def aws(monkeypatch): + return Aws(monkeypatch) + + +BOOT = {"env": {"LENS_SCALING_UNIT": "lens-workers-1"}, "line": LINE, "token_id": 7} + + +def test_a_spot_unit_launches_hardened_and_priced_in_its_zone(aws): + aws.spot_prices(eu_west_1a=0.012, eu_west_1b=0.014) # before launch: the dearest zone + aws.arch("arm64") + aws.ubuntu() + aws.launched(zone="eu-west-1a") + aws.spot_prices(eu_west_1a=0.012) # after: the zone it landed in + got = aws.drv.launch(P, {**SPOT, "steps": ["transcribe"]}, "lens-workers-1", BOOT) + aws.done() + assert got == {"id": "i-1", "address": "172.31.0.5", "market": "spot", "price": 0.012} + + (images,) = aws.params("describe_images") + assert images["Owners"] == ["099720109477"] and "ubuntu-noble-24.04-arm64-server-*" in images["Filters"][0]["Values"][0] + (run,) = aws.params("run_instances") + assert run["ImageId"] == "ami-new" and run["InstanceType"] == "t4g.medium" + assert run["InstanceMarketOptions"] == { + "MarketType": "spot", + "SpotOptions": {"SpotInstanceType": "one-time", "InstanceInterruptionBehavior": "terminate"}, + } + assert run["MetadataOptions"]["HttpTokens"] == "required" # IMDSv2 only + assert run["InstanceInitiatedShutdownBehavior"] == "terminate" # one that stops never keeps costing + assert run["BlockDeviceMappings"][0]["Ebs"] == {"VolumeSize": 30, "VolumeType": "gp3", "Encrypted": True, "DeleteOnTermination": True} + tags = {t["Key"]: t["Value"] for t in run["TagSpecifications"][0]["Tags"]} + assert tags == {"Name": "lens-workers-1", TAG: "lens-workers-1"} + # the user-data is the mesh's install line, named for the unit: its one-time token is the only credential in it + data = run["UserData"] + data = data if data.startswith("#!") else base64.b64decode(data).decode() + assert data.startswith("#!/bin/sh\n") and data.rstrip().endswith(LINE) + assert "export LENS_SCALING_UNIT=lens-workers-1" in data and "export LENS_WORKER_STEPS=transcribe" in data + + +def test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to(aws): + aws.spot_prices(eu_west_1a=0.012) + aws.arch() + aws.ubuntu() + aws.error("run_instances", "InsufficientInstanceCapacity", "no spot t4g.medium in eu-west-1") + aws.on_demand("0.0368") + aws.launched(spot=False) + got = aws.drv.launch(P, SPOT, "lens-workers-1", BOOT) + aws.done() + assert got["market"] == "on-demand" and got["price"] == 0.0368 + first, second = aws.params("run_instances") + assert "InstanceMarketOptions" in first and "InstanceMarketOptions" not in second + + aws.error("run_instances", "SpotMaxPriceTooLow", "your max price is under the spot price") + with pytest.raises(drivers.DriverError, match="SpotMaxPriceTooLow"): + aws.drv.launch(P, {**SPOT, "fallback": False, "max_price": 0.005}, "lens-workers-2", BOOT) # arch and image are cached + aws.done() + assert aws.params("run_instances")[-1]["InstanceMarketOptions"]["SpotOptions"]["MaxPrice"] == "0.0050" + + +def test_a_unit_without_a_price_or_the_mesh_launches_nothing(aws): + with pytest.raises(drivers.DriverError, match="mesh"): + aws.drv.launch(P, SPOT, "lens-workers-1", {**BOOT, "line": None}) + assert aws.calls == [] # no mesh: not even a call to AWS + + aws.error("get_products", "AccessDeniedException", "not allowed", service="pricing") + with pytest.raises(drivers.DriverError, match="pricing:GetProducts"): + aws.drv.launch(P, {**SPOT, "market": "on-demand"}, "lens-workers-1", BOOT) + aws.done() + assert "run_instances" not in aws.ops() # a unit with no price would never count against the budget + + # the admin's own price on the provider wins, and no price is looked up + aws.arch("x86_64") + aws.ubuntu() + aws.launched(spot=False) + got = aws.drv.launch({**P, "price": 0.1}, {**SPOT, "market": "on-demand", "instance_type": "m7i.large"}, "lens-workers-1", BOOT) + aws.done() + assert got["price"] == 0.1 and "get_products" not in aws.ops()[-3:] + assert "ubuntu-noble-24.04-amd64-server-*" in aws.params("describe_images")[-1]["Filters"][0]["Values"][0] + + +def test_describe_says_when_aws_took_a_spot_instance_back(aws): + node = {"external_id": "i-1", "name": "lens-workers-1"} + reservation = lambda state, code=None: { # noqa: E731 + "Reservations": [ + { + "Instances": [ + { + "InstanceId": "i-1", + "State": {"Name": state}, + "PrivateIpAddress": "172.31.0.5", + "InstanceLifecycle": "spot", + **({"StateReason": {"Code": code, "Message": code}} if code else {}), + } + ] + } + ] + } + aws.add("describe_instances", reservation("pending")) + aws.add("describe_instances", reservation("running")) + aws.add("describe_instances", reservation("terminated", "Server.SpotInstanceTermination")) + aws.add("describe_instances", reservation("stopped", "Client.UserInitiatedShutdown")) + aws.add("terminate_instances", {}) # a stopped one takes no jobs and its disk keeps costing: terminated + aws.error("describe_instances", "InvalidInstanceID.NotFound") + assert aws.drv.describe(P, node)["state"] == "pending" + assert aws.drv.describe(P, node) == {"state": "running", "address": "172.31.0.5", "market": "spot"} + assert aws.drv.describe(P, node) == { + "state": "gone", + "why": "AWS took the spot instance back", + "reason": "Server.SpotInstanceTermination", + } + assert aws.drv.describe(P, node)["why"] == "someone stopped it at AWS" + assert aws.drv.describe(P, node)["state"] == "gone" + aws.done() + assert aws.params("terminate_instances") == [{"InstanceIds": ["i-1"]}] + + +def test_terminate_check_and_listing(aws): + aws.add("terminate_instances", {"TerminatingInstances": [{"InstanceId": "i-1"}]}) + aws.error("terminate_instances", "InvalidInstanceID.NotFound") # already gone: no error + aws.drv.terminate(P, {"external_id": "i-1"}) + aws.drv.terminate(P, {"external_id": "i-1"}) + aws.drv.terminate(P, {"external_id": None}) # never launched: nothing to call + + aws.add( + "describe_availability_zones", + {"AvailabilityZones": [{"ZoneName": "eu-west-1b", "State": "available"}, {"ZoneName": "eu-west-1a", "State": "available"}]}, + ) + assert aws.drv.check(P) == {"region": "eu-west-1", "zones": ["eu-west-1a", "eu-west-1b"]} + + tagged = lambda iid, name: {"InstanceId": iid, "Tags": [{"Key": TAG, "Value": name}]} # noqa: E731 + aws.add("describe_instances", {"Reservations": [{"Instances": [tagged("i-1", "lens-a-1")]}], "NextToken": "n"}) + aws.add("describe_instances", {"Reservations": [{"Instances": [tagged("i-2", "lens-a-2")]}]}) + assert aws.drv.listing(P, "lens") == [{"id": "i-1", "name": "lens-a-1"}, {"id": "i-2", "name": "lens-a-2"}] + assert aws.params("describe_instances")[0]["Filters"][0] == {"Name": f"tag:{TAG}", "Values": ["lens-*"]} + assert aws.params("describe_instances")[1]["NextToken"] == "n" + aws.done() + + +def test_the_driver_is_offered_and_its_settings_are_checked(): + drv = drivers.get("aws") + spec = drv.spec() + assert (spec["kind"], spec["priced"], spec["extra"], spec["license"]) == ("node", True, "aws", "Apache-2.0") + assert set(drv.secrets) == {"access_key_id", "secret_access_key"} + assert drivers.check_template(drv, {}) == {"instance_type": "t4g.medium", "market": "spot", "fallback": True, "disk_gb": 30} + with pytest.raises(ValueError, match="market"): + drivers.check_template(drv, {"market": "reserved"}) + + +# ---------- through the scaler ---------- +@pytest.fixture +def h(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +def test_the_scaler_replaces_a_spot_unit_aws_took_back(client, db, cfg, h, aws, monkeypatch): + monkeypatch.setitem(drivers.DRIVERS, "aws", aws.drv) + monkeypatch.setattr(scaling, "bootstrap", lambda db, cfg, g, name: {"env": {"LENS_SCALING_UNIT": name}, "line": LINE, "token_id": None}) + settings.save(db, cfg, "scaling", {"enabled": True}, "root@x.io") + cfg = settings.effective(db, cfg) + aws.add("describe_availability_zones", {"AvailabilityZones": []}) # the check when it's saved + body = { + "name": "AWS", + "driver": "aws", + "settings": {"region": "eu-west-1"}, + "secrets": {"access_key_id": "AKIA", "secret_access_key": "s"}, + } + r = client.post("/api/v1/scaling/providers", headers=h, json=body) + assert r.status_code == 201, r.text + assert r.json()["secrets"] == {"access_key_id": True, "secret_access_key": True} + g = client.post( + "/api/v1/scaling/groups", + headers=h, + json={ + "name": "Workers", + "provider": r.json()["id"], + "min": 1, + "max": 2, + "budget": 20, + "template": SPOT, + "cooldown_seconds": 0, + "warmup_seconds": 0, + }, + ) + assert g.status_code == 201, g.text + gid = g.json()["id"] + + aws.spot_prices(eu_west_1a=0.012) # the scaler's estimate before launching (then cached) + aws.arch() + aws.ubuntu() + aws.launched("i-1", "eu-west-1a") + aws.spot_prices(eu_west_1a=0.012) # the zone it landed in + tick = lambda now: scaling.Scaler(db, lambda: cfg, "test").tick(now=now, lease=False) # noqa: E731 + assert tick(NOW)["launched"] == 1 + (unit,) = scaling.nodes(db, gid) + assert (unit["external_id"], unit["price"], unit["state"]) == ("i-1", 0.012, "launching") + + # its worker joins, then AWS takes the spot instance back: the scaler says why and launches another + db.q("UPSERT $w MERGE {name: $n, heartbeat_at: $t}", w=R("worker", unit["name"]), n=unit["name"], t=NOW.isoformat()) + aws.add("describe_instances", {"Reservations": [{"Instances": [{"InstanceId": "i-1", "State": {"Name": "running"}}]}]}) + tick(NOW + dt.timedelta(seconds=30)) + assert scaling.nodes(db, gid)[0]["state"] == "running" + gone = {"InstanceId": "i-1", "State": {"Name": "terminated"}, "StateReason": {"Code": "Server.SpotInstanceTermination", "Message": "x"}} + aws.add("describe_instances", {"Reservations": [{"Instances": [gone]}]}) + aws.launched("i-2", "eu-west-1a") # prices, the image and the architecture are cached + did = tick(NOW + dt.timedelta(minutes=5)) + aws.done() + assert (did["terminated"], did["launched"]) == (1, 1) + old, new = sorted(scaling.nodes(db, gid), key=lambda n: n["id"]) + assert old["state"] == "terminated" and old["why"] == "AWS took the spot instance back" + assert new["external_id"] == "i-2" and new["state"] == "launching" diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock index 9842acede..949052cce 100644 --- a/fastapi_backend/uv.lock +++ b/fastapi_backend/uv.lock @@ -1949,6 +1949,9 @@ dependencies = [ ] [package.optional-dependencies] +aws = [ + { name = "botocore" }, +] doctr = [ { name = "python-doctr" }, ] @@ -2054,6 +2057,7 @@ dev = [ [package.metadata] requires-dist = [ + { name = "botocore", marker = "extra == 'aws'", specifier = ">=1.43.113" }, { name = "cryptography", specifier = ">=50" }, { name = "ddgs", marker = "extra == 'web'", specifier = ">=9.16,<10" }, { name = "extract-msg", marker = "extra == 'msg'" }, @@ -2108,7 +2112,7 @@ requires-dist = [ { name = "transformers", marker = "extra == 'punctuation'" }, { name = "webauthn", specifier = ">=3.0.1,<4" }, ] -provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "spacy", "injection", "florence", "moondream", "validate", "smb", "sandbox", "web", "msg"] +provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "spacy", "injection", "florence", "moondream", "validate", "smb", "sandbox", "web", "aws", "msg"] [package.metadata.requires-dev] dev = [ diff --git a/threat-model.md b/threat-model.md index 53610f9e7..016817855 100644 --- a/threat-model.md +++ b/threat-model.md @@ -535,7 +535,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-MEDIA-LINK | Media, frames, documents and reports open without a session only through a link the server signed for that path, or a live share link. Text people or models write is never signed. | info-leak | security-critical | *(documented, docs/architecture.md "is never signed")* | Voided by: Knowing `ACCESS_SECRET_KEY` signs any path — `core/security.py:43`; dev compose key — `docker-compose.yml:24`. Lifetime `MEDIA_URL_EXPIRE_SECONDS` — `config.py:30`. Report pages sign every asset of the namespace — `api/pages.py:140-141`. Signing is limited to allow-listed fields — `api/media.py:25-30`. | | P-SHARE | A share link opens one asset, read-only. It expires and can be revoked. Only hashes are stored. A dead link answers the same 410 whatever went wrong. | integrity-bypass | security-critical | *(documented, docs/authentication.md "give read-only access to one asset's player and embed")* | Voided by: Liveness plus asset match only — `domain/auth.py:441-444`; `?s=` routes — `api/deps.py:252` (Q7). | | P-FEEDS | A published feed's address opens that feed alone, read-only, with no session: the feed's XML and the audio of items it lists. Only an editor of the namespace makes a feed, and what it lists is bounded by who already reads that namespace: an item's resource or activity is in the namespace, or in one every reader of the feed's namespace reads too, never a vault's, and a namespace made a vault stops its feeds. The token is found by its hash; rotating or deleting the feed stops the old address at once. Text in an item cannot add markup to a reader. | info-leak | security-critical | *(documented, docs/feeds.md "Private by default")* | Voided by: the token is sealed with the server key, not only hashed, so the namespace's readers (and whoever holds `secret.key`) see the address again — `domain/syndication.py` `token_of`; anyone given the address reads the feed until it is rotated; editors choose what goes in, as does a routine (admins make them) whose workflow nodes publish what they pick, bounded the same way; a resource's title and summary show as they are now, so an edit after publishing shows; the fit is by namespace roles (`domain/syndication.py` `Fit`), not per-asset grants, so an asset later closed by access level still shows while its namespace fits. | -| P-SCALING | Auto scaling launches nothing that costs money unless a person, signed in to the web app, approved the group's terms: its provider, what each unit is, its max, its budget and what happens at its cap. An API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away. A provider's credentials are sealed per provider and field, opened only for a call to it, and never shown. A unit is handed no long-lived Lens secret to start. | integrity-bypass | security-critical | *(documented, docs/scaling.md "The person approves")* | Voided by: an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (`domain/scaling.py` `accrue`), not bills, so a provider can charge more than the budget counts. | +| P-SCALING | Auto scaling launches nothing that costs money unless a person, signed in to the web app, approved the group's terms: its provider, what each unit is, its max, its budget and what happens at its cap. An API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away. A provider's credentials are sealed per provider and field, opened only for a call to it, and never shown. A unit is handed no long-lived Lens secret to start. | integrity-bypass | security-critical | *(documented, docs/scaling.md "The person approves")* | Voided by: an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (`domain/scaling.py` `accrue`), not bills, so a provider can charge more than the budget counts. A cloud unit's user-data holds its one-time join token until it joins or the hour runs out: an AWS EC2 unit's is readable by the account's principals allowed `ec2:DescribeInstanceAttribute` and by anything running on the instance. | | P-AUTH-REQ | A request is authenticated only if its bearer verifies and the account and session are live. Sign-out and disabling take effect at once, for every token kind. Revoking an app grant or API token ends its tokens at once. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Every request checks that the account is still active")* | Voided by: Liveness checked per request — `api/deps.py:92`; JWT algorithm from `ALGORITHM` — `config.py:27`; dev secrets — `docker-compose.yml:24`. No bypass flag: `grep -rniE "disable_auth\|skip_auth\|no_auth\|auth_bypass\|AUTH_DISABLED\|ALLOW_ANON\|DEV_MODE" fastapi_backend/app nextjs-frontend/lib nextjs-frontend/auth.ts nextjs-frontend/proxy.ts docker-compose*.yml install.sh` — 0 hits. | | P-REMOVE | Removing an account ends every way it signs in at once (sessions, passkeys, sign-in links, connected accounts, API tokens, app grants, IIIF tokens) and takes every role, collection role and recording permission. Its email can make a new account, which inherits nothing. Admins only; never yourself, never the Matterbridge account, never the last passkey into a vault unless the admin says its files may be lost. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Removing someone")* | Voided by: the account row stays as a marker with an `.invalid` email — `domain/auth.py` `remove_account`; shared notes, views and comments keep its id. | | P-BEARER-ONLY | `/api/v1` takes bearer tokens only, never cookies. The IIIF authorization flow and the web app's NextAuth handlers are the cookie surfaces. | integrity-bypass | security-critical | *(documented, docs/architecture.md "so there is no CSRF surface")* | Voided by: IIIF auth cookies — `api/iiif.py:232`, `api/iiif.py:333`; external-login state cookie — `api/v1/routes/external.py:34`. | diff --git a/threat-model.yaml b/threat-model.yaml index 04d05c44d..344a784ad 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -1547,7 +1547,7 @@ properties_claimed: tier: security-critical conditions: "auto scaling launches nothing unless a person signed in to the web app approved the group's terms (its provider, what each unit is, its max, its budget, what happens at its cap); an API key, an app, MCP and the assistant acting on its own never approve, and a change that widens the terms takes the approval away; a provider's credentials are sealed per provider and field, opened only for a call to it and never shown; a unit is handed no long-lived Lens secret" violation_symptoms: [integrity-bypass] - x-voided-by: "an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (domain/scaling.py accrue), not bills, so a provider can charge more than the budget counts" + x-voided-by: "an admin's web session approves, and an admin is trusted; a yes on the Approvals page to the assistant's proposal approves the terms that proposal names; within the terms agents may raise the desired count up to max, spending up to the budget; costs are estimates from list prices (domain/scaling.py accrue), not bills, so a provider can charge more than the budget counts; a cloud unit's user-data holds its one-time join token until it joins or the hour runs out: an AWS EC2 unit's is readable by the account's principals allowed ec2:DescribeInstanceAttribute and by anything running on the instance" provenance: {kind: documented, source: 'docs/scaling.md "The person approves"'} - id: P-AUTH-REQ kind: authentication From 4460b97295e023b6298f1a0e0e89f795fdb9ac8e Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 07:55:11 +0000 Subject: [PATCH 2/2] docs(readme): list answering questions about Lens itself as built (port of #578) The requirements check fails on main since #400 was released; the same change as #578, a no-op once that merges. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SETHpH6NUGKT1Bwd6xyPCA --- README.md | 7 +++---- docs/requirements.md | 1 + 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3af487c27..5ed6968e7 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p [API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md); [checks](docs/requirements.md#connect-your-agents)) - **Ask, and get cited answers.** A chat assistant on every page, and by voice, searches your archive by words (and by - meaning, once turned on), reads it, cites it and acts on it with tools: changes wait for your approval. Once an admin + meaning, once turned on), reads it, cites it and acts on it with tools: changes wait for your approval. It answers + questions about Lens itself from its docs and this server's settings. Once an admin turns them on, it also searches the web and reads pages, cited like the archive, and runs Python and bash in a sandbox of each conversation's own. An admin's assistant searches and reads the mail, calendars and files connected to Lens (Gmail, Google Calendar, Google Drive, S3 and the rest), cites messages and events, and brings them in once you @@ -92,7 +93,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages), - [mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records), + [mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [questions about Lens](docs/assistant.md#questions-about-lens-itself), [answer records](docs/assistant.md#what-each-answer-records), [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md); [checks](docs/requirements.md#ask-and-get-cited-answers)) - **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python), @@ -241,8 +242,6 @@ still loading. ([Components](docs/components.md), These are open pull requests, not yet part of Lens: -- **The assistant** answering questions about Lens itself, from its docs and this server - ([#400](https://github.com/adeelahmad/lense/pull/400)). - **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)). - **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)). - **Web clipper:** signing in, or using your API token, not only an import webhook's diff --git a/docs/requirements.md b/docs/requirements.md index 4cf065e74..1799a2373 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -76,6 +76,7 @@ request, to another check that proves the same thing; loosening a requirement or |---|---| | A chat assistant on every page | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_asking_from_a_page`; [page-chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/page-chat.test.tsx); [ask-place.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ask-place.test.tsx) | | By voice | [assistant-mode.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/assistant-mode.test.tsx) `starts a voice chat from the mic`; [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_what_is_said_comes_back_as_text` | +| It answers questions about Lens itself from its docs and this server's settings, for agents over MCP too | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_assistant_answers_about_lens_itself`, `test_admins_see_where_each_setting_is`, `test_agents_ask_lens_over_mcp` | | It searches by words, and by meaning once search by meaning is set up | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_semantic.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_semantic.py) `test_chat_retrieval_by_meaning`, `test_off_or_unconfigured` | | Answers cite their sources, and only what the person may read | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_chat_cites_only_what_you_can_read`; [chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat.test.tsx) `splits [n], [n, m] and [n][m]` | | It acts with tools, and changes wait for approval | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_the_assistant_lists_and_declines_but_never_approves` |