diff --git a/README.md b/README.md index a5eb642c..e48e1820 100644 --- a/README.md +++ b/README.md @@ -186,8 +186,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today; - DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching once the mesh's install - line lands), and AWS, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); + AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, launching + once the mesh's install line lands), and Linux and model hosts are planned. ([Auto scaling](docs/scaling.md); [checks](docs/requirements.md#scale-out)) - **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine, @@ -264,7 +264,7 @@ Next, and not in a pull request yet; each links to where it is tracked: topics, recordings and the other things that have pages ([notes](docs/notes.md)). - **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models ([local models](docs/local-models.md)). -- **Auto scaling:** providers for AWS and Linux hosts over SSH, joining over the Nebula mesh, model hosts +- **Auto scaling:** providers for Linux hosts over SSH, joining over the Nebula mesh, model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings ([auto scaling](docs/scaling.md)). - **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the diff --git a/docs/requirements.md b/docs/requirements.md index 6fa07c2e..89826505 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -228,6 +228,9 @@ request, to another check that proves the same thing; loosening a requirement or | This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` | | DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` | | Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` | +| AWS EC2 units launch as spot or on-demand instances, hardened (IMDSv2, encrypted disk, terminate on shutdown), with only the mesh's one-time join line in their user-data | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_a_spot_unit_launches_hardened_and_priced_in_its_zone`, `test_the_driver_is_offered_and_its_settings_are_checked` | +| A spot unit falls back to on demand when AWS has none; without a price or the mesh, nothing launches | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to`, `test_a_unit_without_a_price_or_the_mesh_launches_nothing` | +| When AWS takes a spot instance back, the scaler says why and launches another | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_describe_says_when_aws_took_a_spot_instance_back`, `test_the_scaler_replaces_a_spot_unit_aws_took_back`, `test_terminate_check_and_listing` | | Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` | ## Keep it up to date and backed up diff --git a/docs/scaling.md b/docs/scaling.md index ed807244..c5945335 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -8,11 +8,11 @@ launched. Everything it does is in the [activity ledger](activity.md), and its c Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools, Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that -runs more `lens worker` processes where Lens runs. In progress: **DigitalOcean** and **Hetzner Cloud** providers are -built (credentials checked, list prices, launching, watching and deleting machines), and launch once the mesh's -install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is **Proxmox VE**, clones of a -VM template on your own server. Planned: providers for AWS and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making -providers and groups in Settings. +runs more `lens worker` processes where Lens runs. In progress: **AWS EC2** (spot or on demand), **DigitalOcean** +and **Hetzner Cloud** providers are built (credentials checked, prices, launching, watching and deleting machines), +and launch once the mesh's install line, which another machine joins Lens with, lands ([the mesh](mesh.md)); so is +**Proxmox VE**, clones of a VM template on your own server. Planned: providers for any Linux host over SSH; model hosts +(RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings. Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms signed in to the web app. Agents can propose and manage groups, but never approve one. @@ -55,6 +55,7 @@ carries none of them. | Driver | Units | Costs | Template | | --- | --- | --- | --- | | `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) | +| `aws` (AWS EC2) | EC2 instances that join over the mesh and run `lens worker` | the spot or on-demand price, an estimate | `instance_type`, `market` (`spot` or `on-demand`), `max_price`, `fallback`, `image`, `disk_gb`, `steps` | | `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` | | `proxmox` (Proxmox VE) | clones of a VM template on `node`, at `url` with an API token (`token_id`, `token_secret` sealed); `verify_tls` (on) and `ca_cert` for a private CA | nothing, unless you give it a price | `vm_template` (its id), `full_clone` (off: linked), `storage`, `pool`, `steps` | | `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` | @@ -82,6 +83,66 @@ enough). Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing. +## AWS EC2 + +Lens grows onto AWS when work waits: from a Raspberry Pi at home to as many EC2 instances as the group's max and +budget allow, and back to none. Each unit is an instance that joins Lens over the [mesh](mesh.md) and runs its worker. + +```http +POST /api/v1/scaling/providers +{"name": "AWS", "driver": "aws", "settings": {"region": "eu-west-1"}, + "secrets": {"access_key_id": "AKIA...", "secret_access_key": "..."}} +``` + +| Provider setting | Default | | +| --- | --- | --- | +| `region` | `us-east-1` | the region units launch in | +| `access_key_id`, `secret_access_key` | | an IAM user's keys, sealed | +| `subnet_id` | the default VPC's | where units launch | +| `security_group_ids` | the VPC's default group | units only reach out (to the mesh's lighthouse), so they need no inbound rule | +| `key_name` | none | an EC2 key pair, to sign in to units yourself | + +| Unit (template) | Default | | +| --- | --- | --- | +| `instance_type` | `t4g.medium` | any EC2 type; Arm (Graviton) and x86 both work | +| `market` | `spot` | `spot`: AWS's spare capacity, often far cheaper, which AWS may take back with two minutes' notice; `on-demand` | +| `max_price` | the on-demand price | the most a spot unit may cost per hour | +| `fallback` | `true` | when AWS has no spot capacity of that type, or none under the max price, launch on demand instead | +| `image` | the latest Ubuntu 24.04 | an AMI; by default Canonical's for the type's architecture, found in the region | +| `disk_gb` | `30` | the unit's disk (gp3, encrypted) | +| `steps` | every step | the steps its worker runs | + +**Spot instances.** When AWS takes a spot instance back, the scaler sees it gone, records why ("AWS took the spot +instance back") and launches another, on demand if spot has run out and `fallback` allows. The job its worker was +running goes back in the queue once its heartbeat stops, so nothing is lost. + +**What it costs.** Each unit's price is an estimate: the spot price in the zone it landed in (from +`DescribeSpotPriceHistory`), or the on-demand list price (from the AWS Price List API). A price you give the provider +wins over both. Without any price Lens launches nothing, because a unit with no price would never count against the +budget. AWS bills separately; the ledger's figures are estimates. + +**Hardened by default.** Every instance requires IMDSv2, has an encrypted disk that goes with it, terminates itself +when it shuts down (a stopped one is terminated too, so its disk doesn't keep costing), and is tagged `lens-unit` with +its name and `lens` with Lens's tag. Its user-data is the one every cloud unit boots with ([machines +elsewhere](#machines-elsewhere)). + +The keys need these permissions (an IAM policy for a user only Lens uses): + +```json +{ + "Version": "2012-10-17", + "Statement": [ + {"Effect": "Allow", "Resource": "*", "Action": [ + "ec2:DescribeAvailabilityZones", "ec2:DescribeInstanceTypes", "ec2:DescribeImages", + "ec2:DescribeSpotPriceHistory", "ec2:DescribeInstances", "ec2:RunInstances", + "ec2:CreateTags", "ec2:TerminateInstances", "pricing:GetProducts"]} + ] +} +``` + +`pricing:GetProducts` is only needed for on-demand prices you don't give yourself. Lens talks to AWS through botocore +(Apache-2.0), fetched the first time an AWS provider is saved. + ## Groups ```http @@ -170,7 +231,10 @@ Turning scaling on (`change_settings scaling`) always asks. ## Refine later -Providers for SSH hosts and model hosts; Proxmox LXC containers; the AWS provider (in its own pull request); revoking a unit's mesh -certificate when it goes; reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro -rate; spot and preemptible instances; warm pools; predictive scaling; health checks beyond +Providers for SSH hosts and model hosts; Proxmox LXC containers; revoking a unit's mesh certificate when it goes; +reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro rate; preemptible and spot +instances at clouds other than AWS; draining an AWS spot unit on its two-minute notice (the node agent reading the +instance's metadata) and handing its job back at once; following the spot price as it changes; AWS sign-in with an +instance role or IAM Identity Center instead of keys; mixed instance types per group; warm pools; predictive scaling; +health checks beyond joining; prices fetched from providers' pricing APIs. diff --git a/fastapi_backend/app/domain/scaling.py b/fastapi_backend/app/domain/scaling.py index dd492862..bfe2635b 100644 --- a/fastapi_backend/app/domain/scaling.py +++ b/fastapi_backend/app/domain/scaling.py @@ -1188,7 +1188,7 @@ def _reconcile(self, cfg, g, drv, p, n, now, did): db.q("UPDATE $r SET error = $e", r=R("scale_node", n["id"]), e=f"{type(e).__name__}: {e}"[:300]) return True if d.get("state") == "gone": - why = "stopped at the provider" if state != "launching" else "never came up at the provider" + why = d.get("why") or ("stopped at the provider" if state != "launching" else "never came up at the provider") _terminate(db, cfg, drv, p, {**n, "external_id": None}, why, state="failed" if state == "launching" else "terminated", now=now) did["terminated"] += 1 return False diff --git a/fastapi_backend/app/domain/scaling_aws.py b/fastapi_backend/app/domain/scaling_aws.py new file mode 100644 index 00000000..3ec7bd98 --- /dev/null +++ b/fastapi_backend/app/domain/scaling_aws.py @@ -0,0 +1,449 @@ +"""AWS EC2 for auto scaling (scaling.py, docs/scaling.md): Lens launches EC2 instances that join it over the mesh and +run `lens worker`, as spot instances (AWS's spare capacity, a fraction of the on-demand price, which AWS may take back +with two minutes' notice) or on demand. + +A spot unit falls back to on demand when AWS has no spot capacity of that type, or none under the group's max price, +unless the group says not to. When AWS takes a spot instance back, `describe` says so and the scaler launches another; +the job its worker was running goes back in the queue (jobs.reap), so nothing is lost. + +What each unit costs is an estimate: the spot price in the instance's zone when it launched, or the on-demand list +price from the AWS Price List API; an admin's price on the provider wins over both. Without any of them Lens doesn't +launch, because a unit with no price would never count against its group's budget. + +Each instance is tagged `lens-unit` with its unit's name (`listing` finds the ones Lens lost by it) and `lens` with +Lens's tag, asks for IMDSv2, encrypts its disk and terminates itself when it shuts down, so a machine that stops never +keeps costing. Its user-data is every cloud unit's (scaling_drivers.user_data): the mesh's install line, whose join +token works once, within the hour, so no long-lived secret is in it. + +It talks to AWS through botocore (Apache-2.0), the `aws` extra, fetched when the first AWS provider is saved. +""" + +from __future__ import annotations + +import datetime as dt +import json +import threading +import time + +from . import jobs +from . import scaling_drivers as drivers +from .scaling_drivers import Driver, DriverError, _field, register + +TAG = "lens-unit" +UBUNTU_OWNER = "099720109477" # Canonical +UBUNTU_IMAGE = "ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-{arch}-server-*" +ARCH = {"x86_64": "amd64", "arm64": "arm64"} +# What run_instances says when there's no spot to have: on demand may still be there +NO_SPOT = { + "InsufficientInstanceCapacity", + "InsufficientCapacity", + "SpotMaxPriceTooLow", + "MaxSpotInstanceCountExceeded", + "Unsupported", +} +GONE_STATES = {"shutting-down", "terminated", "stopping", "stopped"} +WHY = { + "Server.SpotInstanceTermination": "AWS took the spot instance back", + "Server.SpotInstanceShutdown": "AWS took the spot instance back", + "Server.InsufficientInstanceCapacity": "AWS had no capacity left for it", + "Server.InternalError": "AWS stopped it with an internal error", + "Client.InstanceInitiatedShutdown": "it shut itself down", + "Client.UserInitiatedShutdown": "someone stopped it at AWS", + "Client.VolumeLimitExceeded": "the account's disk limit was reached", +} +PRICE_SECONDS = 600 # prices and images are looked up again after this long +PRICING_REGION = "us-east-1" # the Price List API answers for every region from here + + +def _now(): + return dt.datetime.now(dt.timezone.utc) + + +class Ec2(Driver): + id = "aws" + kind = "node" + label = "AWS EC2" + about = ( + "Launches EC2 instances that join Lens over the mesh and run its workers: spot instances for a fraction of the " + "price, falling back to on demand when AWS has no spot capacity, or on demand only." + ) + priced = True + needs_join = True # its units are EC2 instances, which join over the mesh's install line + extra = "aws" + modules = ("botocore",) + license = "Apache-2.0" + fields = { + "region": _field("Region", required=True, default="us-east-1", help="the AWS region units launch in, such as eu-west-1"), + "access_key_id": _field( + "Access key ID", required=True, secret=True, help="an IAM user's key; docs/scaling.md has the policy it needs" + ), + "secret_access_key": _field("Secret access key", required=True, secret=True), + "subnet_id": _field("Subnet", help="the subnet units launch in; none: the region's default VPC"), + "security_group_ids": _field("Security groups", "list", help="none: the VPC's default group (units only need to reach out)"), + "key_name": _field("SSH key pair", help="a key pair's name, to sign in to units yourself; none: no SSH key"), + } + template = { + "instance_type": _field("Instance type", required=True, default="t4g.medium", help="such as t4g.medium (Arm) or m7i.large"), + "market": _field( + "Market", default="spot", options=["spot", "on-demand"], help="spot: AWS's spare capacity, which it may take back" + ), + "max_price": _field("Spot max price (USD per hour)", "number", help="none: up to the on-demand price"), + "fallback": _field("On demand when there's no spot", "boolean", default=True), + "image": _field("AMI", help="none: the latest Ubuntu 24.04 for the instance type's architecture"), + "disk_gb": _field("Disk (GB)", "integer", default=30), + "steps": _field( + "Steps", + "list", + help="the steps these workers run (transcribe, analyze, embed...); none: every step", + options=sorted(jobs.STEPS), + ), + } + + def __init__(self): + self.lock = threading.Lock() + self.clients: dict = {} + self.cache: dict = {} + + # ---------- talking to AWS ---------- + def client(self, p, service="ec2", region=None): + """A botocore client for the provider's account (one per account, region and service, reused).""" + s = p.get("secrets") or {} + region = region or (p.get("settings") or {}).get("region") or "us-east-1" + key = (service, region, s.get("access_key_id"), hash(s.get("secret_access_key"))) + with self.lock: + c = self.clients.get(key) + if c is not None: + return c + import botocore.session + from botocore.config import Config + + c = botocore.session.Session().create_client( + service, + region_name=region, + aws_access_key_id=s.get("access_key_id"), + aws_secret_access_key=s.get("secret_access_key"), + config=Config(connect_timeout=10, read_timeout=30, retries={"max_attempts": 3, "mode": "standard"}, user_agent_extra="lens"), + ) + with self.lock: + self.clients[key] = c + return c + + def _call(self, client, op, **params): + """One AWS call; what AWS said, as a DriverError, when it fails.""" + from botocore.exceptions import BotoCoreError, ClientError + + try: + return getattr(client, op)(**params) + except ClientError as e: + err = e.response.get("Error") or {} + raise AwsError(err.get("Code") or "Error", err.get("Message") or str(e), op) from None + except BotoCoreError as e: + raise DriverError(f"AWS {op}: {e}") from None + + def _cached(self, key, fn): + now = time.monotonic() + with self.lock: + hit = self.cache.get(key) + if hit and now - hit[0] < PRICE_SECONDS: + return hit[1] + value = fn() + with self.lock: + self.cache[key] = (now, value) + return value + + def _region(self, p): + return (p.get("settings") or {}).get("region") or "us-east-1" + + # ---------- what a unit is ---------- + def arch(self, p, itype): + """The instance type's architecture (x86_64 or arm64); DriverError when AWS has no such type here.""" + + def look(): + out = self._call(self.client(p), "describe_instance_types", InstanceTypes=[itype]) + types = out.get("InstanceTypes") or [] + if not types: + raise DriverError(f"AWS has no instance type {itype} in {self._region(p)}") + archs = (types[0].get("ProcessorInfo") or {}).get("SupportedArchitectures") or [] + return "arm64" if "arm64" in archs else "x86_64" + + try: + return self._cached(("arch", self._region(p), itype), look) + except AwsError as e: + if e.code == "InvalidInstanceType": + raise DriverError(f"AWS has no instance type {itype} in {self._region(p)}") from None + raise + + def image(self, p, arch): + """The newest Ubuntu 24.04 server image for this architecture in the provider's region: (id, root device).""" + + def look(): + out = self._call( + self.client(p), + "describe_images", + Owners=[UBUNTU_OWNER], + Filters=[ + {"Name": "name", "Values": [UBUNTU_IMAGE.format(arch=ARCH[arch])]}, + {"Name": "state", "Values": ["available"]}, + {"Name": "architecture", "Values": [arch]}, + ], + ) + images = sorted(out.get("Images") or [], key=lambda i: i.get("CreationDate") or "", reverse=True) + if not images: + raise DriverError(f"there's no Ubuntu 24.04 image for {arch} in {self._region(p)}: give the group an AMI") + return images[0]["ImageId"], images[0].get("RootDeviceName") or "/dev/sda1" + + return self._cached(("image", self._region(p), arch), look) + + def root_device(self, p, image_id): + out = self._call(self.client(p), "describe_images", ImageIds=[image_id]) + images = out.get("Images") or [] + if not images: + raise DriverError(f"there's no AMI {image_id} in {self._region(p)}") + return images[0].get("RootDeviceName") or "/dev/sda1" + + # ---------- prices ---------- + def spot_prices(self, p, itype, zone=None): + """The spot price now in each zone of the region (or in one zone): {zone: USD per hour}.""" + + def look(): + params = {"InstanceTypes": [itype], "ProductDescriptions": ["Linux/UNIX"], "StartTime": _now()} + if zone: + params["AvailabilityZone"] = zone + out = self._call(self.client(p), "describe_spot_price_history", **params) + prices: dict = {} + for h in out.get("SpotPriceHistory") or []: + z = h.get("AvailabilityZone") + if z and z not in prices: # the newest comes first + try: + prices[z] = float(h["SpotPrice"]) + except (KeyError, TypeError, ValueError): + continue + return prices + + return self._cached(("spot", self._region(p), itype, zone), look) + + def on_demand_price(self, p, itype): + """The on-demand list price of a Linux instance of this type in the provider's region, from the AWS Price + List API; None when it doesn't say.""" + + def look(): + pricing = self.client(p, "pricing", PRICING_REGION) + match = { + "instanceType": itype, + "regionCode": self._region(p), + "operatingSystem": "Linux", + "tenancy": "Shared", + "preInstalledSw": "NA", + "capacitystatus": "Used", + "licenseModel": "No License required", + } + out = self._call( + pricing, + "get_products", + ServiceCode="AmazonEC2", + FormatVersion="aws_v1", + Filters=[{"Type": "TERM_MATCH", "Field": k, "Value": v} for k, v in match.items()], + MaxResults=10, + ) + for item in out.get("PriceList") or []: + try: + doc = json.loads(item) if isinstance(item, str) else item + except ValueError: + continue + for term in ((doc.get("terms") or {}).get("OnDemand") or {}).values(): + for dim in (term.get("priceDimensions") or {}).values(): + try: + usd = float((dim.get("pricePerUnit") or {}).get("USD")) + except (TypeError, ValueError): + continue + if usd > 0: + return usd + return None + + try: + return self._cached(("demand", self._region(p), itype), look) + except DriverError: + return None # no pricing:GetProducts permission, say: the admin's price, or none + + def _spot_estimate(self, p, template, zone=None): + try: + prices = self.spot_prices(p, template["instance_type"], zone) + except DriverError: + return None + if not prices: + return None + price = max(prices.values()) # before launch the zone isn't known: the dearest, so budgets err high + cap = template.get("max_price") + return min(price, float(cap)) if cap else price + + def price(self, p, template): + """USD per hour of one unit, an estimate: the spot price (the dearest zone's, under the max price) or the + on-demand list price.""" + if template.get("market", "spot") == "spot": + spot = self._spot_estimate(p, template) + if spot is not None: + return spot + return self.on_demand_price(p, template["instance_type"]) + + # ---------- the calls the scaler makes ---------- + def check(self, p): + """Prove the keys work in the region with a read-only call.""" + out = self._call(self.client(p), "describe_availability_zones") + zones = sorted(z["ZoneName"] for z in out.get("AvailabilityZones") or [] if z.get("State") == "available") + return {"region": self._region(p), "zones": zones} + + def launch(self, p, template, name, boot): + if not boot.get("line"): + raise DriverError("an EC2 unit joins Lens over the mesh: turn the mesh on in Settings › Network first") + st = p.get("settings") or {} + itype = template["instance_type"] + spot = template.get("market", "spot") == "spot" + own_price = p.get("price") + estimate = None if own_price is not None else self.price(p, template) + if own_price is None and estimate is None: + raise DriverError( + f"AWS didn't say what {itype} costs in {self._region(p)}: allow pricing:GetProducts for these keys, " + "or give the provider a price per hour" + ) + arch = self.arch(p, itype) + if template.get("image"): + image_id, root = template["image"], self.root_device(p, template["image"]) + else: + image_id, root = self.image(p, arch) + tags = [{"Key": "Name", "Value": name}, {"Key": TAG, "Value": name}] + if boot.get("tag"): + tags.append({"Key": "lens", "Value": boot["tag"]}) # Lens's tag, as on every provider's machines + params = { + "ImageId": image_id, + "InstanceType": itype, + "MinCount": 1, + "MaxCount": 1, + "UserData": drivers.user_data(boot, template), + "InstanceInitiatedShutdownBehavior": "terminate", + "MetadataOptions": {"HttpTokens": "required", "HttpEndpoint": "enabled"}, + "BlockDeviceMappings": [ + { + "DeviceName": root, + "Ebs": { + "VolumeSize": int(template.get("disk_gb") or 30), + "VolumeType": "gp3", + "Encrypted": True, + "DeleteOnTermination": True, + }, + } + ], + "TagSpecifications": [{"ResourceType": "instance", "Tags": tags}, {"ResourceType": "volume", "Tags": tags}], + } + if st.get("subnet_id"): + params["SubnetId"] = st["subnet_id"] + if st.get("security_group_ids"): + params["SecurityGroupIds"] = list(st["security_group_ids"]) + if st.get("key_name"): + params["KeyName"] = st["key_name"] + market = "on-demand" + if spot: + options: dict = {"SpotInstanceType": "one-time", "InstanceInterruptionBehavior": "terminate"} + if template.get("max_price"): + options["MaxPrice"] = f"{float(template['max_price']):.4f}" + try: + out = self._call( + self.client(p), "run_instances", **params, InstanceMarketOptions={"MarketType": "spot", "SpotOptions": options} + ) + market = "spot" + except AwsError as e: + if e.code not in NO_SPOT or template.get("fallback") is False: + raise + out = None + fell_back = e.message + if out is None: + if own_price is None: + estimate = self.on_demand_price(p, itype) + if estimate is None: + raise DriverError(f"there was no spot {itype} ({fell_back}), and AWS didn't say what on demand costs") + out = self._call(self.client(p), "run_instances", **params) + else: + out = self._call(self.client(p), "run_instances", **params) + inst = (out.get("Instances") or [{}])[0] + if not inst.get("InstanceId"): + raise DriverError("AWS launched nothing") + price = own_price + if price is None: + price = estimate + zone = (inst.get("Placement") or {}).get("AvailabilityZone") + if market == "spot" and zone: + exact = self._spot_estimate(p, template, zone) + price = exact if exact is not None else price + got = {"id": inst["InstanceId"], "address": inst.get("PrivateIpAddress"), "market": market} + if price is not None: + got["price"] = round(float(price), 6) + return got + + def describe(self, p, node): + iid = node.get("external_id") + if not iid: + return {"state": "gone"} + try: + out = self._call(self.client(p), "describe_instances", InstanceIds=[iid]) + except AwsError as e: + if e.code == "InvalidInstanceID.NotFound": + return {"state": "gone", "why": "AWS has no such instance any more"} + raise + insts = [i for r in out.get("Reservations") or [] for i in r.get("Instances") or []] + if not insts: + return {"state": "gone", "why": "AWS has no such instance any more"} + inst = insts[0] + state = (inst.get("State") or {}).get("Name") + if state in GONE_STATES: + code = (inst.get("StateReason") or {}).get("Code") + if state in ("stopping", "stopped"): + self.terminate(p, node) # it takes no jobs, and its disk would keep costing; the scaler won't call again + return {"state": "gone", "why": WHY.get(code) or f"it's {state} at AWS", "reason": code} + out_ = {"state": "running" if state == "running" else "pending"} + if inst.get("PrivateIpAddress"): + out_["address"] = inst["PrivateIpAddress"] + if inst.get("InstanceLifecycle") == "spot": + out_["market"] = "spot" + return out_ + + def terminate(self, p, node): + iid = node.get("external_id") + if not iid: + return + try: + self._call(self.client(p), "terminate_instances", InstanceIds=[iid]) + except AwsError as e: + if e.code != "InvalidInstanceID.NotFound": + raise + + def listing(self, p, tag): + client = self.client(p) + found, token = [], None + while True: + params = { + "Filters": [ + {"Name": f"tag:{TAG}", "Values": [f"{tag}-*"]}, + {"Name": "instance-state-name", "Values": ["pending", "running", "stopping", "stopped"]}, + ], + "MaxResults": 100, + } + if token: + params["NextToken"] = token + out = self._call(client, "describe_instances", **params) + for r in out.get("Reservations") or []: + for i in r.get("Instances") or []: + name = next((t.get("Value") for t in i.get("Tags") or [] if t.get("Key") == TAG), None) + found.append({"id": i["InstanceId"], "name": name}) + token = out.get("NextToken") + if not token: + return found + + +class AwsError(DriverError): + """What AWS said when a call failed: its error code and message.""" + + def __init__(self, code, message, op): + self.code, self.message, self.op = code, message, op + super().__init__(f"AWS {op}: {message} ({code})") + + +register(Ec2()) diff --git a/fastapi_backend/app/domain/scaling_drivers.py b/fastapi_backend/app/domain/scaling_drivers.py index e9e1835e..8b1ef26a 100644 --- a/fastapi_backend/app/domain/scaling_drivers.py +++ b/fastapi_backend/app/domain/scaling_drivers.py @@ -281,3 +281,5 @@ def get(driver_id): register(Local()) + +from . import scaling_aws # noqa: E402,F401 - AWS EC2 registers itself diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml index 35255e11..c3840eef 100644 --- a/fastapi_backend/pyproject.toml +++ b/fastapi_backend/pyproject.toml @@ -69,6 +69,9 @@ sandbox = ["openshell>=0.1.4,<0.2"] # the assistant's web search through DuckDuckGo (web.py): ddgs (MIT), with its Rust HTTP client (primp) and lxml; # fetched when DuckDuckGo is picked in Settings › AI assistant › Web search web = ["ddgs>=9.16,<10"] +# auto scaling onto AWS EC2 (scaling_aws.py): botocore (Apache-2.0); fetched when the first AWS provider is saved in +# Settings › Auto scaling +aws = ["botocore>=1.43.113"] # Outlook .msg emails (extract-msg is GPL-3.0; .eml needs nothing) msg = ["extract-msg"] # secrets read from a Vaultwarden or Bitwarden vault (vaultwarden_secrets.py): python-vaultwarden (Apache-2.0); fetched diff --git a/fastapi_backend/tests/domain/test_scaling_aws.py b/fastapi_backend/tests/domain/test_scaling_aws.py new file mode 100644 index 00000000..3ba8041d --- /dev/null +++ b/fastapi_backend/tests/domain/test_scaling_aws.py @@ -0,0 +1,320 @@ +"""AWS EC2 for auto scaling (scaling_aws.py, docs/scaling.md): spot or on-demand instances that join over the mesh, +with prices as estimates, a fall back to on demand when there's no spot, and AWS taking a spot instance back. AWS is +botocore's Stubber: every call is checked against EC2's own API model, and answered from here.""" + +from __future__ import annotations + +import base64 +import datetime as dt +import json + +import pytest +from botocore.session import Session +from botocore.stub import Stubber + +from app.domain import scaling, settings, store +from app.domain import scaling_drivers as drivers +from app.domain.scaling_aws import TAG, Ec2 +from tests.helpers import login, make_user + +R = store.R +NOW = dt.datetime(2026, 10, 18, 12, 0, tzinfo=dt.UTC) +LINE = "curl -fsSL https://lens.example/install.sh | LENS_JOIN=https://lens.example LENS_JOIN_TOKEN=lensjoin1.7.x sh" +P = { + "id": 1, + "name": "AWS", + "settings": {"region": "eu-west-1"}, + "secrets": {"access_key_id": "AKIA", "secret_access_key": "s"}, + "price": None, +} +SPOT = {"instance_type": "t4g.medium", "market": "spot", "fallback": True, "disk_gb": 30} + + +def _client(service, region): + return Session().create_client(service, region_name=region, aws_access_key_id="AKIA", aws_secret_access_key="s") + + +class Aws: + """An Ec2 driver whose clients are stubbed; `calls` is every call it made, with its parameters.""" + + def __init__(self, monkeypatch): + self.drv = Ec2() + self.ec2, self.pricing = _client("ec2", "eu-west-1"), _client("pricing", "us-east-1") + self.stubs = {"ec2": Stubber(self.ec2), "pricing": Stubber(self.pricing)} + self.calls = [] + clients = {"ec2": self.ec2, "pricing": self.pricing} + monkeypatch.setattr(self.drv, "client", lambda p, service="ec2", region=None: clients[service]) + real = self.drv._call + + def call(client, op, **params): + self.calls.append((op, params)) + return real(client, op, **params) + + monkeypatch.setattr(self.drv, "_call", call) + for s in self.stubs.values(): + s.activate() + + def add(self, op, response=None, service="ec2"): + self.stubs[service].add_response(op, response or {}) + + def error(self, op, code, message="no", service="ec2"): + self.stubs[service].add_client_error(op, service_error_code=code, service_message=message, http_status_code=400) + + def ops(self): + return [op for op, _ in self.calls] + + def params(self, op): + return [p for o, p in self.calls if o == op] + + def done(self): + for s in self.stubs.values(): + s.assert_no_pending_responses() + + # the answers a launch reads + def spot_prices(self, **zones): + self.add( + "describe_spot_price_history", + {"SpotPriceHistory": [{"AvailabilityZone": z.replace("_", "-"), "SpotPrice": str(v)} for z, v in zones.items()]}, + ) + + def arch(self, arch="arm64"): + self.add( + "describe_instance_types", + {"InstanceTypes": [{"InstanceType": "t4g.medium", "ProcessorInfo": {"SupportedArchitectures": [arch]}}]}, + ) + + def ubuntu(self): + self.add( + "describe_images", + { + "Images": [ + {"ImageId": "ami-old", "CreationDate": "2026-01-01T00:00:00.000Z", "RootDeviceName": "/dev/sda1"}, + {"ImageId": "ami-new", "CreationDate": "2026-09-01T00:00:00.000Z", "RootDeviceName": "/dev/sda1"}, + ] + }, + ) + + def launched(self, iid="i-1", zone="eu-west-1b", spot=True): + inst = {"InstanceId": iid, "PrivateIpAddress": "172.31.0.5", "Placement": {"AvailabilityZone": zone}} + if spot: + inst["InstanceLifecycle"] = "spot" + self.add("run_instances", {"Instances": [inst]}) + + def on_demand(self, usd="0.0368"): + doc = {"terms": {"OnDemand": {"x": {"priceDimensions": {"y": {"pricePerUnit": {"USD": usd}}}}}}} + self.add("get_products", {"PriceList": [json.dumps(doc)]}, service="pricing") + + +@pytest.fixture +def aws(monkeypatch): + return Aws(monkeypatch) + + +BOOT = {"env": {"LENS_SCALING_UNIT": "lens-workers-1"}, "line": LINE, "token_id": 7, "tag": "lens"} + + +def test_a_spot_unit_launches_hardened_and_priced_in_its_zone(aws): + aws.spot_prices(eu_west_1a=0.012, eu_west_1b=0.014) # before launch: the dearest zone + aws.arch("arm64") + aws.ubuntu() + aws.launched(zone="eu-west-1a") + aws.spot_prices(eu_west_1a=0.012) # after: the zone it landed in + got = aws.drv.launch(P, {**SPOT, "steps": ["transcribe"]}, "lens-workers-1", BOOT) + aws.done() + assert got == {"id": "i-1", "address": "172.31.0.5", "market": "spot", "price": 0.012} + + (images,) = aws.params("describe_images") + assert images["Owners"] == ["099720109477"] and "ubuntu-noble-24.04-arm64-server-*" in images["Filters"][0]["Values"][0] + (run,) = aws.params("run_instances") + assert run["ImageId"] == "ami-new" and run["InstanceType"] == "t4g.medium" + assert run["InstanceMarketOptions"] == { + "MarketType": "spot", + "SpotOptions": {"SpotInstanceType": "one-time", "InstanceInterruptionBehavior": "terminate"}, + } + assert run["MetadataOptions"]["HttpTokens"] == "required" # IMDSv2 only + assert run["InstanceInitiatedShutdownBehavior"] == "terminate" # one that stops never keeps costing + assert run["BlockDeviceMappings"][0]["Ebs"] == {"VolumeSize": 30, "VolumeType": "gp3", "Encrypted": True, "DeleteOnTermination": True} + tags = {t["Key"]: t["Value"] for t in run["TagSpecifications"][0]["Tags"]} + assert tags == {"Name": "lens-workers-1", TAG: "lens-workers-1", "lens": "lens"} + # the user-data is the mesh's install line, named for the unit: its one-time token is the only credential in it + data = run["UserData"] + data = data if data.startswith("#!") else base64.b64decode(data).decode() + assert data.startswith("#!/bin/sh\n") and data.rstrip().endswith(LINE) + assert "export LENS_SCALING_UNIT=lens-workers-1" in data and "export LENS_WORKER_STEPS=transcribe" in data + + +def test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to(aws): + aws.spot_prices(eu_west_1a=0.012) + aws.arch() + aws.ubuntu() + aws.error("run_instances", "InsufficientInstanceCapacity", "no spot t4g.medium in eu-west-1") + aws.on_demand("0.0368") + aws.launched(spot=False) + got = aws.drv.launch(P, SPOT, "lens-workers-1", BOOT) + aws.done() + assert got["market"] == "on-demand" and got["price"] == 0.0368 + first, second = aws.params("run_instances") + assert "InstanceMarketOptions" in first and "InstanceMarketOptions" not in second + + aws.error("run_instances", "SpotMaxPriceTooLow", "your max price is under the spot price") + with pytest.raises(drivers.DriverError, match="SpotMaxPriceTooLow"): + aws.drv.launch(P, {**SPOT, "fallback": False, "max_price": 0.005}, "lens-workers-2", BOOT) # arch and image are cached + aws.done() + assert aws.params("run_instances")[-1]["InstanceMarketOptions"]["SpotOptions"]["MaxPrice"] == "0.0050" + + +def test_a_unit_without_a_price_or_the_mesh_launches_nothing(aws): + with pytest.raises(drivers.DriverError, match="mesh"): + aws.drv.launch(P, SPOT, "lens-workers-1", {**BOOT, "line": None}) + assert aws.calls == [] # no mesh: not even a call to AWS + + aws.error("get_products", "AccessDeniedException", "not allowed", service="pricing") + with pytest.raises(drivers.DriverError, match="pricing:GetProducts"): + aws.drv.launch(P, {**SPOT, "market": "on-demand"}, "lens-workers-1", BOOT) + aws.done() + assert "run_instances" not in aws.ops() # a unit with no price would never count against the budget + + # the admin's own price on the provider wins, and no price is looked up + aws.arch("x86_64") + aws.ubuntu() + aws.launched(spot=False) + got = aws.drv.launch({**P, "price": 0.1}, {**SPOT, "market": "on-demand", "instance_type": "m7i.large"}, "lens-workers-1", BOOT) + aws.done() + assert got["price"] == 0.1 and "get_products" not in aws.ops()[-3:] + assert "ubuntu-noble-24.04-amd64-server-*" in aws.params("describe_images")[-1]["Filters"][0]["Values"][0] + + +def test_describe_says_when_aws_took_a_spot_instance_back(aws): + node = {"external_id": "i-1", "name": "lens-workers-1"} + reservation = lambda state, code=None: { # noqa: E731 + "Reservations": [ + { + "Instances": [ + { + "InstanceId": "i-1", + "State": {"Name": state}, + "PrivateIpAddress": "172.31.0.5", + "InstanceLifecycle": "spot", + **({"StateReason": {"Code": code, "Message": code}} if code else {}), + } + ] + } + ] + } + aws.add("describe_instances", reservation("pending")) + aws.add("describe_instances", reservation("running")) + aws.add("describe_instances", reservation("terminated", "Server.SpotInstanceTermination")) + aws.add("describe_instances", reservation("stopped", "Client.UserInitiatedShutdown")) + aws.add("terminate_instances", {}) # a stopped one takes no jobs and its disk keeps costing: terminated + aws.error("describe_instances", "InvalidInstanceID.NotFound") + assert aws.drv.describe(P, node)["state"] == "pending" + assert aws.drv.describe(P, node) == {"state": "running", "address": "172.31.0.5", "market": "spot"} + assert aws.drv.describe(P, node) == { + "state": "gone", + "why": "AWS took the spot instance back", + "reason": "Server.SpotInstanceTermination", + } + assert aws.drv.describe(P, node)["why"] == "someone stopped it at AWS" + assert aws.drv.describe(P, node)["state"] == "gone" + aws.done() + assert aws.params("terminate_instances") == [{"InstanceIds": ["i-1"]}] + + +def test_terminate_check_and_listing(aws): + aws.add("terminate_instances", {"TerminatingInstances": [{"InstanceId": "i-1"}]}) + aws.error("terminate_instances", "InvalidInstanceID.NotFound") # already gone: no error + aws.drv.terminate(P, {"external_id": "i-1"}) + aws.drv.terminate(P, {"external_id": "i-1"}) + aws.drv.terminate(P, {"external_id": None}) # never launched: nothing to call + + aws.add( + "describe_availability_zones", + {"AvailabilityZones": [{"ZoneName": "eu-west-1b", "State": "available"}, {"ZoneName": "eu-west-1a", "State": "available"}]}, + ) + assert aws.drv.check(P) == {"region": "eu-west-1", "zones": ["eu-west-1a", "eu-west-1b"]} + + tagged = lambda iid, name: {"InstanceId": iid, "Tags": [{"Key": TAG, "Value": name}]} # noqa: E731 + aws.add("describe_instances", {"Reservations": [{"Instances": [tagged("i-1", "lens-a-1")]}], "NextToken": "n"}) + aws.add("describe_instances", {"Reservations": [{"Instances": [tagged("i-2", "lens-a-2")]}]}) + assert aws.drv.listing(P, "lens") == [{"id": "i-1", "name": "lens-a-1"}, {"id": "i-2", "name": "lens-a-2"}] + assert aws.params("describe_instances")[0]["Filters"][0] == {"Name": f"tag:{TAG}", "Values": ["lens-*"]} + assert aws.params("describe_instances")[1]["NextToken"] == "n" + aws.done() + + +def test_the_driver_is_offered_and_its_settings_are_checked(): + drv = drivers.get("aws") + spec = drv.spec() + assert (spec["kind"], spec["priced"], spec["needs_join"], spec["extra"], spec["license"]) == ("node", True, True, "aws", "Apache-2.0") + assert set(drv.secrets) == {"access_key_id", "secret_access_key"} + assert drivers.check_template(drv, {}) == {"instance_type": "t4g.medium", "market": "spot", "fallback": True, "disk_gb": 30} + with pytest.raises(ValueError, match="market"): + drivers.check_template(drv, {"market": "reserved"}) + + +# ---------- through the scaler ---------- +@pytest.fixture +def h(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +def test_the_scaler_replaces_a_spot_unit_aws_took_back(client, db, cfg, h, aws, monkeypatch): + monkeypatch.setitem(drivers.DRIVERS, "aws", aws.drv) + monkeypatch.setattr(scaling, "can_join", lambda cfg: True) # the mesh is on and has its install line + monkeypatch.setattr( + scaling, "bootstrap", lambda db, cfg, g, name: {"env": {"LENS_SCALING_UNIT": name}, "line": LINE, "token_id": None, "tag": "lens"} + ) + settings.save(db, cfg, "scaling", {"enabled": True}, "root@x.io") + cfg = settings.effective(db, cfg) + aws.add("describe_availability_zones", {"AvailabilityZones": []}) # the check when it's saved + body = { + "name": "AWS", + "driver": "aws", + "settings": {"region": "eu-west-1"}, + "secrets": {"access_key_id": "AKIA", "secret_access_key": "s"}, + } + r = client.post("/api/v1/scaling/providers", headers=h, json=body) + assert r.status_code == 201, r.text + assert r.json()["secrets"] == {"access_key_id": True, "secret_access_key": True} + g = client.post( + "/api/v1/scaling/groups", + headers=h, + json={ + "name": "Workers", + "provider": r.json()["id"], + "min": 1, + "max": 2, + "budget": 20, + "template": SPOT, + "cooldown_seconds": 0, + "warmup_seconds": 0, + }, + ) + assert g.status_code == 201, g.text + gid = g.json()["id"] + + aws.spot_prices(eu_west_1a=0.012) # the scaler's estimate before launching (then cached) + aws.arch() + aws.ubuntu() + aws.launched("i-1", "eu-west-1a") + aws.spot_prices(eu_west_1a=0.012) # the zone it landed in + tick = lambda now: scaling.Scaler(db, lambda: cfg, "test").tick(now=now, lease=False) # noqa: E731 + assert tick(NOW)["launched"] == 1 + (unit,) = scaling.nodes(db, gid) + assert (unit["external_id"], unit["price"], unit["state"]) == ("i-1", 0.012, "launching") + + # its worker joins, then AWS takes the spot instance back: the scaler says why and launches another + db.q("UPSERT $w MERGE {name: $n, heartbeat_at: $t}", w=R("worker", unit["name"]), n=unit["name"], t=NOW.isoformat()) + aws.add("describe_instances", {"Reservations": [{"Instances": [{"InstanceId": "i-1", "State": {"Name": "running"}}]}]}) + tick(NOW + dt.timedelta(seconds=30)) + assert scaling.nodes(db, gid)[0]["state"] == "running" + gone = {"InstanceId": "i-1", "State": {"Name": "terminated"}, "StateReason": {"Code": "Server.SpotInstanceTermination", "Message": "x"}} + aws.add("describe_instances", {"Reservations": [{"Instances": [gone]}]}) + aws.launched("i-2", "eu-west-1a") # prices, the image and the architecture are cached + did = tick(NOW + dt.timedelta(minutes=5)) + aws.done() + assert (did["terminated"], did["launched"]) == (1, 1) + old, new = sorted(scaling.nodes(db, gid), key=lambda n: n["id"]) + assert old["state"] == "terminated" and old["why"] == "AWS took the spot instance back" + assert new["external_id"] == "i-2" and new["state"] == "launching" diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock index bb992139..3933de36 100644 --- a/fastapi_backend/uv.lock +++ b/fastapi_backend/uv.lock @@ -1989,6 +1989,9 @@ dependencies = [ ] [package.optional-dependencies] +aws = [ + { name = "botocore" }, +] doctr = [ { name = "python-doctr" }, ] @@ -2098,6 +2101,7 @@ dev = [ [package.metadata] requires-dist = [ + { name = "botocore", marker = "extra == 'aws'", specifier = ">=1.43.113" }, { name = "cryptography", specifier = ">=50" }, { name = "ddgs", marker = "extra == 'web'", specifier = ">=9.16,<10" }, { name = "extract-msg", marker = "extra == 'msg'" }, @@ -2153,7 +2157,7 @@ requires-dist = [ { name = "transformers", marker = "extra == 'punctuation'" }, { name = "webauthn", specifier = ">=3.0.1,<4" }, ] -provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "spacy", "injection", "florence", "moondream", "validate", "smb", "sandbox", "web", "msg", "vaultwarden"] +provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "spacy", "injection", "florence", "moondream", "validate", "smb", "sandbox", "web", "aws", "msg", "vaultwarden"] [package.metadata.requires-dev] dev = [