Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,9 +210,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud and Proxmox VE providers are in progress (built, with
machines that join over the mesh to transcribe and tell voices apart, not yet tried on a real cloud), and Linux and
model hosts are planned. ([Auto scaling](docs/scaling.md);
AWS EC2 (spot or on demand), DigitalOcean, Hetzner Cloud, Proxmox VE and Linux hosts over SSH providers are in
progress (built, with machines that join over the mesh to transcribe and tell voices apart, not yet tried on a real
cloud), and model hosts are planned. ([Auto scaling](docs/scaling.md);
[checks](docs/requirements.md#scale-out))
- **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a
notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine,
Expand Down Expand Up @@ -288,9 +288,8 @@ Next, and not in a pull request yet; each links to where it is tracked:
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
- **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
([local models](docs/local-models.md)).
- **Auto scaling:** providers for Linux hosts over SSH, joining over the Nebula mesh, model hosts
(RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings
([auto scaling](docs/scaling.md)).
- **Auto scaling:** model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups
in Settings ([auto scaling](docs/scaling.md)).
- **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the
activity ledger ([speech providers](docs/speech-providers.md)).
- **Storage:** uploads and assets' media in the file store, and a storage key to download
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,7 @@ request, to another check that proves the same thing; loosening a requirement or
| DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
| A group of machines elsewhere counts only the jobs a worker node takes, and never launches one for others | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_group_elsewhere_counts_only_the_jobs_a_node_takes` |
| Proxmox VE: clones of a VM template, started and booted through the guest agent, found by tag, stopped and deleted; a clone whose agent never answers says so | [test_scaling_proxmox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_proxmox.py) `test_proxmox_clones_boots_through_the_guest_agent_and_deletes`, `test_a_clone_whose_agent_never_answers_says_so`, `test_a_clone_still_booting_is_pending` |
| Linux hosts over SSH: host keys trusted on first use and held to, workers started per host with a script that removes itself, found, counted and stopped | [test_scaling_ssh.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_ssh.py) `test_host_specs`, `test_ssh_trusts_host_keys_on_first_use_and_holds_to_them`, `test_ssh_starts_counts_and_stops_workers` |
| AWS EC2 units launch as spot or on-demand instances, hardened (IMDSv2, encrypted disk, terminate on shutdown), with only the mesh's one-time join line in their user-data | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_a_spot_unit_launches_hardened_and_priced_in_its_zone`, `test_the_driver_is_offered_and_its_settings_are_checked` |
| A spot unit falls back to on demand when AWS has none; without a price or the mesh, nothing launches | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_without_spot_capacity_it_falls_back_to_on_demand_unless_told_not_to`, `test_a_unit_without_a_price_or_the_mesh_launches_nothing` |
| When AWS takes a spot instance back, the scaler says why and launches another | [test_scaling_aws.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_aws.py) `test_describe_says_when_aws_took_a_spot_instance_back`, `test_the_scaler_replaces_a_spot_unit_aws_took_back`, `test_terminate_check_and_listing` |
Expand Down
19 changes: 15 additions & 4 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the A
Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
runs more `lens worker` processes where Lens runs. In progress: **AWS EC2** (spot or on demand), **DigitalOcean**
and **Hetzner Cloud** providers are built (credentials checked, prices, launching, watching and deleting machines),
and their machines join over the [mesh](mesh.md) as worker nodes that transcribe and tell voices apart; so is
**Proxmox VE**, clones of a VM template on your own server. They haven't been tried on a real cloud machine yet. Planned: providers for any Linux host over SSH; model hosts
(RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making providers and groups in Settings.
and their machines join over the [mesh](mesh.md) as worker nodes that transcribe and tell voices apart; so are
**Proxmox VE**, clones of a VM template on your own server, and **Linux hosts over SSH**, workers started on machines
you have. They haven't been tried on a real cloud machine yet. Planned: model hosts (RunPod, Lambda, Hugging Face,
SageMaker, Bedrock) for models; and making providers and groups in Settings.

Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
signed in to the web app. Agents can propose and manage groups, but never approve one.
Expand Down Expand Up @@ -58,6 +59,7 @@ carries none of them.
| `aws` (AWS EC2) | EC2 instances that join over the mesh and run `lens worker` | the spot or on-demand price, an estimate | `instance_type`, `market` (`spot` or `on-demand`), `max_price`, `fallback`, `image`, `disk_gb`, `steps` |
| `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` |
| `proxmox` (Proxmox VE) | clones of a VM template on `node`, at `url` with an API token (`token_id`, `token_secret` sealed); `verify_tls` (on) and `ca_cert` for a private CA | nothing, unless you give it a price | `vm_template` (its id), `full_clone` (off: linked), `storage`, `pool`, `steps` |
| `ssh` (Linux hosts over SSH) | `lens worker` started on `hosts` (`user@host` or `user@host:port`), signed in with `private_key` (sealed; `passphrase` if it has one) | nothing, unless you give it a price | `per_host` (1): workers on each host; `steps` |
| `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` |

A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where
Expand All @@ -83,6 +85,15 @@ deleted. Letting a unit go stops the clone and deletes it with its disks. The AP
start, stop and delete VMs and use their guest agent (the `PVEVMAdmin` role on the template and the clones' pool is
enough).

An SSH unit is the first-boot script started on one of the provider's hosts, the first with fewer than `per_host` of
Lens's workers running, in its own process group (`setsid`) under `~/.lens-scaling/`: the script is written there
readable only by that user and deletes itself as it starts, since it holds the unit's join token; the unit's process
id and log stay beside it. Letting a unit go stops its process group and removes its files. Host keys are trusted on
first use: saving or checking the provider records each host's key on it (`check.host_keys`, kept when a later check
fails), and every other call refuses a host whose key differs. A host whose key changed for a good reason is taken off
the provider and added back. The SSH client is Paramiko (LGPL-2.1, used unmodified), the `ssh` extra, fetched the
first time an SSH provider is saved.

Until the mesh is on (Settings › Network), a group on these providers waits (`waiting_for` says so) and launches
nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing.

Expand Down Expand Up @@ -234,7 +245,7 @@ Turning scaling on (`change_settings scaling`) always asks.

## Refine later

Providers for SSH hosts and model hosts; Proxmox LXC containers; revoking a unit's mesh certificate when it goes;
Providers for model hosts; Proxmox LXC containers; revoking a unit's mesh certificate when it goes;
reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro rate; preemptible and spot
instances at clouds other than AWS; draining an AWS spot unit on its two-minute notice (the node agent reading the
instance's metadata) and handing its job back at once; following the spot price as it changes; AWS sign-in with an
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/domain/scaling.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
import time

from . import activity, budgets, jobs, schedule, settings, store
from . import scaling_clouds, scaling_proxmox # noqa: F401 - registers their drivers
from . import scaling_clouds, scaling_proxmox, scaling_ssh # noqa: F401 - registers their drivers
from . import scaling_drivers as drivers
from .store import R

Expand Down Expand Up @@ -224,6 +224,8 @@ def _check_call(db, cfg, p, drv):
result = {"ok": True, **(out if isinstance(out, dict) else {})}
except Exception as e: # noqa: BLE001 - what the provider said, for the person
result = {"ok": False, "error": f"{type(e).__name__}: {e}"[:500] if not isinstance(e, drivers.DriverError) else str(e)[:500]}
before = p.get("check") or {}
result |= {k: before[k] for k in drv.kept if k in before} # what a failed check mustn't forget (SSH host keys)
db.q("UPDATE $r SET checked_at = $t, check = $c", r=R("scale_provider", p["id"]), t=store.now(), c=result)
return result

Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/scaling_drivers.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ class Driver:
label = ""
about = ""
priced = True # it charges money: its groups need a budget
kept: tuple[str, ...] = () # what a failed check keeps of the last one's answer
needs_join = False # its units are other machines, which join over the mesh's install line (scaling.bootstrap)
extra = None # the optional dependency (pyproject.toml) it needs, fetched when a provider of it is saved
modules = () # what that extra installs, to see whether it's here
Expand Down
Loading
Loading