Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
cloud, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
DigitalOcean and Hetzner Cloud providers are in progress (built, launching once the mesh's install line lands), and
AWS, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
[checks](docs/requirements.md#scale-out))
- **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a
notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine,
Expand Down Expand Up @@ -253,9 +254,9 @@ Next, and not in a pull request yet; each links to where it is tracked:
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
- **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
([local models](docs/local-models.md)).
- **Auto scaling:** providers for AWS, DigitalOcean, Hetzner, Proxmox and Linux hosts over SSH, joining over the
Nebula mesh, model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in
Settings ([auto scaling](docs/scaling.md)).
- **Auto scaling:** providers for AWS, Proxmox and Linux hosts over SSH, joining over the Nebula mesh, model hosts
(RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in Settings
([auto scaling](docs/scaling.md)).
- **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the
activity ledger ([speech providers](docs/speech-providers.md)).
- **Storage:** uploads and assets' media in the file store, and a storage key to download
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Each unit's cost goes into the activity ledger as an estimate, and a group stops at its budget's cap | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_at_its_budgets_cap_a_group_stops` |
| Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` |
| This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
| DigitalOcean and Hetzner Cloud: credentials checked, list prices, machines launched with a one-use join token, watched, found by tag and deleted; a group waits for the mesh's install line before launching | [test_scaling_clouds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_clouds.py) `test_a_digitalocean_group_waits_for_the_install_line_then_launches_droplets`, `test_hetzner_prices_in_euros_and_its_errors_read_as_hetzners`, `test_the_cloud_drivers_are_listed_and_need_the_mesh` |
| Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` |

## Keep it up to date and backed up
Expand Down
26 changes: 22 additions & 4 deletions docs/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ launched. Everything it does is in the [activity ledger](activity.md), and its c

Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools,
Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
runs more `lens worker` processes where Lens runs. Planned: providers for AWS, DigitalOcean, Hetzner, Proxmox and any
Linux host over SSH, joining Lens over its Nebula mesh network; model hosts (RunPod, Lambda, Hugging Face, SageMaker,
Bedrock) for models; and making providers and groups in Settings.
runs more `lens worker` processes where Lens runs. In progress: **DigitalOcean** and **Hetzner Cloud** providers are
built (credentials checked, list prices, launching, watching and deleting machines), and launch once the mesh's
install line, which another machine joins Lens with, lands ([the mesh](mesh.md)). Planned: providers for AWS, Proxmox
and any Linux host over SSH; model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock) for models; and making
providers and groups in Settings.

Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
signed in to the web app. Agents can propose and manage groups, but never approve one.
Expand Down Expand Up @@ -53,10 +55,24 @@ carries none of them.
| Driver | Units | Costs | Template |
| --- | --- | --- | --- |
| `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) |
| `digitalocean` | droplets in `region`, with `token` (sealed), optional `ssh_keys` and `vpc_uuid` | DigitalOcean's hourly list price | `size` (`s-1vcpu-2gb`), `image` (`ubuntu-24-04-x64`), `steps` |
| `hetzner` (Hetzner Cloud) | servers in `location`, with `token` (sealed), optional `ssh_keys` | Hetzner's hourly gross price in euros, times `usd_per_eur` (1.1; set your rate) | `server_type` (`cx22`), `image` (`ubuntu-24.04`), `steps` |

A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where
it runs.

### Machines elsewhere

A cloud provider's unit is a new machine. Its first boot runs a script (user-data) that sets `LENS_SCALING_UNIT` (its
name) and `LENS_WORKER_STEPS` (the template's steps), then the mesh's install line with a join token made for that
unit, usable once within an hour. The machine joins Lens over the [mesh](mesh.md) and starts `lens worker`; the
database credential travels inside the sealed join answer, so the user-data holds no long-lived secret. Each machine
carries Lens's tag (`scaling.tag`: a DigitalOcean tag, a Hetzner label `lens=<tag>`), so Lens's machines can be told
apart. Letting a unit go deletes its machine; a machine switched off at the provider counts as gone and is deleted.

Until the mesh is on and has its install line, a group on these providers waits (`waiting_for` says so) and launches
nothing, since its machines couldn't join. They call the providers' REST APIs with httpx, so they install nothing.

## Groups

```http
Expand Down Expand Up @@ -145,5 +161,7 @@ Turning scaling on (`change_settings scaling`) always asks.

## Refine later

Providers beyond This machine; spot and preemptible instances; warm pools; predictive scaling; health checks beyond
Providers for Proxmox, SSH hosts and model hosts; the AWS provider (in its own pull request); revoking a unit's mesh
certificate when it goes; reporting machines with Lens's tag that Lens has no unit for (`listing`); a live euro
rate; spot and preemptible instances; warm pools; predictive scaling; health checks beyond
joining; prices fetched from providers' pricing APIs.
30 changes: 21 additions & 9 deletions fastapi_backend/app/domain/scaling.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
import time

from . import activity, budgets, jobs, schedule, settings, store
from . import scaling_clouds # noqa: F401 - registers the cloud drivers
from . import scaling_drivers as drivers
from .store import R

Expand Down Expand Up @@ -539,6 +540,7 @@ def group_view(db, g, now=None):
st = budgets.status(db, _budget_ref(g["id"]), b, now) if b else None
approval = g.get("approval")
t = terms({**g, "budget": b.get("usd") if b else None}) # the budget lives in the budget table, not on the row
drv = drivers.DRIVERS.get((db.one("SELECT driver FROM $r", r=R("scale_provider", int(g["provider"]))) or {}).get("driver"))
out = {
"id": g["id"],
"name": g.get("name"),
Expand All @@ -560,6 +562,7 @@ def group_view(db, g, now=None):
"approval": approval,
"approved": bool(approval) and not widens(approval.get("terms") or {}, t),
"terms": t,
"needs_join": bool(drv and drv.needs_join),
"budget": {"usd": b.get("usd"), "period": b.get("period"), "state": st["state"], "spent": st["spent"], "left": st["left"]}
if b
else None,
Expand Down Expand Up @@ -593,6 +596,8 @@ def waiting_for(view, cfg=None):
return "a person's approval of its terms, in Settings › Auto scaling"
if view.get("capped"):
return f"its budget: {view['capped']}"
if cfg is not None and view.get("needs_join") and not can_join(cfg):
return "the mesh's install line, which other machines join Lens with (in progress, docs/mesh.md)"
return None


Expand Down Expand Up @@ -951,26 +956,32 @@ def evaluate(g, m, units, now, state=None):

# ---------- the scaler ----------
def mesh_module():
"""The Nebula mesh (mesh.py, docs/network.md) when this Lens has it, else None."""
"""The Nebula mesh (mesh.py, docs/mesh.md) when this Lens has it, else None."""
try:
from . import mesh
except ImportError:
return None
return mesh


def can_join(cfg):
"""Whether another machine can join Lens now: the mesh is on and has the install line machines join with."""
mesh = mesh_module()
if mesh is None or not (getattr(mesh, "conf", lambda c: {})(cfg) or {}).get("enabled"):
return False
return bool(getattr(mesh, "make_token", None) and getattr(mesh, "join_line", None))


def bootstrap(db, cfg, g, name):
"""How a unit joins Lens: a worker started here gets its name and inherits the rest; a machine elsewhere gets the
mesh's install line (`LENS_JOIN=… LENS_JOIN_TOKEN=… sh install.sh`) with a one-time join token good for an hour,
named for the unit, so the provider's user-data carries no long-lived secret: the database credential travels
inside the sealed join answer. {env, line, token_id} (line and token_id None without the mesh)."""
boot = {"env": {"LENS_SCALING_UNIT": name}, "line": None, "token_id": None}
mesh = mesh_module()
if mesh is None or not (getattr(mesh, "conf", lambda c: {})(cfg) or {}).get("enabled"):
return boot
make, line = getattr(mesh, "make_token", None), getattr(mesh, "join_line", None)
if not (make and line):
inside the sealed join answer. {env, line, token_id, tag} (line and token_id None without the mesh)."""
boot = {"env": {"LENS_SCALING_UNIT": name}, "line": None, "token_id": None, "tag": conf(cfg)["tag"]}
if not can_join(cfg):
return boot
mesh = mesh_module()
make, line = mesh.make_token, mesh.join_line
row, token = make(db, cfg, by="scaling", role="worker", name=name, hours=1, uses=1)
boot["line"], boot["token_id"] = line(cfg, token), (row or {}).get("id")
return boot
Expand Down Expand Up @@ -1035,6 +1046,7 @@ def _group(self, cfg, g, now, did):
launching = [n for n in units if n.get("state") == "launching"]
view = group_view(db, g, now)
may_launch = enabled(cfg) and view["enabled"] and not view["paused"] and view["approved"] and not capped
joinable = not drv.needs_join or can_join(cfg)
lo, hi, desired = g.get("min") or 0, g.get("max") or 0, g.get("desired") or 0
if may_launch and g.get("rules"):
want, state, said, new_min, new_max = evaluate(
Expand All @@ -1057,7 +1069,7 @@ def _group(self, cfg, g, now, did):
target = 0
elif capped:
target = lo if g.get("at_cap") == "min" else 0
elif not view["approved"] or view["paused"]:
elif not view["approved"] or view["paused"] or not joinable:
target = min(target, len(running) + len(launching)) # nothing new; what runs stays
have = len(running) + len(launching)
if target > have:
Expand Down
Loading
Loading