Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,12 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
what cost most. Routine choices go to small decision models, and the LLM only when it adds value.
([Activity and costs](docs/activity.md), [budgets](docs/budgets.md#one-budget-for-everything),
[the meter](docs/budgets.md#the-meter-and-the-costs-page); [checks](docs/requirements.md#see-what-it-costs))
- **Scale out.** Scaling groups add workers when work waits and let them go when it doesn't, by target, step and
schedule rules, within min and max and a monthly budget, with every launch, join and hour of cost in the activity
ledger. A group launches nothing until you approve its terms in Settings › Auto scaling; agents propose and manage
groups over the API, MCP and the assistant, and never approve one. It runs more workers on this machine today;
cloud, Proxmox, Linux and model hosts are planned. ([Auto scaling](docs/scaling.md);
[checks](docs/requirements.md#scale-out))
- **Keep it up to date and backed up.** Lens checks for a new release once a day and tells admins by email or a
notification. One click in Settings › Updates (or one API call) upgrades it through a small agent on the machine,
showing each step, and rolls back when the new version doesn't start; nothing installs unless an admin says so. The
Expand Down Expand Up @@ -246,6 +252,9 @@ Next, and not in a pull request yet; each links to where it is tracked:
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
- **Local models:** GPU builds of llama.cpp fetched by Lens, and local embedding models
([local models](docs/local-models.md)).
- **Auto scaling:** providers for AWS, DigitalOcean, Hetzner, Proxmox and Linux hosts over SSH, joining over the
Nebula mesh, model hosts (RunPod, Lambda, Hugging Face, SageMaker, Bedrock), and making providers and groups in
Settings ([auto scaling](docs/scaling.md)).
- **Speech providers:** their summaries, chapters and entities as optional imports, and per-minute prices in the
activity ledger ([speech providers](docs/speech-providers.md)).
- **Storage:** uploads and assets' media in the file store, and a storage key to download
Expand Down Expand Up @@ -285,7 +294,7 @@ The guides are on the [docs site](https://adeelahmad.github.io/lense/) and in th
[the graph as files](docs/file-tree.md), [NFS share](docs/file-shares.md), [SMB share](docs/smb.md),
[linked data (RDF)](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md)
- **Models and cost:** [local models](docs/local-models.md), [speech providers](docs/speech-providers.md),
[activity and costs](docs/activity.md), [budgets](docs/budgets.md)
[activity and costs](docs/activity.md), [budgets](docs/budgets.md), [auto scaling](docs/scaling.md)
- **Operate:** [encryption](docs/encryption.md), [storage](docs/storage.md), [database](docs/database.md),
[logging](docs/logging.md), [telemetry](docs/telemetry.md), [Prometheus](docs/prometheus.md),
[usage analytics](docs/analytics.md)
Expand Down
2 changes: 1 addition & 1 deletion docs/budgets.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Budgets

A budget caps what a routine, automation, pipeline, workflow or namespace may cost. Budgets are off unless an admin sets one.
A budget caps what a routine, automation, pipeline, workflow, namespace or [scaling group](scaling.md) may cost. Budgets are off unless an admin sets one.
What has been spent comes from the [activity ledger](activity.md), so a budget counts everything done for that
resource: a namespace's budget counts every job in it, and a routine's counts its runs and the jobs it queued.

Expand Down
14 changes: 14 additions & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,20 @@ request, to another check that proves the same thing; loosening a requirement or
| The Costs page's charts | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| Routine choices go to small decision models, the LLM only when it adds value | [test_decisions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_decisions.py) `test_the_decision_model_answers_when_it_has_a_key`, `test_the_language_model_decides_without_one_and_when_it_fails` |

## Scale out

| Requirement | Checks |
|---|---|
| Off until an admin turns it on; off, every unit is let go and the scaler writes nothing | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_scaling_is_off_and_lists_its_drivers`, `test_the_settings_section`, `test_this_machine_runs_more_workers_at_no_cost` |
| Providers keep their credentials sealed, shown only as set, and prove them with a read-only call | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_a_providers_credentials_are_sealed_and_checked` |
| A group launches nothing until a person signed in to the web app approves its terms; agents never approve, and widening the terms takes the approval away | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_only_a_person_in_the_web_app_approves_a_groups_terms`; [test_scaling_rules.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_rules.py) `test_what_widens_an_approval` |
| Target, step and schedule rules pick the number of units, within min and max | [test_scaling_rules.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_scaling_rules.py) `test_a_target_rule_keeps_its_metric_at_the_target`, `test_a_step_rule_fires_once_its_condition_has_held`, `test_a_schedule_rule_sets_the_bounds_when_its_time_comes`, `test_the_largest_proposal_wins`; [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_rules_scale_on_waiting_jobs_and_on_a_schedule` |
| The scaler launches units, sees them join, drains busy workers before letting them go, and fails and pauses on units that don't come up | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_a_unit_that_never_joins_fails_and_failures_pause_the_group`, `test_protected_and_terminated_units` |
| Each unit's cost goes into the activity ledger as an estimate, and a group stops at its budget's cap | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_the_scaler_launches_joins_drains_and_terminates`, `test_at_its_budgets_cap_a_group_stops` |
| Settings › Auto scaling shows each group's terms in words and approves them in one click | [settings-scaling.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-scaling.test.tsx) `shows each group's terms and approves them in one click`, `puts the terms and the state in words` |
| This machine runs more workers at no cost | [test_scaling.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling.py) `test_this_machine_runs_more_workers_at_no_cost` |
| Agents manage scaling in the assistant and over MCP: proposals wait for the person, actions within the terms happen at once | [test_scaling_agents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_scaling_agents.py) `test_the_assistant_proposes_and_the_person_approves_in_the_web_app`, `test_a_setup_conversation_makes_a_group_that_still_waits_for_the_person`, `test_only_admins_get_the_scaling_tools`, `test_mcp_clients_manage_scaling_but_never_approve` |

## Keep it up to date and backed up

| Requirement | Checks |
Expand Down
149 changes: 149 additions & 0 deletions docs/scaling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# Auto scaling

Auto scaling adds workers when there's work waiting and lets them go when there isn't, the way a cloud's auto
scaling groups do, within a budget you set. A **provider** is where units launch, a **scaling group** says what each
unit is, how many there may be and the rules that pick the number, and a **unit** is one machine or process the group
launched. Everything it does is in the [activity ledger](activity.md), and its cost counts against the group's
[budget](budgets.md).

Status: in progress. Built: groups, rules, approvals, budgets, the scaler, the API, MCP and assistant tools,
Settings › Auto scaling (the switch, and each group's terms to approve or pause), and **This machine**, a provider that
runs more `lens worker` processes where Lens runs. Planned: providers for AWS, DigitalOcean, Hetzner, Proxmox and any
Linux host over SSH, joining Lens over its Nebula mesh network; model hosts (RunPod, Lambda, Hugging Face, SageMaker,
Bedrock) for models; and making providers and groups in Settings.

Auto scaling is **off** until an admin turns it on, and a group launches nothing until a person approves its terms
signed in to the web app. Agents can propose and manage groups, but never approve one.

## Turn it on

**Settings › Auto scaling** turns it on, or:

```http
PUT /api/v1/settings/scaling
{"enabled": true}
```

| Setting | Default | |
| --- | --- | --- |
| `enabled` | `false` | the master switch: off, every unit is let go and nothing launches |
| `tick_seconds` | `30` | how often the scaler looks (10 to 600) |
| `join_minutes` | `15` | how long a new unit may take to join before it counts as failed (1 to 120) |
| `accrue_minutes` | `5` | how often a running unit's cost goes into the ledger (1 to 60) |
| `tag` | `lens` | the prefix of every unit's name, so they can be told apart at the provider |

One process runs the scaler at a time: the API and every `lens worker` (except those started with `--steps` or
`--no-schedule`, as scaled workers are) take turns holding its lease. While scaling is off and nothing is left to let
go, the scaler doesn't even take the lease.

## Providers

```http
POST /api/v1/scaling/providers
{"name": "Here", "driver": "local"}
```

`GET /api/v1/scaling/drivers` lists the kinds of provider, with the settings each asks for (`fields`) and what a group
picks for each unit (`template`). A provider's credentials (`secrets`) are sealed when saved, bound to that provider
and field, and opened only for a call to it; the API shows only whether each one is set. Saving a provider proves its
credentials with a read-only call (`check`), kept on the provider; `POST /scaling/providers/{id}/check` runs it again.
A driver that needs a library fetches it the first time a provider of its kind is saved, so the default install
carries none of them.

| Driver | Units | Costs | Template |
| --- | --- | --- | --- |
| `local` (This machine) | more `lens worker` processes where the scaler runs | nothing | `steps`: the steps they run (none: every step) |

A worker on This machine stops with the process that started it, and the scaler that takes over starts them again where
it runs.

## Groups

```http
POST /api/v1/scaling/groups
{"name": "Transcribers", "provider": 1, "template": {"steps": ["transcribe"]}, "min": 0, "max": 4,
"rules": [{"type": "step", "metric": "jobs_waiting", "above": 10, "minutes": 2, "add": 2},
{"type": "step", "metric": "jobs_waiting", "below": 1, "minutes": 10, "remove": "50%"}]}
```

| Field | Default | |
| --- | --- | --- |
| `name`, `provider` | | |
| `template` | the driver's defaults | what each unit is |
| `min`, `max` | `0`, `1` | the bounds, up to 100 |
| `desired` | `min` | units wanted now; the rules change it |
| `cooldown_seconds` | `300` | after scaling out or in, how long before the next change the same way |
| `warmup_seconds` | `300` | how long a launching unit may take to join before more are judged |
| `terminate` | `oldest` | which goes first when scaling in: `oldest` or `newest`. Idle units go before busy ones, protected ones never |
| `rules` | none | up to 20, below |
| `budget` | none | USD a month; needed on a provider that charges |
| `at_cap` | `stop` | at the budget's cap: `stop` every unit, or keep `min` |
| `enabled` | `true` | off, its units are let go |

`PATCH /scaling/groups/{id}` changes one, `POST /scaling/groups/{id}/desired {"desired": 3}` asks for a number now
(within min and max), `POST /scaling/groups/{id}/pause {"paused": true, "why": "…"}` stops new launches while what runs
stays, and `DELETE /scaling/groups/{id}` lets its units go and removes it. `GET /scaling/groups/{id}/history` is what
happened to it, from the ledger. Each group says `waiting_for`: why it launches nothing now, in words.

### Rules

| Type | Fields | Proposes |
| --- | --- | --- |
| `target` | `metric`, `target` | units × metric ÷ target, rounded up: keeps the metric near the target |
| `step` | `metric`, `above` or `below`, `minutes` (default 1), `add` or `remove` (a count or a percentage like `"50%"`) | once the condition has held for `minutes`, that many more or fewer |
| `schedule` | `cron`, `timezone`, and any of `min`, `max`, `desired` | at each time the cron names, the new bounds and count |

When several rules speak, the largest number wins; the result stays within min and max. Each rule comes back with
`says`, the rule in words, and the group's `last_eval` keeps what each said last time.

| Metric | |
| --- | --- |
| `jobs_waiting` | jobs queued (for a group's `steps`, those that run them) |
| `oldest_wait_minutes` | how long the oldest queued job has waited |
| `workers_busy_share` | the share of live workers running a job |
| `worker_load` | the live workers' average load |
| `model_calls_per_minute` | model calls over the last 5 minutes |
| `model_seconds` | how long a model call took on average over the last 5 minutes |

`GET /api/v1/scaling/metrics` gives them now.

## The person approves

A group launches nothing until a person approves its **terms**: the provider, what each unit is, the max, the budget
and what happens at its cap. Settings › Auto scaling lists each group with its terms in words and an **Approve**
button; `POST /scaling/groups/{id}/approve` approves them as they stand, and only from a session signed in to the web
app: an API key, an app or an agent gets 403. A group made or changed by a person in the web app
is approved as it's saved, since that is their pick.

Within the approved terms anyone who may change the group, agents included, acts at once: asking for a number of units,
lowering the max or the budget, pausing, letting a unit go. A change that widens the terms (another provider or unit,
a higher max or budget, keeping min at the cap) takes the approval away until a person gives it again.

## The scaler

Every `tick_seconds`, for each group, the scaler:

1. puts each running unit's cost since last time into the ledger (`scaling.unit`, an estimate from its price per hour);
2. checks the group's budget and the `all` budget: at the cap the group is `capped` and lets its units go (or keeps
min, with `at_cap: "min"`);
3. looks at each unit at its provider and in Lens. A worker unit is running once its worker beats; one that hasn't
joined within `join_minutes`, or whose worker has been silent for 10 minutes, fails and is let go;
4. runs the rules and launches or lets go units to reach the desired count. A busy worker drains first: its job goes
back to the queue after the step it's on, and the unit goes once it's idle.

Three launches or joins that fail in a row pause the group, with the reason. Every launch, join, drain, termination,
failure and change is a ledger row linked to the unit, its group and its provider.

## Agents

The same in the assistant (admins, in the server tools) and over MCP: `scaling` reads everything,
`propose_scaling_group` and `change_scaling_group` make and change groups, `set_scaling_desired`,
`pause_scaling_group` and `terminate_scaling_unit` act within the approved terms. In the assistant, proposing or
changing a group waits for the person's yes; a yes on the Approvals page in the web app also approves the group's
terms, while a group the assistant makes in a setup conversation, or an agent over MCP, still waits for the person.
Turning scaling on (`change_settings scaling`) always asks.

## Refine later

Providers beyond This machine; spot and preemptible instances; warm pools; predictive scaling; health checks beyond
joining; prices fetched from providers' pricing APIs.
Loading
Loading