Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 61 additions & 67 deletions ci/clicks/baseline-phone.json

Large diffs are not rendered by default.

128 changes: 61 additions & 67 deletions ci/clicks/baseline.json

Large diffs are not rendered by default.

13 changes: 3 additions & 10 deletions ci/clicks/journeys.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -146,20 +146,13 @@ export const JOURNEYS = [
id: "setup-wizard",
module: "Setup",
feature: "First-run setup wizard, every step with its defaults",
note: "The namespaces come from the install's configuration, so the wizard opens on the model provider. Finish with defaults (one click) skips the remaining steps.",
note: "It asks only what has no default: the model provider, filled in from the local model found (and a budget for a cloud one). The survey's Lens started before its stand-in installer, so the model isn't set yet; after the real web installer, which sets it, the wizard doesn't open. The namespace, storage, apps and telemetry keep their defaults.",
async run(ctx) {
const { page, meter, base } = ctx;
await page.getByRole("heading", { name: "Connect a model provider" }).waitFor();
await page.getByRole("button", { name: /Ollama 2 models/ }).waitFor();
for (const [heading, button] of [
["Connect a model provider", "Save and continue"],
["Storage", "Save and continue"],
["Apps and AI assistants", "Save and continue"],
["Telemetry", "Save and finish"],
]) {
await page.getByRole("heading", { name: heading, exact: true }).waitFor();
await meter.click(page.getByRole("button", { name: button }), `${button} (${heading})`);
}
const save = page.getByRole("button", { name: "Save and continue" });
await meter.click(save, "Save and continue (Connect a model provider)");
await page.waitForURL(`${base}/`);
},
},
Expand Down
35 changes: 22 additions & 13 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,31 +42,40 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v

## First-run setup

A fresh install (no accounts when the API first starts) walks its first admin through setup in the web app:
A fresh install (no accounts when the API first starts) walks its first admin through setup in the web app, asking
only what has no answer yet and no default:

1. **Admin account**, with the one-time setup code from the log (`make setup-code`), so a stranger who finds a new
public server can't claim it. The admin signs in with a passkey ([Authentication](authentication.md#signing-in-passkeys)).
Skipped when `LENS_ADMIN_EMAIL` creates the admin at startup; then open the sign-in link the log prints (or sign
in with `LENS_ADMIN_PASSWORD`, if you set one).
2. **Namespace**: the first one, with its knowledge graph shared or isolated. Already done when archive.yaml or
`LENS_NAMESPACE` names namespaces, or an install script made one.
3. **Model provider**: an OpenAI-compatible server (OpenAI, Ollama, llama.cpp, LM Studio, vLLM), its model and key,
with a test. Servers running nearby and [cloud providers](#cloud-model-providers) are a click.
4. **Storage**: where data lives (shown; set in archive.yaml and `SURREAL_URL`), where Lens keeps its own files (this
machine or a storage connection, [Storage](storage.md)), the largest upload, and optionally a folder inside
`sources.local_roots` to watch.
5. **Apps and AI**: whether apps and AI assistants (Claude, ChatGPT, Cursor and other MCP clients) may sign people in
with their Lens account ([OAuth](authentication.md#oauth)), on unless turned off, how long their tokens last, and
the MCP server's address to add to an assistant.
6. **Telemetry**: off unless chosen ([Telemetry](telemetry.md)).
2. **Model provider**, while there is none: an OpenAI-compatible server (OpenAI, Ollama, llama.cpp, LM Studio, vLLM),
its model and key, with a test. Servers running nearby and [cloud providers](#cloud-model-providers) are a click.
3. **Budget**, for a paid (cloud) model with none ([Budgets](budgets.md#one-budget-for-everything)); a model on this
machine or its network costs nothing, so it isn't asked.

When the model is set already, by the [web installer](get-started.md#the-web-installer) (which asks for the budget
with it) or by `.env`, there is nothing left to ask: the wizard finishes at startup and the first admin goes straight
to Lens. These have defaults, so the wizard doesn't ask them; each is a click away there, and in Settings:

* **Namespace**: the first one, with its knowledge graph shared or isolated; `archive` (shared) unless chosen. Already
there when archive.yaml or `LENS_NAMESPACE` names namespaces, or an install script made one.
* **Storage**: where data lives (shown; set in archive.yaml and `SURREAL_URL`), where Lens keeps its own files (this
machine or a storage connection, [Storage](storage.md)), the largest upload, and optionally a folder inside
`sources.local_roots` to watch.
* **Apps and AI**: whether apps and AI assistants (Claude, ChatGPT, Cursor and other MCP clients) may sign people in
with their Lens account ([OAuth](authentication.md#oauth)), on unless turned off, how long their tokens last, and
the MCP server's address to add to an assistant.
* **Telemetry**: off unless chosen ([Telemetry](telemetry.md)).

Every step can be skipped, and the whole wizard too; all of it stays in Settings. Values from the environment win and
show locked. Installs that already had accounts never see the wizard. The API side is `GET /api/v1/setup`,
`POST /api/v1/setup/namespace`, `PUT /api/v1/setup/llm`, `GET /api/v1/setup/llm/detect` (model servers running on
this machine or the Docker host, with their models), `GET /api/v1/setup/llm/providers` (the
[cloud providers](#cloud-model-providers)), `PUT /api/v1/setup/storage`, `PUT /api/v1/setup/oauth`,
`PUT /api/v1/setup/telemetry` and `POST /api/v1/setup/finish`
(admins); `GET /api/v1/auth/status` says whether it is still pending (`wizard_pending`).
(admins); `GET /api/v1/setup` says what is still to ask (`todo`: `llm`, `budget`), and `GET /api/v1/auth/status`
whether the wizard is still pending (`wizard_pending`).

## Frontend environment

Expand Down
9 changes: 5 additions & 4 deletions docs/get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,10 +140,11 @@ On first start the API log prints a setup code, and a link that fills it in (`ma
```
No accounts yet. Create the first admin in the web app with setup code: … (or open http://localhost:3000/setup?code=…, which fills it in)
```
Open the link (or the web app, and enter the code) and create the admin account. A short wizard then asks for the first
namespace, the model provider, a [budget](budgets.md#one-budget-for-everything) (for a cloud model), storage, apps and
AI, and telemetry. Once a model is set, **Finish with defaults** skips the rest; change any of it later in Settings
([Configuration](configuration.md#first-run-setup) lists the `.env` values that answer it instead).
Open the link (or the web app, and enter the code) and create the admin account. A short wizard then asks only what
has no default: the model provider, and a [budget](budgets.md#one-budget-for-everything) for a cloud model. After the
web installer, which asks for both, it doesn't open at all. The first namespace, storage, apps and AI, and telemetry
keep their defaults, a click away in the wizard and in Settings ([Configuration](configuration.md#first-run-setup)
lists the `.env` values that answer it instead).

After the first sign-in, a short welcome card points to the two places most people come back for: the assistant, which
you can ask straight from the card, and **Reports**, with the charts of what is in the archive. Asking, opening
Expand Down
4 changes: 2 additions & 2 deletions docs/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ and notes' images and attachments ([Notes](notes.md#files-and-images)). **Planne
storage key.

The files Lens makes its own, starting with notes' attachments, go where **Settings → Storage** (the `files` section)
says. The setup wizard's Storage step asks the same: on this machine, or on a storage connection, which can be added
right there (the same dialog as Sources) and is checked before it's saved.
says. The setup wizard's Storage step (not asked, a click away there) offers the same: on this machine, or on a storage
connection, which can be added right there (the same dialog as Sources) and is checked before it's saved.

| Setting | |
|---|---|
Expand Down
2 changes: 1 addition & 1 deletion docs/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Any of these:
- **Settings → Telemetry**: turn on **Send telemetry** and set the **OTLP endpoint**. **Send a test span** checks the
address and headers first, even while it is off. The section shows whether it is on, where it sends, and how the
last exports went.
- **The setup wizard** asks on a fresh install (its last step); it is off unless you choose on.
- **The setup wizard**'s Telemetry step, a click away on a fresh install (it isn't asked: off is its default).
- **.env**, for the API and every worker:

```sh
Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/core/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ def prepare(self) -> None:
if auth.account_count(self.db) == 0:
setup.mark_fresh(self.db, self.base) # a fresh install: the web app walks the first admin through setup
setup.apply_env(self.db, self.base)
setup.finish_if_ready(self.db, self.base) # the installer (or .env) answered everything the wizard asks
if settings.keep_passwords(self.db, self.base):
log.info("People here sign in with passwords, so passwords stay on; an admin can turn them off in Settings > Sign-in")
if auth.account_count(self.db) == 0:
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/domain/ops_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,8 @@ def status(db, cfg, base):
return {
"missing": missing,
"setup_wizard_pending": view["pending"],
# what the wizard still asks: llm (the model provider), budget (for a paid model)
"setup_todo": view["todo"],
"model_provider": {k: v for k, v in view["llm"]["values"].items() if k != "api_key"}
| {"api_key_set": bool((view["llm"]["values"].get("api_key") or {}).get("set")), "locked_by_env": view["llm"]["locked"]},
"namespaces": sorted(names.values()),
Expand Down
44 changes: 44 additions & 0 deletions fastapi_backend/app/domain/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,12 @@

import base64
import binascii
import ipaddress
import json
import logging
import os
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor

Expand Down Expand Up @@ -62,6 +64,47 @@ def mark_fresh(db, cfg=None):
settings.save(db, cfg, "encryption", {"files": True}, user="setup")


def local_model(base_url):
"""Whether a model server is on this machine or its network (free to call), not a paid cloud provider: a name
without a dot (a container next to Lens, localhost), a private or loopback address, or a .local, .internal, .lan
or .home.arpa name. The web app's isLocalModel (setup-wizard.tsx) says the same."""
host = (urllib.parse.urlsplit(base_url or "").hostname or "").strip("[]")
if not host:
return False
if "." not in host and ":" not in host:
return True
try:
ip = ipaddress.ip_address(host)
except ValueError:
return host.endswith((".local", ".internal", ".lan", ".home.arpa"))
return ip.is_private or ip.is_loopback or ip.is_unspecified


def todo(db, cfg, shown=None):
"""What the wizard still has to ask, in order: only what has no answer yet and no default. The model provider,
while there is none (then the budget too, if the one chosen is paid); the overall budget, for a paid model with
none, unless the web installer set that model (it asks for the budget with it). Everything else (the first
namespace, storage, apps signing in, telemetry) has a default and stays in Settings, so it isn't asked."""
shown = shown or settings.effective(db, cfg, reveal=False)
llm = shown["llm"]
if not (llm.get("base_url") and llm.get("model")):
return ["llm"]
seeded = db.one("SELECT sections FROM $r", r=R(*SEEDED)) or {}
if not local_model(llm["base_url"]) and not budgets.get(db, budgets.ALL) and "llm" not in (seeded.get("sections") or []):
return ["budget"]
return []


def finish_if_ready(db, cfg):
"""Finish the wizard on its own when there is nothing left to ask (todo), as after the web installer: the first
admin then goes straight to Lens. Returns whether it did."""
if not pending(db) or todo(db, cfg):
return False
finish(db, "setup")
log.info("Setup has nothing left to ask (the model is set), so the setup wizard won't open; everything is in Settings")
return True


def finish(db, user=None, skipped=False):
"""Finish (or skip) the wizard. An archive still without a namespace gets the default one, so importing, watching
folders and the graph have somewhere to go without asking first. Returns its name when it made one."""
Expand Down Expand Up @@ -167,6 +210,7 @@ def view(db, cfg):
admin = env_admin()
return {
"pending": pending(db),
"todo": todo(db, cfg, shown),
"admin": {"from_env": bool(admin)},
"namespace": {
"existing": names,
Expand Down
3 changes: 3 additions & 0 deletions fastapi_backend/app/schemas/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ class SetupBudgetView(ResponseModel):

class SetupView(ResponseModel):
pending: bool
# the steps the wizard still has to ask, in order: "llm" (the model provider) while there is none, "budget" for a
# paid model with no overall budget; the rest has defaults and stays in Settings
todo: list[Literal["llm", "budget"]] = []
admin: SetupAdmin
namespace: SetupNamespaceView
llm: SetupLlmView
Expand Down
53 changes: 53 additions & 0 deletions fastapi_backend/tests/api/test_setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -309,3 +309,56 @@ def test_log_prints_a_setup_link(fresh, caplog, monkeypatch):
with caplog.at_level("WARNING", logger="lens"):
fresh()
assert "with setup code: a b&c (or open http://localhost:3000/setup?code=a+b%26c, which fills it in)" in caplog.text


def test_the_wizard_asks_only_what_has_no_answer(fresh):
"""The model provider while there is none, then the budget for a paid one; the rest has defaults (todo)."""
app, client = fresh()
code = app.state.archive.setup_code
r = client.post("/api/v1/auth/setup", json={"code": code, "email": "ada@x.io", "password": "admin password 1"})
h = {"Authorization": f"Bearer {r.json()['access_token']}"}
assert client.get("/api/v1/setup", headers=h).json()["todo"] == ["llm"]
client.put("/api/v1/setup/llm", json={"base_url": "http://ollama:11434/v1", "model": "llama3"}, headers=h)
assert client.get("/api/v1/setup", headers=h).json()["todo"] == [] # a model on this network costs nothing
client.put("/api/v1/setup/llm", json={"base_url": "https://api.openai.com/v1", "model": "gpt-x", "api_key": "sk"}, headers=h)
assert client.get("/api/v1/setup", headers=h).json()["todo"] == ["budget"]
client.put("/api/v1/setup/budget", json={"hard_usd": 25}, headers=h)
assert client.get("/api/v1/setup", headers=h).json()["todo"] == []
assert client.get("/api/v1/auth/status").json()["wizard_pending"] is True # until finished (or the next start)


def test_the_installers_model_leaves_nothing_to_ask(fresh, monkeypatch, caplog):
"""With the web installer's model (and its budget question asked there), the wizard finishes at startup: the first
admin goes straight to Lens, which has the default namespace."""
monkeypatch.setenv("LENS_SETUP_SETTINGS", '{"llm": {"base_url": "https://api.openai.com/v1", "model": "gpt-x"}}')
with caplog.at_level("INFO", logger="lens"):
app, client = fresh()
assert client.get("/api/v1/auth/status").json() == {"setup_required": True, "wizard_pending": False, "passwords": False}
assert "the setup wizard won't open" in caplog.text
assert sorted(store.space_names(app.state.db).values()) == ["archive"]


def test_a_paid_model_from_the_environment_still_asks_for_a_budget(fresh, monkeypatch):
from app.domain import setup

monkeypatch.setenv("LENS_LLM_BASE_URL", "https://api.openai.com/v1")
monkeypatch.setenv("LENS_LLM_MODEL", "gpt-x")
app, client = fresh()
assert client.get("/api/v1/auth/status").json()["wizard_pending"] is True
assert setup.todo(app.state.db, app.state.archive.base) == ["budget"]


def test_tells_a_model_on_this_network_from_a_cloud_provider():
from app.domain import setup

for u in (
"http://localhost:11434/v1",
"http://host.docker.internal:1234/v1",
"http://ollama:11434/v1",
"http://192.168.1.5:8080/v1",
"http://[::1]:11434",
"http://nas.lan/v1",
):
assert setup.local_model(u), u
for u in ("https://api.openai.com/v1", "https://openrouter.ai/api/v1", "", None, "http://8.8.8.8/v1"):
assert not setup.local_model(u), u
Loading
Loading