Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,24 +189,27 @@ services:
networks: [lens]

# Prometheus, node-exporter and Grafana (docs/prometheus.md): charts of Lens and this machine over time, ready made.
# Opt-in: in .env set COMPOSE_PROFILES=monitoring, LENS_PROMETHEUS=on, LENS_PROMETHEUS_TOKEN (openssl rand -hex 32)
# and LENS_GRAFANA_URL=http://localhost:3001, then `docker compose up -d`. Grafana listens where the web app does
# (LENS_BIND), on LENS_GRAFANA_PORT (3001); anyone who reaches it may view, and only the admin may change things.
# Opt-in: Admin → System health or Settings → Telemetry turns it on (the upgrade agent adds `monitoring` to
# COMPOSE_PROFILES and starts these). Prometheus scrapes with the token Lens writes to .lens-control/prometheus-token,
# or LENS_PROMETHEUS_TOKEN when set. Grafana listens where the web app does (LENS_BIND), on LENS_GRAFANA_PORT (3001);
# anyone who reaches it may view, and only the admin may change things.
prometheus:
image: prom/prometheus:${PROMETHEUS_VERSION:-v3.5.0}
profiles: [monitoring]
security_opt: ["no-new-privileges:true"]
entrypoint: ["/bin/sh", "-c"]
command:
- >-
printf %s "$$LENS_PROMETHEUS_TOKEN" > /prometheus/lens-token &&
{ if [ -n "$$LENS_PROMETHEUS_TOKEN" ]; then printf %s "$$LENS_PROMETHEUS_TOKEN" > /prometheus/lens-token;
else ln -sf /control/prometheus-token /prometheus/lens-token; fi; } &&
exec /bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus
--storage.tsdb.retention.time=$${LENS_METRICS_DAYS:-30}d
environment:
LENS_PROMETHEUS_TOKEN: ${LENS_PROMETHEUS_TOKEN:-}
LENS_METRICS_DAYS: ${LENS_METRICS_DAYS:-30}
volumes:
- ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./.lens-control:/control:ro # prometheus-token, written by Lens when monitoring is turned on
- prometheus-data:/prometheus
restart: unless-stopped
networks: [lens]
Expand Down
35 changes: 20 additions & 15 deletions docs/prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,26 @@ endpoint: Lens sends nothing anywhere. For pushing traces and metrics to an Open
Lens doesn't build its own charts: with Docker, the `monitoring` profile runs [Prometheus](https://prometheus.io/),
[node-exporter](https://github.com/prometheus/node_exporter) (the machine's CPU, memory, disk and network) and
[Grafana](https://grafana.com/oss/grafana/) next to Lens, already connected, with the Lens dashboard as Grafana's home
page. Add to `.env`:

```sh
COMPOSE_PROFILES=monitoring # with others, comma-separated (matterbridge,monitoring)
LENS_PROMETHEUS=on
LENS_PROMETHEUS_TOKEN=3f9c... # openssl rand -hex 32
LENS_GRAFANA_URL=http://localhost:3001 # the address you open Grafana at; System health links to it
# GRAFANA_ADMIN_PASSWORD=... # optional; otherwise admin/admin, and Grafana asks to change it
# LENS_METRICS_DAYS=30 # how long Prometheus keeps history
```

then `docker compose -f docker-compose.prod.yml up -d`. Grafana listens where the web app does (`LENS_BIND`: this
machine only unless Lens was opened to the network) on port 3001 (`LENS_GRAFANA_PORT`). Anyone who reaches it can view
the dashboards, which hold no personal content; only Grafana's admin can change them. Prometheus and node-exporter are
reachable only inside Lens's Docker network.
page.

**Turn it on in the web app**: **Admin → System health** or **Settings → Telemetry → Prometheus**, switch on
**Charts and history**. That's all: Lens turns `/metrics` on, makes a scrape token if there is none, and the machine's
upgrade agent (the one that [installs new releases](updates.md)) adds `monitoring` to `COMPOSE_PROFILES` in `.env` and
starts the three containers; the page shows how it goes and then links to Grafana. Off stops and removes them (their
history stays in the `prometheus-data` volume) and turns `/metrics` off again if this switch turned it on. Agents do
the same with `PUT /api/v1/admin/monitoring` (`{"on": true}`) or the MCP tools `monitoring` and `set_monitoring`.

Grafana listens where the web app does (`LENS_BIND`: this machine only unless Lens was opened to the network) on port
3001 (`LENS_GRAFANA_PORT`), and System health links to it on the address you opened Lens at, unless
**Grafana address** says otherwise. Anyone who reaches it can view the dashboards, which hold no personal content;
only Grafana's admin can change them (admin/admin until changed, or `GRAFANA_ADMIN_PASSWORD`). Prometheus and
node-exporter are reachable only inside Lens's Docker network. Prometheus reads the scrape token from
`.lens-control/prometheus-token`, which Lens keeps in step with Settings (or from `LENS_PROMETHEUS_TOKEN` when set), and
keeps 30 days of history (`LENS_METRICS_DAYS`).

Without an agent (Lens not installed with `install.sh`, or it couldn't set one up) the switch shows the one line to
run on the machine instead: `docker compose -f docker-compose.prod.yml --profile monitoring up -d`. An install in a
microsandbox has no Docker to run them in.

**Admin → System health** shows the server now (CPU, memory, load, uptime) and links to Grafana for history.

Expand Down
5 changes: 5 additions & 0 deletions docs/updates.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ The containers see the control folder as `/control` (`LENS_CONTROL_DIR`). `POST
The agent installs only a release tag (`vX.Y.Z`) or the branch the install already follows, whoever writes the
request, and the API refuses a release older than the one running (restore a backup instead).

The same agent turns [Charts and history](prometheus.md#prometheus-and-grafana-ready-made) on and off: a request with
`action=services` and `monitoring=on` or `off` (in the same inbox file, the one every agent watches) adds or removes
`monitoring` in `COMPOSE_PROFILES` in `.env` and starts or stops Prometheus, node-exporter and Grafana, and nothing
else; it reports in `services-status` and `services.log`.

With no agent (Lens wasn't installed with install.sh, the machine has neither systemd nor launchd, or install.sh ran
with `LENS_UPGRADE_AGENT=0`), the API gives the one line to run on the machine instead (`manual`).

Expand Down
36 changes: 36 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
filetree,
graph_history,
library,
monitoring,
notebook,
rdf,
render,
Expand Down Expand Up @@ -1600,3 +1601,38 @@ def set_file_storage(
settings.save(ctx.db, ctx.request.app.state.archive.base, "files", changes, ctx.user.email)
auth.audit(ctx.db, ctx.user.as_audit(), "settings.save", "files", sorted(changes) + ["mcp"])
return {"saved": changes, "check": checked}


# ---------- monitoring: Prometheus and Grafana next to Lens (admins) ----------
@tool(
"monitoring",
"Monitoring",
"Whether Prometheus, node-exporter and Grafana run next to Lens (charts of Lens and this machine over time), how "
"the machine's last change went, and Grafana's port. Admins only.",
)
def monitoring_status(ctx: Context) -> dict[str, Any]:
if not ctx.user.admin:
raise ToolError("only admins see or change monitoring")
return monitoring.view(ctx.db) | {"settings": f"{ctx.web}/settings/telemetry"}


@tool(
"set_monitoring",
"Turn monitoring on or off",
"Have this machine start (on true) or stop Prometheus, node-exporter and Grafana. On also turns /metrics on with "
"a scrape token when it has none. The machine's upgrade agent does it; monitoring follows along. Admins only, with "
"a token that has the write scope.",
Arg("on", "boolean", "start (true) or stop (false)", required=True),
writes=True,
)
def set_monitoring(ctx: Context, on: bool) -> dict[str, Any]:
if not ctx.user.admin:
raise ToolError("only admins see or change monitoring")
if not ctx.user.can_write:
raise ToolError("this token is read-only: ask for a token with the write scope to change monitoring")
try:
got = monitoring.turn(ctx.db, ctx.request.app.state.archive.base, ctx.cfg, bool(on), by=ctx.user.email)
except monitoring.MonitoringError as e:
raise ToolError(str(e)) from None
auth.audit(ctx.db, ctx.user.as_audit(), "lens.monitoring", None, {"on": bool(on), "via": "mcp"})
return got
3 changes: 3 additions & 0 deletions fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
local_llm,
machine,
matterbridge,
monitoring,
semantic,
settings,
sources,
Expand Down Expand Up @@ -92,6 +93,8 @@ def update_settings(section: str, body: dict[str, Any], user: AdminWriter, reque
with domain_errors():
settings.save(db, request.app.state.archive.base, section, body, user.email)
auth.audit(db, user.as_audit(), "settings.save", section, sorted(body))
if section == "telemetry" and "prometheus_token" in body:
monitoring.sync_token(request.app.state.settings.current()) # the monitoring Prometheus scrapes with the new one
if section == "encryption" and "files" in body and bool(body["files"]) != bool(files_were):
cfg = request.app.state.settings.current()
keyring.convert(db, cfg, bool(cfg["encryption"].get("files"))) # the files already kept follow, in the background
Expand Down
25 changes: 22 additions & 3 deletions fastapi_backend/app/api/v1/routes/updates.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@

import contextlib

from fastapi import APIRouter, HTTPException
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import FileResponse

from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
from app.domain import auth, backups, updates
from app.domain import auth, backups, monitoring, updates
from app.schemas.backups import BackupSchedule, RestoreWork
from app.schemas.common import Ok
from app.schemas.updates import Backup, Backups, UpdateStatus, UpgradeRequest
from app.schemas.updates import Backup, Backups, Monitoring, MonitoringRequest, UpdateStatus, UpgradeRequest

router = APIRouter(tags=["updates"])

Expand Down Expand Up @@ -46,6 +46,25 @@ def start_upgrade(user: AdminWriter, db: Db, cfg: Cfg, body: UpgradeRequest | No
return UpdateStatus.model_validate(updates.view(db, cfg))


@router.get("/admin/monitoring")
def get_monitoring(user: AdminReader, db: Db) -> Monitoring:
"""Whether Prometheus, node-exporter and Grafana run next to Lens (the monitoring profile), and how the machine's
last change went."""
return Monitoring.model_validate(monitoring.view(db))


@router.put("/admin/monitoring", status_code=202)
def set_monitoring(body: MonitoringRequest, user: AdminWriter, request: Request, db: Db, cfg: Cfg) -> Monitoring:
"""Have this machine start or stop Prometheus, node-exporter and Grafana. On also turns /metrics on with a scrape
token when it has none. GET /admin/monitoring follows along."""
try:
got = monitoring.turn(db, request.app.state.archive.base, cfg, body.on, by=user.email)
except monitoring.MonitoringError as e:
raise HTTPException(409, str(e)) from None
auth.audit(db, user.as_audit(), "lens.monitoring", None, {"on": body.on})
return Monitoring.model_validate(got)


@router.get("/admin/backups")
def list_backups(user: AdminReader, db: Db, cfg: Cfg) -> Backups:
"""The database backups in <data_dir>/backups, newest first; those being written; the restore running here (or
Expand Down
131 changes: 131 additions & 0 deletions fastapi_backend/app/domain/monitoring.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
"""Turning the monitoring profile (Prometheus, node-exporter, Grafana) on and off from the web app (docs/prometheus.md).

The containers can't start other containers, so the machine's upgrade agent does it, as it installs releases
(updates.py, packaging/host/lens-upgrade.sh). Turning monitoring on makes sure /metrics is on with a scrape token
(generated when none is set), writes that token where the monitoring Prometheus reads it (`<control>/prometheus-token`),
then drops a request in the agent's inbox: the agent adds `monitoring` to COMPOSE_PROFILES in the install's .env and
starts the three containers; off removes it and stops them. How it went is in `<control>/services-status`.

The request goes in the inbox under the same name as an upgrade (`inbox/upgrade`, with `action=services`), because that
file is what every agent installed so far watches for; the agent never installs anything for it.
"""

from __future__ import annotations

import datetime as dt
import os
import secrets

from . import settings, store, updates

ROW = store.R("app_service", "monitoring")
ACTIVE = ("queued", "running")
TOKEN_FILE = "prometheus-token"
STATUS_FILE = "services-status"
GRAFANA_PORT = 3001


class MonitoringError(RuntimeError):
"""Monitoring can't be turned on or off; the message says why, for the admin."""


def _row(db):
return db.one("SELECT * OMIT id FROM $r", r=ROW) or {}


def _put(db, run):
db.q("UPSERT $r CONTENT $d", r=ROW, d=run)


def manual():
"""The line to run on the machine when there is no agent."""
where = os.environ.get("LENS_HOST_DIR") or ""
cd = f"cd {where} && " if where and updates.REF.match(where.lstrip("/~")) else ""
return f"{cd}docker compose -f docker-compose.prod.yml --profile monitoring up -d"


def view(db):
"""Whether monitoring is on (as last asked), and how the agent's last run went."""
run = _row(db)
out = {
"on": bool(run.get("on")),
"state": run.get("state"),
"error": run.get("error"),
"asked_at": run.get("asked_at"),
"finished_at": None,
"grafana_port": GRAFANA_PORT,
"agent": updates.agent(),
"manual": manual(),
}
d = updates.control_dir()
if d and run:
s = updates._kv(d / STATUS_FILE)
if s.get("id") == run.get("run"):
out["state"] = s.get("state") or out["state"]
out["finished_at"] = s.get("finished_at")
if s.get("grafana_port", "").isdigit():
out["grafana_port"] = int(s["grafana_port"])
if s.get("state") == "failed":
out["error"] = s.get("error") or f"the machine couldn't change it (exit {s.get('code', '?')})"
if out["state"] == "queued" and (d / "inbox" / "upgrade").exists():
try:
waited = (dt.datetime.now(dt.timezone.utc) - dt.datetime.fromisoformat(str(run.get("asked_at")))).total_seconds()
except ValueError:
waited = 0
if waited > updates.PICKUP_SECONDS:
out["error"] = "the machine hasn't picked it up yet: is its upgrade agent running?"
return out


def sync_token(cfg, create=False):
"""Keep the token the monitoring Prometheus scrapes with the same as /metrics's, once monitoring was turned on."""
d = updates.control_dir()
token = ((cfg or {}).get("telemetry") or {}).get("prometheus_token")
if not d or not d.is_dir():
return
f = d / TOKEN_FILE
if not create and not f.exists():
return
if not token:
f.unlink(missing_ok=True)
return
part = d / f".{TOKEN_FILE}.{secrets.token_hex(4)}"
part.write_text(str(token), encoding="utf-8")
part.chmod(0o644) # Prometheus runs as its own user; the token reads metrics only
part.replace(f)


def turn(db, base, cfg, on, by=None):
"""Ask the machine to start (on) or stop the monitoring containers; returns view()."""
if not updates.agent():
raise MonitoringError("this machine has no upgrade agent to start containers: run the line shown on it instead")
d = updates.control_dir()
if (d / "inbox" / "upgrade").exists():
raise MonitoringError("the machine is busy with another request (an upgrade?): try again in a minute")
run, now_ = _row(db), view(db)
if now_["state"] in ACTIVE and not now_["error"]:
raise MonitoringError("the machine is still changing it: wait for that to finish")
t = cfg.get("telemetry") or {}
locked = settings.locked("telemetry")
changes, ours = {}, bool(run.get("turned_on_metrics"))
if on:
if not t.get("prometheus") and "prometheus" not in locked:
changes["prometheus"] = True
ours = True
if not t.get("prometheus_token") and "prometheus_token" not in locked:
changes["prometheus_token"] = secrets.token_hex(32)
elif ours and t.get("prometheus") and "prometheus" not in locked:
changes["prometheus"] = False # monitoring turned /metrics on: off again with it
ours = False
if changes:
settings.save(db, base, "telemetry", changes, by)
if on:
token = changes.get("prometheus_token") or t.get("prometheus_token")
sync_token({"telemetry": {"prometheus_token": token}}, create=True)
rid = secrets.token_hex(8)
now = dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds")
part = d / "inbox" / f".upgrade.{rid}"
part.write_text(f"id={rid}\naction=services\nmonitoring={'on' if on else 'off'}\nby={by or ''}\nat={now}\n", encoding="utf-8")
part.replace(d / "inbox" / "upgrade")
_put(db, {"run": rid, "on": bool(on), "by": by, "asked_at": now, "state": "queued", "error": None, "turned_on_metrics": ours})
return view(db)
15 changes: 15 additions & 0 deletions fastapi_backend/app/schemas/updates.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,18 @@ class RemoteBackups(ResponseModel):
last_name: str | None = None
last_error: str | None = Field(None, description="why copying the last backup failed")
copying: CopyWork


class Monitoring(ResponseModel):
on: bool = Field(description="the monitoring profile (Prometheus, node-exporter, Grafana) as last asked")
state: Literal["queued", "running", "done", "failed"] | None = Field(None, description="how the machine's last change went")
error: str | None = None
asked_at: str | None = None
finished_at: str | None = None
grafana_port: int = Field(3001, description="the port Grafana listens on, where the web app does")
agent: str | None = Field(None, description="what changes it on this machine (systemd, launchd); none: run `manual` there")
manual: str = Field(description="the line that turns it on by hand, on the machine Lens runs on")


class MonitoringRequest(RequestModel):
on: bool = Field(description="start (true) or stop (false) Prometheus, node-exporter and Grafana")
11 changes: 11 additions & 0 deletions fastapi_backend/tests/api/test_mcp.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ def test_clients_are_sent_to_sign_in(client, app, env):
"write_note",
"create_podcast",
"set_file_storage",
"set_monitoring",
]
assert next(t for t in tools if t["name"] == "search")["inputSchema"]["required"] == ["query"]
# API tokens and sessions work too
Expand Down Expand Up @@ -459,6 +460,16 @@ def test_file_storage_tools(client, db, env, folder):
assert "only admins" in tool_error(client, env["h"], "set_file_storage", store="local")


def test_monitoring_tools(client, db, env, monkeypatch):
monkeypatch.delenv("LENS_CONTROL_DIR", raising=False)
assert "only admins" in tool_error(client, env["h"], "monitoring")
make_user(db, "root@x.io", "root password 1", admin=True)
ha = login(client, "root@x.io", "root password 1")
got = tool(client, ha, "monitoring")
assert (got["on"], got["agent"]) == (False, None) and got["settings"].endswith("/settings/telemetry")
assert "no upgrade agent" in tool_error(client, ha, "set_monitoring", on=True)


def test_what_the_archive_says_reaches_agents_as_data(client, db, cfg, env):
said = f"Mallory: {injections.PLAIN}\nMallory: {injections.CHATML}\nMallory: {injections.SMUGGLED}"
rid = ingest.import_text(db, cfg, "pods", said, title="Mallory's talk")
Expand Down
Loading
Loading