Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Analytics.** A new Analytics page (admins) shows active people per day, week and month, stickiness, churn rate,
retention by sign-up week and feature use, worked out on the server from the activity ledger and sign-ins. Each day
Lens also keeps a few counted events (install, daily active, feature used, upgrade, inactive, return). They can
optionally go to your own Google Analytics 4 property too; that is off by default and is set in Settings → Usage
analytics or the setup wizard ([Usage analytics](docs/analytics.md)).

## 0.9.0 <small>October 15, 2026</small> {id="0.9.0"}

- **Evidence PDFs hold everything Lens found.** After each resource's own file come pages of what Lens knows about
Expand Down
4 changes: 4 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ x-backend: &backend
LENS_TELEMETRY: ${LENS_TELEMETRY:-}
LENS_TELEMETRY_ENDPOINT: ${LENS_TELEMETRY_ENDPOINT:-}
LENS_TELEMETRY_HEADERS: ${LENS_TELEMETRY_HEADERS:-}
# opt-in usage events to your own Google Analytics 4 property (docs/analytics.md): unset, off unless turned on in Settings
LENS_ANALYTICS: ${LENS_ANALYTICS:-}
LENS_GA_MEASUREMENT_ID: ${LENS_GA_MEASUREMENT_ID:-}
LENS_GA_API_SECRET: ${LENS_GA_API_SECRET:-}
# Prometheus (docs/prometheus.md): /metrics, off unless LENS_PROMETHEUS=on with a LENS_PROMETHEUS_TOKEN; the
# monitoring profile below scrapes it and charts it in Grafana at LENS_GRAFANA_URL
LENS_PROMETHEUS: ${LENS_PROMETHEUS:-}
Expand Down
100 changes: 100 additions & 0 deletions docs/analytics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Usage analytics

Lens has built-in analytics: the **Analytics** page (admins) shows how Lens is being used, with active people, churn,
retention, which features get used and the events it counts each day. All of it is worked out and kept on your own
machine. Optionally, the same daily events also go to a **Google Analytics 4** property you own, to see them across
every install you run.

Status: **built**: the Analytics page, the daily events, the Google Analytics export, its settings and the wizard
toggle.

## The numbers

`GET /api/v1/admin/analytics?days=30` (admins) works these out from what Lens already keeps: the
[activity ledger](activity.md)'s rows for API calls a person made, and sign-ins in the audit log. Nothing new is
collected for them and nothing leaves the machine.

| Number | What it is |
| --- | --- |
| `daily` | each day of the period: how many people were active, and how many actions they took |
| `dau`, `wau`, `mau` | people active today, in the last 7 days, in the last 30 days |
| `stickiness` | average daily active people ÷ monthly active people (how much of the month a regular uses Lens) |
| `active_users`, `new_users`, `accounts` | people active in the period, accounts made in it, accounts not disabled |
| `churn` | of the people active in the period before this one (`previous_active`), how many came back (`retained`) and how many didn't (`churned`); `rate` = churned ÷ previous_active; `returning` were active now but not before |
| `retention` | for each of the last 8 weeks, the accounts made that week and the share of them active that week and each week after |
| `features` | each feature (from the API routes: recordings, chats, graph, routines…) with its actions and people |
| `last_active_day`, `days_since_active` | when anyone last used this install |

A person is active on a day when they changed something through the API or signed in. With `activity.reads` on,
reads count too. The ledger keeps `activity.keep_days` (365), which is as far back as the numbers can go.

## Events

Once a day, a worker (or `lens watch`) counts the day before and keeps the result in the `usage_event` table, with
Google Analytics on or off. The Analytics page lists them, and so does `GET /api/v1/admin/analytics/events`. Each one
shows whether it was sent to Google Analytics, is waiting to be sent, or is kept here only. Events are kept for
`activity.keep_days`.

## Google Analytics

**Off unless you turn it on.** It needs a GA4 property of your own: its **measurement id** (`G-…`) and a
**Measurement Protocol API secret** (in GA: Admin → Data streams → your stream → Measurement Protocol API secrets).

- **Settings → Usage analytics**: turn it on and set the id and secret. **Send a test event** sends one `lens_test`
event to GA's validation server, which checks it without recording it.
- **The setup wizard**, on its telemetry step.
- **.env**: `LENS_ANALYTICS=on`, `LENS_GA_MEASUREMENT_ID=G-…`, `LENS_GA_API_SECRET=…`. What .env sets wins over
Settings, which show those fields locked.

Turned on, the daily events are also sent, for the install as a whole. A send that fails is tried again 10 minutes
later. Events kept while Google Analytics was off are never sent later. These are the events:

| Event | When | Parameters |
| --- | --- | --- |
| `lens_install` | analytics first runs on this install | `version`, `platform` |
| `lens_setup_done` | the setup wizard was finished or skipped | `skipped` (0 or 1) |
| `lens_daily_active` | at least one person used Lens the day before | `active_users`, `actions` |
| `lens_feature_used` | each feature used the day before (up to 20) | `feature`, `actions`, `users` |
| `lens_upgrade` | Lens runs a new version | `from_version`, `to_version` |
| `lens_inactive` | nobody has used Lens for `analytics.inactive_days` (30) | `days` |
| `lens_return` | someone used Lens again after `lens_inactive` | `days_away` |

Every event also carries `session_id` (the day) and `engagement_time_msec`, so GA counts the install as an active
user that day, and the user properties `lens_version` and `lens_platform` (like `linux-x86_64`).

**What is sent:** the install's id (a random UUID made on first use, not derived from the machine), the version, the
platform and the counts above. **Never** a name, email, account id, title, file name, namespace, prompt or any text.
Like any web request, it reaches Google from your server's address (GA4 doesn't keep IP addresses). `analytics.events` chooses which events are sent; leave one out and it never is.

### Churn and retention in GA

Each install is one GA user. In GA:

- **Reports → Retention** shows installs coming back day after day and week after week, from `lens_daily_active`.
- **Churn rate** for a period is installs active in the period before that sent no `lens_daily_active` in it. An
Exploration (Cohort exploration, with `lens_install` as the inclusion event and `lens_daily_active` as the return
event) gives it by install week. `lens_inactive` counts installs that went quiet: a self-hosted install that was
removed can't say so, so `inactive_days` without use is the uninstall-like signal.
- **Feature use**: register `feature` as an event-scoped custom dimension (Admin → Custom definitions), and `actions`
and `active_users` as custom metrics, then break `lens_feature_used` down by `feature`.

## Settings

| Setting | Default | What it does |
| --- | --- | --- |
| `analytics.enabled` | `false` | send to Google Analytics |
| `analytics.measurement_id` | none | the GA4 measurement id, like `G-ABC123XYZ` |
| `analytics.api_secret` | none | the Measurement Protocol API secret (encrypted at rest, never shown again) |
| `analytics.region` | `global` | `eu` sends to `region1.google-analytics.com`, which keeps collection in the EU |
| `analytics.events` | all | which of the events above are sent |
| `analytics.inactive_days` | `30` | days without anyone using Lens before `lens_inactive` |

`GET /api/v1/settings/analytics/status` shows whether it is on, this install's id, which events are on, and how the
last send went. Changing these settings through the setup assistant always waits for an admin's approval, as
telemetry does.

## Refine later

- Choose the period in Settings (it shows the last 30 days).
- Per-namespace numbers.
- Optional per-person pseudonymous ids in GA, for multi-person installs.
4 changes: 3 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v
| `LENS_SETUP_SETTINGS` | | settings to start with, as JSON `{"section": {"key": value}}` (or that JSON base64url-encoded): saved once at startup as if made in the app, so unlike the variables above they stay editable there. The [web installer](get-started.md#the-web-installer) writes its answers here; a `budget` section (`hard_usd`, `soft_usd`, `period`, `on_over`, `price_in`, `price_out`) sets [the overall budget](budgets.md#one-budget-for-everything) |
| `LENS_SETUP_WIZARD` | | `off`: never show the setup wizard |
| `LENS_TELEMETRY` / `LENS_TELEMETRY_ENDPOINT` / `LENS_TELEMETRY_HEADERS` | off | opt-in OpenTelemetry traces and metrics, sent only to this OTLP/HTTP endpoint ([Telemetry](telemetry.md)); `LENS_TELEMETRY=off` keeps it off whatever Settings say |
| `LENS_ANALYTICS` / `LENS_GA_MEASUREMENT_ID` / `LENS_GA_API_SECRET` | off | opt-in usage events sent to your own Google Analytics 4 property ([Usage analytics](analytics.md)) |
| `LENS_TARGET` / `LENS_BIND` | | how install.sh set the machine up: the edition (`full` or `lean`) and where the web app's port is published (`0.0.0.0`: the network, `127.0.0.1`: this machine only). `docker-compose.prod.yml` and a microsandbox pass them on from `.env`, and Settings show them; unset, Lens says it doesn't know. They're changed in Settings, which has the install run again ([Updates](updates.md#setting-the-machine-up-again)) |
| `LENS_LOG_LEVEL` / `LENS_LOG_FORMAT` | info / text | the log level and console format for the API and every worker ([Logging](logging.md)); win over Settings → Logs |
| `FRONTEND_URL` | `http://localhost:3000` | the web app's address as people use it: links in emails, and where apps send people to sign in ([OAuth](authentication.md#oauth)) |
Expand Down Expand Up @@ -67,7 +68,8 @@ to Lens. These have defaults, so the wizard doesn't ask them; each is a click aw
* **Apps and AI**: whether apps and AI assistants (Claude, ChatGPT, Cursor and other MCP clients) may sign people in
with their Lens account ([OAuth](authentication.md#oauth)), on unless turned off, how long their tokens last, and
the MCP server's address to add to an assistant.
* **Telemetry**: off unless chosen ([Telemetry](telemetry.md)).
* **Telemetry**: off unless chosen ([Telemetry](telemetry.md)), and Google Analytics usage events, off unless
chosen ([Usage analytics](analytics.md)).

Every step can be skipped, and the whole wizard too; all of it stays in Settings. Values from the environment win and
show locked. Installs that already had accounts never see the wizard. The API side is `GET /api/v1/setup`,
Expand Down
4 changes: 4 additions & 0 deletions fastapi_backend/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ CORS_ORIGINS=["http://localhost:3000"]
# LENS_TELEMETRY_ENDPOINT=http://localhost:4318 # your OpenTelemetry collector's OTLP/HTTP address
# LENS_TELEMETRY_HEADERS= # optional: key=value pairs, comma-separated, URL-encoded

# --- Usage analytics (docs/analytics.md): counted events sent to your own Google Analytics 4 property, off unless on ---
# LENS_ANALYTICS=on
# LENS_GA_MEASUREMENT_ID=G-ABC123XYZ
# LENS_GA_API_SECRET= # a Measurement Protocol API secret from the GA data stream
# --- Logging (docs/logging.md): level and console format for the API and every worker; win over Settings → Logs ---
# LENS_LOG_LEVEL=info # debug, info, warning, error or critical
# LENS_LOG_FORMAT=text # text, or json for one JSON object per line (for a log shipper)
Expand Down
32 changes: 32 additions & 0 deletions fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
from app.core.middleware import host_name
from app.domain import (
analytics,
anytopdf,
auth,
blobs,
Expand Down Expand Up @@ -45,6 +46,9 @@
updates,
)
from app.schemas.admin import (
AnalyticsMetrics,
AnalyticsStatus,
AnalyticsTestResult,
AuditEntry,
BridgeStatus,
BridgeTestResult,
Expand Down Expand Up @@ -79,6 +83,7 @@
TelemetryTestResult,
TikaTestResult,
TunnelStatus,
UsageEvent,
)
from app.schemas.auth import AccountToken
from app.schemas.common import Ok
Expand Down Expand Up @@ -408,6 +413,33 @@ def test_telemetry(user: AdminWriter, cfg: Cfg) -> TelemetryTestResult:
return TelemetryTestResult(ok=ok, error=error, ms=ms)


@router.get("/settings/analytics/status")
def analytics_status(user: AdminReader, cfg: Cfg, db: Db) -> AnalyticsStatus:
"""Whether usage statistics go to Google Analytics, which events, and how the last send went. Off by default."""
return AnalyticsStatus.model_validate(analytics.status(db, cfg))


@router.post("/settings/analytics/test")
def test_analytics(user: AdminWriter, cfg: Cfg, db: Db) -> AnalyticsTestResult:
"""Send one lens_test event to Google Analytics' validation server (checked there, not recorded), on or off."""
ok, error, messages = analytics.test(db, cfg)
return AnalyticsTestResult(ok=ok, error=error, messages=messages)


@router.get("/admin/analytics")
def usage_metrics(user: AdminReader, db: Db, days: int = Query(30, ge=1, le=180)) -> AnalyticsMetrics:
"""Daily, weekly and monthly active people, churn, retention by sign-up week and feature use over the last `days`
days, worked out here from the activity ledger and sign-ins; nothing is sent anywhere."""
return AnalyticsMetrics.model_validate(analytics.metrics(db, days))


@router.get("/admin/analytics/events")
def usage_events(user: AdminReader, db: Db, limit: int = Query(200, ge=1, le=1000)) -> list[UsageEvent]:
"""The events Lens counted each day (install, daily active, feature used, upgrade, inactive...), newest first, and
whether each went to Google Analytics. Kept here whether or not Google Analytics is on."""
return [UsageEvent.model_validate(r) for r in analytics.recent_events(db, limit)]


@router.get("/settings/logs/status")
def log_status(user: AdminReader, cfg: Cfg) -> LogStatus:
"""The API process's log level, its log file, and how sending to syslog is going (off by default)."""
Expand Down
15 changes: 12 additions & 3 deletions fastapi_backend/app/api/v1/routes/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,14 +95,23 @@ def save_storage(body: SetupStorage, user: AdminWriter, request: Request, db: Db

@router.put("/telemetry")
def save_telemetry(body: SetupTelemetry, user: AdminWriter, request: Request, db: Db) -> SetupSaved:
"""Opt in to telemetry: traces and metrics sent only to this endpoint (an OTLP collector). Off unless chosen."""
"""Opt in to telemetry: traces and metrics sent only to this endpoint (an OTLP collector), and usage statistics
to your own Google Analytics property when `analytics` is true. Each is off unless chosen."""
try:
saved = setup.save_telemetry(db, request.app.state.archive.base, body.enabled, body.endpoint, user.email)
base = request.app.state.archive.base
saved = setup.save_telemetry(db, base, body.enabled, body.endpoint, user.email)
shared = (
setup.save_analytics(db, base, body.analytics, body.measurement_id, body.api_secret, user.email)
if body.analytics is not None
else []
)
except ValueError as e:
raise HTTPException(400, str(e)) from None
if saved:
auth.audit(db, user.as_audit(), "settings.save", "telemetry", saved)
return SetupSaved(saved=saved)
if shared:
auth.audit(db, user.as_audit(), "settings.save", "analytics", shared)
return SetupSaved(saved=saved + [f"analytics.{k}" for k in shared])


@router.put("/budget")
Expand Down
Loading
Loading