Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
- bump the python-minor group ([#253](https://github.com/adeelahmad/lense/pull/253))
- bump the javascript-minor group ([#316](https://github.com/adeelahmad/lense/pull/316))

- **Logs in the web app, and to a syslog server.** The API and every worker now write structured log lines with a
level set in **Settings → Logs** (and per logger, like `httpx warning`), to the console as text or JSON and to a file
per process under `data_dir/logs`. The same page shows the latest lines from all of them, with search and filters.
Turned on there, lines go to a syslog server too (RFC 5424 over UDP, TCP or TLS), from a background queue so a slow
server never holds Lens up; off by default. Keys, tokens and passwords are replaced with *** before any line is
written or sent ([Logging](docs/logging.md)).

## 0.7.0 <small>October 10, 2026</small> {id="0.7.0"}

- **Prometheus metrics.** Lens can serve `/metrics` for Prometheus and Grafana: jobs by status and pipeline, imports,
Expand Down
1 change: 1 addition & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ with AES-GCM and are write-only: the API reports whether one is set, never its v
| `LENS_SETUP_SETTINGS` | | settings to start with, as JSON `{"section": {"key": value}}` (or that JSON base64url-encoded): saved once at startup as if made in the app, so unlike the variables above they stay editable there. The [web installer](get-started.md#the-web-installer) writes its answers here; a `budget` section (`hard_usd`, `soft_usd`, `period`, `on_over`, `price_in`, `price_out`) sets [the overall budget](budgets.md#one-budget-for-everything) |
| `LENS_SETUP_WIZARD` | | `off`: never show the setup wizard |
| `LENS_TELEMETRY` / `LENS_TELEMETRY_ENDPOINT` / `LENS_TELEMETRY_HEADERS` | off | opt-in OpenTelemetry traces and metrics, sent only to this OTLP/HTTP endpoint ([Telemetry](telemetry.md)); `LENS_TELEMETRY=off` keeps it off whatever Settings say |
| `LENS_LOG_LEVEL` / `LENS_LOG_FORMAT` | info / text | the log level and console format for the API and every worker ([Logging](logging.md)); win over Settings → Logs |
| `FRONTEND_URL` | `http://localhost:3000` | the web app's address as people use it: links in emails, and where apps send people to sign in ([OAuth](authentication.md#oauth)) |
| `CORS_ORIGINS` | `["http://localhost:3000"]` | origins allowed to call the API from a browser |
| `OPENAPI_URL` | `/openapi.json` | `""` disables `/docs` and the schema |
Expand Down
60 changes: 60 additions & 0 deletions docs/logging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Logging

Every Lens process (the API, each `lens worker`, `lens watch`) writes one structured record per log line, and sends it
to up to three places:

- **The console**, which is what `docker logs` and `docker compose logs` show: as text, or as one JSON object per line
for a log shipper (**Format**).
- **A file per process** under `data_dir/logs` (`api-<host>.jsonl`, `worker-<host>.jsonl`, …), one JSON object per
line, rotated at **File size** and kept **Files kept** times. **Settings → Logs** reads these, so its viewer shows the
workers' lines as well as the API's. A file no process has written for 14 days (an old container's) is removed.
- **A syslog server**, off by default (below).

Change any of it in **Settings → Logs**; the API and every worker follow within a few seconds, without a restart.
`LENS_LOG_LEVEL` and `LENS_LOG_FORMAT` in .env win over Settings, which then show those fields locked.

## What a record holds

```json
{"ts": "2026-10-10T20:02:19.809Z", "level": "warning", "logger": "lens.email", "msg": "…",
"process": "worker", "pid": 42, "host": "lens-worker-1", "exc": "Traceback …", "fields": {"job": 7}}
```

`exc` is there when the line carries an error's traceback, and `fields` when the code logged extra values.

**No secrets.** Before a line goes anywhere it is redacted: bearer and basic credentials, `key=value` pairs whose key
names a key, token, secret, password, cookie or authorization, the password in a URL (`https://user:***@host`), and
well-known key shapes (`sk-…`, `ghp_…`, `xox…-`, JSON web tokens) become `***`. Extra fields whose name says key, token,
secret or password are dropped to `***` too.

## Levels

**Level** (info by default) is what is kept on the console and in the files. **Logger levels** gives single loggers
their own, one per line, like `httpx warning` to quiet every outgoing request, or `lens.bridge debug` to follow the
chat-room bridge closely. Lens's own loggers start with `lens`; the web server's access log is `uvicorn.access`.

## Sending to a syslog server

Turn on **Send to syslog** and set the server's **Host** and **Port**. Lens speaks
[RFC 5424](https://www.rfc-editor.org/rfc/rfc5424) over:

| Transport | Standard | Framing |
| --- | --- | --- |
| UDP (default, port 514) | RFC 5426 | one line per datagram |
| TCP | RFC 6587 | octet counting (`<length> <message>`), or one line per newline for servers that want that |
| TLS (usually port 6514) | RFC 5425 | as TCP; the server's certificate is checked unless **Verify the certificate** is off |

Each message is `<PRI>1 TIMESTAMP HOST APP-NAME PROCESS.PID LOGGER - MESSAGE`, where PRI combines **Facility**
(local0 by default) with the line's level, APP-NAME is **App name** (`lens`), and MESSAGE is the line (and its
traceback) or, with **Message format** JSON, the whole record above. Only lines at **Syslog level** or above are sent.
Messages longer than 8 KB are cut.

Lines are queued and sent from a background thread, so a slow or unreachable server never holds up Lens. While the
server can't be reached Lens retries, waiting up to 30 seconds between tries, and once 10,000 lines are waiting new
ones are dropped and counted. **Send a test line** sends one message to the saved server, even while sending is off,
and Settings → Logs shows how many lines the API has sent and dropped and the last error.

rsyslog, syslog-ng, Graylog, Grafana Loki (through promtail or Alloy's syslog receiver), Splunk, Papertrail and most
log tools take RFC 5424 syslog.

Not yet built: sending straight to Loki or another HTTP endpoint, and client certificates for TLS.
4 changes: 4 additions & 0 deletions fastapi_backend/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,9 @@ CORS_ORIGINS=["http://localhost:3000"]
# LENS_TELEMETRY_ENDPOINT=http://localhost:4318 # your OpenTelemetry collector's OTLP/HTTP address
# LENS_TELEMETRY_HEADERS= # optional: key=value pairs, comma-separated, URL-encoded

# --- Logging (docs/logging.md): level and console format for the API and every worker; win over Settings → Logs ---
# LENS_LOG_LEVEL=info # debug, info, warning, error or critical
# LENS_LOG_FORMAT=text # text, or json for one JSON object per line (for a log shipper)

# Uncomment to disable /docs and /openapi.json
# OPENAPI_URL=""
36 changes: 35 additions & 1 deletion fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import pathlib
import shutil
import time
from typing import Any
from typing import Any, Literal

from fastapi import APIRouter, BackgroundTasks, HTTPException, Query, Request

Expand All @@ -24,6 +24,7 @@
keyring,
llm,
local_llm,
logs,
machine,
mailbox,
matterbridge,
Expand Down Expand Up @@ -57,6 +58,9 @@
IndexQueued,
LlmTestResult,
LocalLlmStatus,
LogLine,
LogStatus,
LogTestResult,
MailboxStatus,
MailTestResult,
NfsStatus,
Expand Down Expand Up @@ -363,6 +367,36 @@ def test_telemetry(user: AdminWriter, cfg: Cfg) -> TelemetryTestResult:
return TelemetryTestResult(ok=ok, error=error, ms=ms)


@router.get("/settings/logs/status")
def log_status(user: AdminReader, cfg: Cfg) -> LogStatus:
"""The API process's log level, its log file, and how sending to syslog is going (off by default)."""
return LogStatus.model_validate(logs.status(cfg))


@router.post("/settings/logs/test")
def test_logs(user: AdminWriter, cfg: Cfg) -> LogTestResult:
"""Send one line to the saved syslog server now (on or off), to check the address, port and transport."""
ok, error, ms = logs.test_send(cfg)
return LogTestResult(ok=ok, error=error, ms=ms)


@router.get("/admin/logs")
def read_logs(
user: AdminReader,
cfg: Cfg,
limit: int = Query(200, ge=1, le=2000),
level: Literal["debug", "info", "warning", "error", "critical"] | None = Query(None, description="this level and above"),
q: str | None = Query(None, max_length=200, description="text the message or error contains"),
process: str | None = Query(None, max_length=40, description="api, worker or watcher"),
logger: str | None = Query(None, max_length=100, description="a logger name, or the start of one (lens.email)"),
before: str | None = Query(None, max_length=40, description="a line's ts: only older lines (the next page)"),
) -> list[LogLine]:
"""The latest log lines from the API and every worker (their files under data_dir/logs), newest first. Lines are
redacted as they're written: no keys, tokens or passwords."""
rows = logs.read(cfg, limit, level, q, process, logger, before)
return [LogLine.model_validate(r) for r in rows]


@router.get("/audit")
def list_audit(user: AdminReader, db: Db, limit: int = Query(200, ge=1, le=1000)) -> list[AuditEntry]:
"""Who changed what, newest first."""
Expand Down
35 changes: 20 additions & 15 deletions fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,7 @@ def platform_main(argv, config):
https,
jobs,
local_llm,
logs,
nfs_share,
notify,
passkeys,
Expand Down Expand Up @@ -372,24 +373,26 @@ def password():
print(f"password changed for {a.email}")
elif a.cmd == "worker":
telemetry.set_role("worker")
wk = jobs.Worker(db, C, a.name, a.steps.split(",") if a.steps else None, log=print)
logs.setup("worker") # every line as a record: the console, the log file and syslog (docs/logging.md)
say = logs.printer("lens.worker")
wk = jobs.Worker(db, C, a.name, a.steps.split(",") if a.steps else None, log=say)
if a.once:
print(f"ran {wk.drain()} job(s)")
else:
print(f"worker {wk.name} runs {', '.join(sorted(wk.can))}; Ctrl-C to stop")
stop = threading.Event()
notify.start(db, C, stop, name=wk.name, log=print) # sends notifications too (docs/notifications.md)
notify.start(db, C, stop, name=wk.name, log=say) # sends notifications too (docs/notifications.md)
if not (a.steps or a.no_schedule): # and scans watched folders and runs routines, as `lens watch` does
routines.start(db, C, stop, log=print)
sensors.start(db, C, stop, log=print, name=wk.name) # and the sensor hub, while sensors are on
smb_share.start(db, C, stop, name=wk.name, log=print) # and the SMB share of the file tree, while it's on
shares.start(db, C, stop, log=print, name=wk.name) # and the file shares (S3, WebDAV), while they're on
channels.start_all(db, C, stop, lambda: cfg, name=wk.name, log_fn=print) # and the chat channels, while they're on
tunnel.start(db, C, stop, name=wk.name, log=print) # and the Cloudflare tunnel, while it's on
https.start(db, C, stop, name=wk.name, log=print) # and HTTPS, where the ports are (LENS_HTTPS_LISTEN)
local_llm.start(db, C, stop, name=wk.name, log=print) # and the local chat model, while it's on
nfs_share.start(db, C, stop, name=wk.name, log=print) # and the file tree over NFS, while it's on
updates.start(db, C, stop, log=print) # and the daily check for a new release (docs/updates.md)
routines.start(db, C, stop, log=say)
sensors.start(db, C, stop, log=say, name=wk.name) # and the sensor hub, while sensors are on
smb_share.start(db, C, stop, name=wk.name, log=say) # and the SMB share of the file tree, while it's on
shares.start(db, C, stop, log=say, name=wk.name) # and the file shares (S3, WebDAV), while they're on
channels.start_all(db, C, stop, lambda: cfg, name=wk.name, log_fn=say) # and the chat channels, while they're on
tunnel.start(db, C, stop, name=wk.name, log=say) # and the Cloudflare tunnel, while it's on
https.start(db, C, stop, name=wk.name, log=say) # and HTTPS, where the ports are (LENS_HTTPS_LISTEN)
local_llm.start(db, C, stop, name=wk.name, log=say) # and the local chat model, while it's on
nfs_share.start(db, C, stop, name=wk.name, log=say) # and the file tree over NFS, while it's on
updates.start(db, C, stop, log=say) # and the daily check for a new release (docs/updates.md)
_stop_on_term()
try:
wk.loop(stop)
Expand All @@ -401,13 +404,15 @@ def password():
print(f"ran {routines.run_due(db, C(), print)} routine(s)")
else:
telemetry.set_role("watcher")
logs.setup("watcher")
say = logs.printer("lens.watch")
print("watching storage sources and running routines; Ctrl-C to stop")
_stop_on_term()
sensors.start(db, C, threading.Event(), log=print, name=f"watch-{os.getpid()}")
sensors.start(db, C, threading.Event(), log=say, name=f"watch-{os.getpid()}")
try:
while True:
sources.poll_due(db, C(), print)
routines.run_due(db, C(), print)
sources.poll_due(db, C(), say)
routines.run_due(db, C(), say)
time.sleep(max(5, min(C()["sources"]["check_seconds"], routines.CHECK_SECONDS)))
except KeyboardInterrupt:
pass
Expand Down
Loading
Loading