Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
metadata and a `.txt` of the text; everyone sees what the web app shows them. Browse and read it through the API
(`/api/v1/tree`) and the MCP tools `list_files` and `read_file` ([The graph as files](docs/file-tree.md)).
S3, WebDAV, SMB and NFS servers that share it are in progress.

- **HTTPS with automatic certificates.** Lens can serve itself at `https://` your domain with a certificate it gets
and renews on its own from Let's Encrypt or ZeroSSL (or any ACME authority), proving the domain over HTTP, TLS-ALPN
or DNS, with Cloudflare, Route 53 and the other caddy-dns providers, so wildcards and machines the internet can't
reach work too. Run the install line with `LENS_HTTPS=1` to open ports 443 and 80, then set it up with
`PUT /api/v1/settings/https`; the Settings page comes next. Off by default ([HTTPS](docs/https.md)).

- **Budget meter and Costs page.** The top bar shows what the namespace picked spent against its budget, used and
left, highlighted when near or over, and follows the namespace switcher; it opens the details for a day, week or
month. **Costs** in the left menu shows spend by day, model and resource, and sets the namespace's budget
Expand Down
18 changes: 18 additions & 0 deletions docker-compose.https.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# HTTPS with certificates Lens gets and renews itself (docs/https.md), set up in Settings › HTTPS.
# install.sh adds this file when LENS_HTTPS=1 is in .env (or the web app asked for it): the worker gets the HTTPS
# and HTTP ports, and runs Caddy there while HTTPS is on in Settings. By hand:
# docker compose -f docker-compose.prod.yml -f docker-compose.https.yml up -d
# Something else on this machine using 443 or 80? Set LENS_HTTPS_PORT / LENS_HTTP_PORT to other ports (with the DNS
# challenge, port 80 isn't needed at all).
services:
backend:
environment:
LENS_HTTPS_PORT: ${LENS_HTTPS_PORT:-443}
worker:
environment:
LENS_HTTPS_LISTEN: "1"
LENS_HTTPS_PORT: ${LENS_HTTPS_PORT:-443}
ports:
# Caddy listens on 8443 and 8080 in the container (it runs as an ordinary user, who can't use ports below 1024)
- "${LENS_HTTPS_BIND:-0.0.0.0}:${LENS_HTTPS_PORT:-443}:8443"
- "${LENS_HTTPS_BIND:-0.0.0.0}:${LENS_HTTP_PORT:-80}:8080"
2 changes: 2 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ this machine or the Docker host, with their models), `PUT /api/v1/setup/storage`
| `AUTH_TRUST_HOST` | `true` behind a proxy or in Docker |
| `TRUST_PROXY_HEADERS` | `true` when a reverse proxy in front of the web app sets `X-Forwarded-Host`, `-Proto` and `-For`: they're passed on to the API, which names that address in OAuth discovery and reads the visitor's address for IP groups and throttles. Off, the web app reports the `Host` the browser sent and the address it was reached from |
| `LENS_WEB_PROXY_HOSTS` | hosts whose `X-Forwarded-For` the web app passes on with `TRUST_PROXY_HEADERS` off (`backend,worker` in the Docker Compose files, for the Cloudflare tunnel) |
| `LENS_HTTPS`, `LENS_HTTPS_PORT`, `LENS_HTTP_PORT`, `LENS_HTTPS_BIND` | install.sh: `LENS_HTTPS=1` publishes the HTTPS and HTTP ports (443 and 80 by default) for [HTTPS](https.md) |
| `LENS_HTTPS_LISTEN` | `1` in the process that has the HTTPS ports and runs Caddy (the worker in `docker-compose.https.yml`); `LENS_CADDY`: a caddy to use instead of fetching one |

## archive.yaml

Expand Down
111 changes: 111 additions & 0 deletions docs/https.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# HTTPS with automatic certificates

Lens can serve itself at `https://lens.example.com` with a certificate it gets and renews on its own, from
[Let's Encrypt](https://letsencrypt.org) or [ZeroSSL](https://zerossl.com), with no reverse proxy to set up. It runs
[Caddy](https://caddyserver.com), which answers on the HTTPS port, gets a certificate for each domain over ACME,
renews it about a month before it runs out and passes visitors on to the web app. Passkeys work there, since browsers
allow them on https:// addresses.

Status: **built** in the API and the installer; the **Settings › HTTPS** page in the web app is in progress (until
then, `PUT /api/v1/settings/https` sets it). Off by default.

For a machine the internet can't reach at all, [Remote access](remote-access.md) (a Cloudflare tunnel) is the
other way to an https:// address.

## Turning it on

1. Point the domain's DNS at this machine (an A or AAAA record), unless you use the DNS challenge.
2. Give Lens the HTTPS ports: run the install line again with `LENS_HTTPS=1`. It publishes ports 443 and 80 to the
worker, where Caddy runs, and remembers it in `.env`, so upgrades keep them.

```sh
curl -fsSL https://raw.githubusercontent.com/adeelahmad/lense/main/install.sh | LENS_HTTPS=1 sh
```

Something else on the machine uses 443 or 80? `LENS_HTTPS_PORT` and `LENS_HTTP_PORT` choose others (Lens's
addresses then carry the port, like `https://lens.example.com:8443`). `LENS_HTTPS_BIND` limits them to one
address. Rootless Docker can't publish ports below 1024 unless `net.ipv4.ip_unprivileged_port_start` allows it.
3. In **Settings › HTTPS**: the domain, an email address for the certificate authority (optional; it warns you before
a certificate expires), and turn it on.

Settings › HTTPS shows whether a process has the ports, the certificates Lens holds and when each runs out, and
Caddy's last lines, so a domain that can't be verified says why.

## Proving the domain is yours

| Challenge | What needs to reach this machine | Wildcards |
| --- | --- | --- |
| **HTTP** (default) | Port 80 or 443 from the internet: the certificate authority visits `http://<domain>/.well-known/acme-challenge/` or the HTTPS port itself (TLS-ALPN) | No |
| **TLS-ALPN only** | Port 443 | No |
| **DNS** | Nothing: Lens adds a TXT record through your DNS provider's API | Yes, `*.example.com` |

The DNS challenge suits a machine on a home network or behind a firewall, as long as the address people use reaches
it (on the LAN, say). Pick the provider and give what it asks for:

| Provider | Fields |
| --- | --- |
| Cloudflare | an API token with Zone › Zone › Read and Zone › DNS › Edit |
| Amazon Route 53 | access key ID and secret of an IAM user allowed to list the hosted zones and change their records; region and hosted zone ID optional |
| DigitalOcean, Duck DNS, Gandi, deSEC, Netlify, Vultr, Linode, IONOS, Bunny | an API token |
| Porkbun, Namecheap, OVHcloud, DNSimple, netcup, Azure, Google Cloud DNS, PowerDNS, RFC 2136 (BIND, Knot) | the fields Settings lists |

Any other [caddy-dns](https://github.com/caddy-dns) plugin works too: its name as the provider (like `hetzner`), and
its fields as JSON. `dns_package` names a plugin outside caddy-dns (`github.com/<owner>/<repo>`).

## Certificate authorities

| `ca` | |
| --- | --- |
| `both` (default) | Let's Encrypt, and ZeroSSL when Let's Encrypt can't issue (an outage, a rate limit) |
| `letsencrypt` | Let's Encrypt only. `staging: true` uses its staging service while you try things out (browsers don't trust those certificates) |
| `zerossl` | ZeroSSL only. Lens gets ZeroSSL's account credentials with the email address; or give the EAB key ID and HMAC key from your ZeroSSL account |
| `custom` | any ACME directory (`ca_url`), like a company's own step-ca; EAB credentials when it needs them |

## Settings

All in Settings › HTTPS (the `https` section of `PUT /api/v1/settings/https`):

| Key | Default | |
| --- | --- | --- |
| `enabled` | `false` | |
| `domains` | `[]` | the names to serve, like `lens.example.com` |
| `email` | `""` | the ACME account's contact |
| `ca`, `ca_url`, `staging` | `both` | see above |
| `eab_key_id`, `eab_hmac_key` | | external account binding (the HMAC key is a secret) |
| `challenge` | `http` | `http`, `tls-alpn` or `dns` |
| `dns_provider`, `dns_package`, `dns_credentials` | | the DNS challenge's provider, plugin and fields (a JSON object; a secret) |
| `dns_resolvers`, `propagation` | | resolvers to check the TXT record with, and how long to wait for it (`2m`) |
| `redirect_http` | `true` | send http:// to https:// |
| `hsts` | `false` | tell browsers to use https:// only, for a year. Turn it on once HTTPS works: turning HTTPS off later leaves browsers unable to reach the http:// address |
| `origin` | | the web app, as the worker reaches it (`LENS_HTTPS_ORIGIN`, else `LENS_TUNNEL_ORIGIN`: `http://frontend:3000` in Docker) |
| `https_port`, `http_port` | `8443`, `8080` | where Caddy listens in the container (install.sh publishes 443 and 80 to them) |

## How it runs

- **Caddy** is fetched into `data_dir/bin` the first time it's needed: Caddy's release from GitHub, checked against
its published checksums, or, for a DNS provider, a build with that provider's plugin from Caddy's download service
(`caddyserver.com/api/download`). A `caddy` on `PATH`, or `LENS_CADDY`, is used instead when it has the plugin.
- **Where**: only a process with `LENS_HTTPS_LISTEN=1` runs it. install.sh's `docker-compose.https.yml` gives the
worker the ports and sets it; in a [microsandbox](microsandbox.md) it's the sandbox. Of those, one process holds a
lease and runs Caddy; another takes over within 30 seconds if it stops.
- **What it keeps**: the certificates and the ACME account in `data_dir/https/caddy`, in the archive volume, so a
rebuilt container doesn't ask for them again (certificate authorities limit how often). The configuration Lens
writes, `data_dir/https/caddy.json`, holds the DNS provider's credentials, so it's readable only by Lens (mode 600)
and passed by path, not on a command line.
- **Visitors**: Caddy passes the browser's `Host` on with `X-Forwarded-For` and `X-Forwarded-Proto: https`. The web
app trusts `X-Forwarded-For` from the worker (`LENS_WEB_PROXY_HOSTS`), so throttles and IP groups see the visitor.
Passkeys and sign-in follow the https:// address, and email links use the first domain unless
`notifications.app_url` says otherwise.
- **Changes**: a saved change restarts Caddy with the new configuration within a few seconds.

By hand, with Docker:

```sh
docker compose -f docker-compose.prod.yml -f docker-compose.https.yml up -d
```

## Refine later

- A button in Settings that opens the ports through the upgrade agent, so step 2 needs no terminal.
- The install wizard asking for the domain.
- Caddy in the image, for machines that can't download it.
3 changes: 3 additions & 0 deletions fastapi_backend/app/api/deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,7 @@ def web_origin(request: Request) -> str:
FRONTEND_URL. Called directly, the API's own address."""
from app.config import settings as env
from app.core.middleware import host_name
from app.domain import https

h = request.headers
host = (h.get("x-forwarded-host") or "").split(",")[0].strip()
Expand All @@ -400,6 +401,8 @@ def web_origin(request: Request) -> str:
shaped = re.fullmatch(r"[A-Za-z0-9.\-]+(:\d+)?|\[[0-9A-Fa-f:.]+\](:\d+)?", host)
if shaped and host.lower() in _tunnel_hosts(request, cfg):
return f"https://{host.lower()}" # Cloudflare serves it over https; cloudflared reaches the web app over http
if shaped and host_name(host) in https.hosts(cfg):
return f"https://{host.lower()}" # Caddy serves it over https (Settings › HTTPS), maybe at another port
if shaped and proto in ("http", "https") and (trusted or host_name(host) in _web_hosts(cfg)):
return f"{proto}://{host}"
return env.FRONTEND_URL.rstrip("/")
17 changes: 17 additions & 0 deletions fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
blobs,
bridge,
decide,
https,
jobs,
keyring,
llm,
Expand All @@ -44,6 +45,8 @@
FileStoreTestResult,
FileStoreTry,
Health,
HttpsProvider,
HttpsStatus,
IndexQueued,
LlmTestResult,
LocalLlmStatus,
Expand Down Expand Up @@ -178,6 +181,20 @@ def tunnel_status(user: AdminReader, cfg: Cfg, db: Db) -> TunnelStatus:
return TunnelStatus(**tunnel.status(db, cfg))


@router.get("/settings/https/status")
def https_status(user: AdminReader, cfg: Cfg, db: Db) -> HttpsStatus:
"""How HTTPS (Settings › HTTPS) is doing: whether a process has the HTTPS ports and runs Caddy, the certificates it
holds and when they run out, and Caddy's last lines."""
return HttpsStatus(**https.status(db, cfg))


@router.get("/settings/https/providers")
def https_providers(user: AdminReader) -> list[HttpsProvider]:
"""The DNS providers Settings › HTTPS lists for the DNS challenge, with the fields each needs. Any other caddy-dns
plugin works too, by name."""
return [HttpsProvider(**p) for p in https.providers()]


@router.post("/settings/bridge/test")
def test_bridge(user: AdminWriter, cfg: Cfg, db: Db) -> BridgeTestResult:
"""Check that Matterbridge answers at its address with its token, and that the account to answer as exists."""
Expand Down
18 changes: 17 additions & 1 deletion fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,22 @@ def platform_main(argv, config):
import threading
import time

from .domain import auth, bridge, jobs, local_llm, notify, passkeys, routines, sensors, settings, sources, telemetry, tunnel, updates
from .domain import (
auth,
bridge,
https,
jobs,
local_llm,
notify,
passkeys,
routines,
sensors,
settings,
sources,
telemetry,
tunnel,
updates,
)

ap = argparse.ArgumentParser(prog="lens")
sub = ap.add_subparsers(dest="cmd", required=True)
Expand Down Expand Up @@ -366,6 +381,7 @@ def password():
sensors.start(db, C, stop, log=print, name=wk.name) # and the sensor hub, while sensors are on
bridge.start(db, C, stop, lambda: cfg, name=wk.name, log_fn=print) # and the chat-room bridge, while it's on
tunnel.start(db, C, stop, name=wk.name, log=print) # and the Cloudflare tunnel, while it's on
https.start(db, C, stop, name=wk.name, log=print) # and HTTPS, where the ports are (LENS_HTTPS_LISTEN)
local_llm.start(db, C, stop, name=wk.name, log=print) # and the local chat model, while it's on
updates.start(db, C, stop, log=print) # and the daily check for a new release (docs/updates.md)
_stop_on_term()
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/core/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
bridge,
components,
content_types,
https,
jobs,
local_llm,
notify,
Expand Down Expand Up @@ -92,6 +93,7 @@ def start_background(self) -> None:
sensors.start(self.db, self.current, self.stop, log=log.info, name=f"api-{os.getpid()}")
bridge.start(self.db, self.current, self.stop, lambda: self.base, name=f"api-{os.getpid()}", log_fn=log.warning)
tunnel.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
https.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
local_llm.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
updates.start(self.db, self.current, self.stop, log=log.info)

Expand Down
Loading
Loading