From 312ddb09e142b5b1ef364ee8ab2697d4a3038bb3 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Thu, 15 Oct 2026 11:47:40 +0000 Subject: [PATCH 1/2] feat(sensors): reach the MQTT hub through remote access, over WebSockets at /mqtt The broker also listens for MQTT over WebSockets (sensors.mqtt_web, off by default; sensors.mqtt_web_port 9883) and the web app relays /mqtt to it, upgrades included, so the Cloudflare tunnel and HTTPS reach the hub. There every client signs in with a hub login (never anonymous) and failed logins are throttled per visitor, read from X-Forwarded-For as trusted proxies report it. Docs say why SMB, NFS and syslog stay on the local network. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01CXpr1EWCfUD5tb1SmSwfDa --- CHANGELOG.md | 6 + README.md | 6 +- cloudron/supervisord.conf | 3 +- docker-compose.prod.yml | 4 +- docker-compose.yml | 4 +- docs/configuration.md | 1 + docs/https.md | 6 +- docs/remote-access.md | 8 +- docs/requirements.md | 1 + docs/sensors.md | 24 +++- fastapi_backend/app/domain/ipgroups.py | 12 ++ fastapi_backend/app/domain/mqtt.py | 107 ++++++++++++-- fastapi_backend/app/domain/sensors.py | 54 ++++++- fastapi_backend/app/domain/settings.py | 5 +- fastapi_backend/app/domain/shares.py | 8 +- fastapi_backend/app/domain/store.py | 5 +- fastapi_backend/app/schemas/sensors.py | 1 + fastapi_backend/tests/api/test_sensors.py | 113 ++++++++++++++- nextjs-frontend/__tests__/lens-relay.test.ts | 132 ++++++++++++++++-- .../__tests__/sensors-model.test.ts | 15 ++ .../app/openapi-client/types.gen.ts | 6 + nextjs-frontend/components/sensors/parts.tsx | 5 +- .../components/sensors/sensor-detail.tsx | 2 + .../components/sensors/sensor-model.ts | 23 ++- .../components/settings/bodies.tsx | 4 + nextjs-frontend/components/settings/model.ts | 16 +++ nextjs-frontend/lens-relay.js | 112 +++++++++++---- nextjs-frontend/lens-server.js | 8 +- nextjs-frontend/openapi.json | 6 + nextjs-frontend/proxy.ts | 6 +- packaging/qnap/qpkg/shared/docker-compose.yml | 4 +- .../synology/package/compose/compose.yaml.in | 4 +- threat-model.json | 17 ++- threat-model.md | 8 +- threat-model.yaml | 25 +++- 35 files changed, 660 insertions(+), 101 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9623712e6..dd06487c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version `automation_actions`, `save_automation`, `test_automation` and `run_automation`, with problems in plain words. The assistant saves them switched off and asks before switching one on or running it ([Agents](docs/automations.md#agents)). +- **MQTT through Remote access.** Devices away from home can publish to the sensor hub, and subscribe to it, at the + web app's own address: MQTT over WebSockets at `/mqtt` (`wss://` under HTTPS), through the Cloudflare tunnel and + HTTPS too. It's off until an admin turns on **MQTT at the web app's address too** in Settings › Sensors, and there + every client signs in with a hub login, with too many wrong passwords from one address turned away for a while. It's + the same broker as on port 1883. SMB, NFS and syslog stay on the local network, and Remote access now says why + ([Sensors](docs/sensors.md#at-the-web-apps-address), [Remote access](docs/remote-access.md#what-it-reaches)). - **File shares through Remote access.** S3 and WebDAV can now be reached at the web app's own address, `/s3` and `/dav`, so they work through the Cloudflare tunnel and HTTPS as well as on the network, and on Cloudron, which publishes only the web app. It's off until an admin turns on **At the web app's address too** in Settings › File diff --git a/README.md b/README.md index 99d840496..b68c66d47 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p agents over MCP get the archive as data, and what Lens caught goes into the audit log. Serve Lens over HTTPS, set up in Settings, with certificates from Let's Encrypt or ZeroSSL that renew on their own, answering only for its own domains and, if you choose, only to devices holding a client certificate; or reach it from anywhere through a - Cloudflare tunnel with nothing opened on your router. ([Encryption](docs/encryption.md), [access](docs/access.md), + Cloudflare tunnel with nothing opened on your router. Either way, the S3 and WebDAV shares and the MQTT hub answer at + that same address once you turn them on. ([Encryption](docs/encryption.md), [access](docs/access.md), [prompt injection](docs/assistant.md#third-party-content-and-prompt-injection), [HTTPS](docs/https.md), [remote access](docs/remote-access.md); [checks](docs/requirements.md#stay-private-and-safe)) - **Use open standards.** RDF and Dublin Core for every record, OAuth for apps and OpenID Connect sign-in, MCP, MQTT, @@ -211,8 +212,7 @@ These are open pull requests, not yet part of Lens: - **Web clipper:** signing in, or using your API token, not only an import webhook's ([#406](https://github.com/adeelahmad/lense/pull/406)). - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)). -- **Shares:** S3 and WebDAV reached through remote access ([#402](https://github.com/adeelahmad/lense/pull/402)), - and files added over them imported ([#408](https://github.com/adeelahmad/lense/pull/408)). +- **Shares:** files added over S3 and WebDAV imported ([#408](https://github.com/adeelahmad/lense/pull/408)). ## Planned diff --git a/cloudron/supervisord.conf b/cloudron/supervisord.conf index 45665f8bd..239a0dfd5 100644 --- a/cloudron/supervisord.conf +++ b/cloudron/supervisord.conf @@ -54,7 +54,8 @@ redirect_stderr=true priority=40 user=cloudron directory=/app/code/frontend -; lens-relay.js: the file shares at the app's address, /s3 and /dav, once allowed in Settings › File shares +; lens-relay.js: the file shares at the app's address, /s3 and /dav, once allowed in Settings › File shares, and +; MQTT over WebSockets at /mqtt, once allowed in Settings › Sensors environment=NODE_OPTIONS="--require /app/code/frontend/lens-relay.js" command=/app/code/frontend/node_modules/.bin/next start --hostname 0.0.0.0 --port 3000 autorestart=true diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 227f34b16..7f80ae116 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -151,9 +151,11 @@ services: AUTH_TRUST_HOST: "true" # the containers running the Cloudflare tunnel: their X-Forwarded-For names the visitor (lens-server.js) LENS_WEB_PROXY_HOSTS: backend,worker - # the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares + # the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares, + # and MQTT over WebSockets at /mqtt, once allowed in Settings › Sensors LENS_S3_URL: http://worker:9555 LENS_WEBDAV_URL: http://worker:9556 + LENS_MQTT_WS_URL: http://worker:9883 # no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel) ports: # this machine only by default; LENS_BIND=0.0.0.0 to reach it from the network (install.sh does on a server) diff --git a/docker-compose.yml b/docker-compose.yml index 36b933e5c..a21d264c3 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -118,10 +118,12 @@ services: AUTH_TRUST_HOST: "true" # no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel) OPENAPI_OUTPUT_FILE: ./shared-data/openapi.json - # the file shares at the web app's address, /s3 and /dav, once allowed in Settings › File shares (lens-relay.js) + # the file shares at the web app's address, /s3 and /dav, once allowed in Settings › File shares, and MQTT over + # WebSockets at /mqtt, once allowed in Settings › Sensors (lens-relay.js) NODE_OPTIONS: --require /app/lens-relay.js LENS_S3_URL: http://worker:9555 LENS_WEBDAV_URL: http://worker:9556 + LENS_MQTT_WS_URL: http://worker:9883 volumes: - ./nextjs-frontend:/app - /app/node_modules # the image's, renewed with it (see the backend's /opt/venv) diff --git a/docs/configuration.md b/docs/configuration.md index cc778fe21..c83a44aa7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -91,6 +91,7 @@ whether the wizard is still pending (`wizard_pending`). | `TRUST_PROXY_HEADERS` | `true` when a reverse proxy in front of the web app sets `X-Forwarded-Host`, `-Proto` and `-For`: they're passed on to the API, which names that address in OAuth discovery and reads the visitor's address for IP groups and throttles. Off, the web app reports the `Host` the browser sent and the address it was reached from | | `LENS_WEB_PROXY_HOSTS` | hosts whose `X-Forwarded-For` the web app passes on with `TRUST_PROXY_HEADERS` off (`backend,worker` in the Docker Compose files, for the Cloudflare tunnel) | | `LENS_S3_URL`, `LENS_WEBDAV_URL` | where the web app finds the file shares it relays at `/s3` and `/dav` once Settings › File shares allows it (`http://worker:9555` and `http://worker:9556` in the Docker Compose files; unset, this machine's ports 9555 and 9556). See [The graph as files](file-tree.md#at-the-web-apps-address) | +| `LENS_MQTT_WS_URL` | where the web app finds the sensor hub's MQTT over WebSockets, which it relays at `/mqtt` once Settings › Sensors allows it (`http://worker:9883` in the Docker Compose files; unset, this machine's port 9883). See [Sensors](sensors.md#at-the-web-apps-address) | | `LENS_HTTPS`, `LENS_HTTPS_PORT`, `LENS_HTTP_PORT`, `LENS_HTTPS_BIND` | install.sh: `LENS_HTTPS=1` publishes the HTTPS and HTTP ports (443 and 80 by default) for [HTTPS](https.md) | | `LENS_HTTPS_LISTEN` | `1` in the process that has the HTTPS ports and runs Caddy (the worker in `docker-compose.https.yml`); `LENS_CADDY`: a caddy to use instead of fetching one | diff --git a/docs/https.md b/docs/https.md index 3bc22a2bf..e0010ecc8 100644 --- a/docs/https.md +++ b/docs/https.md @@ -114,8 +114,10 @@ All in Settings › HTTPS (`PUT /api/v1/settings/https`; `POST /api/v1/settings/ `notifications.app_url` says otherwise. - **Changes**: a saved change restarts Caddy with the new configuration within a few seconds. - **What it serves**: the web app, and what the web app serves at its own address: the API, the MCP server at `/mcp`, - and the file shares at `/s3` and `/dav` once Settings › File shares allows them there - ([The graph as files](file-tree.md#at-the-web-apps-address)). MQTT, syslog, SMB and NFS keep their own ports. + the file shares at `/s3` and `/dav` once Settings › File shares allows them there + ([The graph as files](file-tree.md#at-the-web-apps-address)), and MQTT over WebSockets at `/mqtt` once Settings › + Sensors does ([Sensors](sensors.md#at-the-web-apps-address)). MQTT on port 1883, syslog, SMB and NFS keep their own + ports. By hand, with Docker: diff --git a/docs/remote-access.md b/docs/remote-access.md index 1a8d47bb9..928c2f2d1 100644 --- a/docs/remote-access.md +++ b/docs/remote-access.md @@ -54,7 +54,13 @@ The tunnel serves the web app and what the web app serves at its own address: | The MCP server (`/mcp`, with OAuth) | Yes ([MCP server](mcp.md)) | | Sensors that push over HTTP (`/api/v1/sensors/push/…`) | Yes ([Sensors](sensors.md)) | | The file shares: S3 at `/s3` and WebDAV at `/dav` | Once an admin turns on **At the web app's address too** in Settings › File shares ([The graph as files](file-tree.md#at-the-web-apps-address)) | -| MQTT, syslog, SMB and NFS | No: they aren't HTTP, so they stay on their own ports, for the local network | +| The sensor hub's MQTT broker, over WebSockets at `/mqtt` | Once an admin turns on **MQTT at the web app's address too** in Settings › Sensors ([Sensors](sensors.md#at-the-web-apps-address)) | +| MQTT on port 1883, syslog, SMB and NFS | No: they aren't HTTP, so they stay on their own ports, for the local network | + +SMB and NFS stay on the local network: neither travels over HTTP, and both are made for a LAN (NFS has no sign-in at +all). To open the graph's files from elsewhere, use WebDAV at `/dav`, which macOS, Windows and Linux mount as a drive, +or S3 at `/s3`. Syslog has no sign-in either, so it stays local too; devices elsewhere send readings to a +[webhook](sensors.md#ways-in) or publish over MQTT at `/mqtt`. ## Where it sends visitors diff --git a/docs/requirements.md b/docs/requirements.md index 6fd4395cb..7350acac2 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -216,6 +216,7 @@ request, to another check that proves the same thing; loosening a requirement or | Chat-room answers have the links they built broken up; agents over MCP get the archive as data | [test_prompt_injection.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_prompt_injection.py) `test_a_room_gets_the_links_the_answer_built_broken_up`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_what_the_archive_says_reaches_agents_as_data` | | HTTPS set up in Settings, with certificates that renew, only for its own domains and optionally only for client certificates | [test_https.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_https.py) `test_https_is_set_up_in_the_app`, `test_caddys_configuration_follows_the_settings`, `test_strict_sni_client_certificates_and_tls_13` | | Remote access through a Cloudflare tunnel | [test_tunnel.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_tunnel.py) `test_remote_access_is_set_up_in_the_app`, `test_a_quick_tunnel_runs_in_one_process_and_stops_when_turned_off`, `test_a_managed_tunnel_is_made_routed_and_given_a_dns_record` | +| The S3 and WebDAV shares and the MQTT hub at the web app's address (`/s3`, `/dav`, `/mqtt` over WebSockets), each off until turned on | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_sensors.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sensors.py) `test_broker_over_websockets`, `test_mqtt_over_websockets_takes_hub_logins_only`; [lens-relay.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/lens-relay.test.ts) `relays any method, path and body to the share, and its answer back`, `passes the handshake on with the visitor and the prefix, then joins the two connections` | | The threat model's three files agree | [threat-model.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/threat-model.yml) `validate`, `Validate the threat model`; [validate_threat_model.py](https://github.com/adeelahmad/lense/blob/main/ci/validate_threat_model.py) | ## Use open standards diff --git a/docs/sensors.md b/docs/sensors.md index 809c4a41c..d22dc2d8b 100644 --- a/docs/sensors.md +++ b/docs/sensors.md @@ -48,6 +48,27 @@ network reach them. Run natively, it runs in `lens worker` (and in the API when it runs background work). Several processes can run it: the first to get the port serves it, and the others try again every minute. +## At the web app's address + +Devices away from home (a phone, a sensor at another site) can reach the hub wherever the web app opens, through +[Remote access](remote-access.md) or [HTTPS](https.md): MQTT over WebSockets at `/mqtt`. An admin turns it on with +**MQTT at the web app's address too** in Settings → Sensors (`sensors.mqtt_web`, off by default, and only with MQTT on). + +- The address is the web app's with `wss://` and `/mqtt` (`ws://` on plain http), such as + `wss://lens.example.com/mqtt`. The sensors page shows it next to the MQTT port once it's on. Any client with a + WebSocket transport takes it, for example paho-mqtt (`transport="websockets"`, path `/mqtt`) or mqtt.js in a browser. +- Every client signs in with a hub login. Clients without one are turned away here even when `sensors.mqtt_anonymous` + lets them in on port 1883, and eight wrong passwords from one address in 15 minutes turn that address away for a + while, as sign-in does. +- It is the same broker: what a client publishes over WebSockets reaches subscribers on port 1883 and becomes a sensor + as it would there (in the login's namespace), and the other way round. + +How it works: the worker listens for MQTT over WebSockets on `sensors.mqtt_web_port` (9883, on `sensors.bind` like +1883), and the web app passes `/mqtt` on to it (`nextjs-frontend/lens-relay.js`, at `LENS_MQTT_WS_URL`: +`http://worker:9883` in the Docker Compose files, else this machine's port 9883). The broker reads each visitor's address +from the web app's `X-Forwarded-For`, which it trusts from the hosts in `LENS_TRUSTED_PROXY_HOSTS` and this machine. In +Docker, port 9883 isn't published: devices reach it through the web app. + ## Organising itself A device publishing to the hub, or a host sending syslog, becomes a sensor the first time it does, marked **new**. Its @@ -137,6 +158,7 @@ folders feeding their namespaces): ## Refine later -Notifications for alert patterns, sensors for namespace members (admins only for now), MQTT 5, the port mappings of +Notifications for alert patterns, sensors for namespace members (admins only for now), MQTT 5, per-topic access for +hub logins (any login can subscribe to `#`, at `/mqtt` too), the port mappings of the Synology, QNAP, Proxmox and Cloudron packages; in the web app, a chart with axes and zoom, paging back through readings, and labelling many log patterns at once. diff --git a/fastapi_backend/app/domain/ipgroups.py b/fastapi_backend/app/domain/ipgroups.py index bcac945a0..5f2a4d277 100644 --- a/fastapi_backend/app/domain/ipgroups.py +++ b/fastapi_backend/app/domain/ipgroups.py @@ -152,6 +152,18 @@ def client_address(peer, port, forwarded, trusted): return None +LOOPBACK = ("127.0.0.0/8", "::1/128") + + +def visitor(cfg, peer, port, forwarded): + """The address a connection to one of Lens's own servers (the file shares, MQTT over WebSockets) comes from, for + throttles: what the web app relaying it (and proxies it trusts: server.trusted_proxies, LENS_TRUSTED_PROXY_HOSTS, + this machine) report in X-Forwarded-For, else the connection's peer.""" + trusted = tuple((cfg.get("server") or {}).get("trusted_proxies") or ()) + proxy_hosts() + LOOPBACK + addr = client_address(peer, port or 1, forwarded or "", trusted) + return str(addr) if addr else peer + + # ---------- what an address opens ---------- class Network(NamedTuple): """What the visitor's address opens: whole namespaces and chosen recordings, each with the name of the IP group diff --git a/fastapi_backend/app/domain/mqtt.py b/fastapi_backend/app/domain/mqtt.py index 63dcbab18..ea9f97ecf 100644 --- a/fastapi_backend/app/domain/mqtt.py +++ b/fastapi_backend/app/domain/mqtt.py @@ -4,15 +4,17 @@ to Mosquitto, and anything can subscribe to it: it is an ordinary local broker for MQTT 3.1 and 3.1.1. Every message published to it is also handed to Lens (sensors.py), which is what makes it a hub rather than only a broker. -What it does: CONNECT with a username and password (checked by the caller), PUBLISH at QoS 0, 1 and 2 (acknowledged -as the level asks), SUBSCRIBE and UNSUBSCRIBE with + and # wildcards, retained messages, last wills, keep-alive and -PING. What it doesn't (refine later): MQTT 5 (a client asking for it is told the protocol isn't supported, and most -fall back to 3.1.1), persistent sessions (every session is clean), and delivery above QoS 0 to subscribers. +What it does: CONNECT with a username and password (checked by the caller), PUBLISH at QoS 0, 1 and 2 (acknowledged as +the level asks), SUBSCRIBE and UNSUBSCRIBE with + and # wildcards, retained messages, last wills, keep-alive and PING, +over TCP and over WebSockets (`WebSockets`, which the web app relays at /mqtt). What it doesn't (refine later): MQTT 5 +(a client asking for it is told the protocol isn't supported, and most fall back to 3.1.1), persistent sessions (every +session is clean), and delivery above QoS 0 to subscribers. """ from __future__ import annotations import contextlib +import logging import secrets import socket import socketserver @@ -26,6 +28,9 @@ # CONNACK return codes ACCEPTED, BAD_PROTOCOL, BAD_CLIENT_ID, BAD_LOGIN, NOT_AUTHORIZED = 0, 1, 2, 4, 5 MAX_RETAINED = 10_000 +# what the WebSocket server says: every connection opening and closing is too much to keep (logs.levels can lower it) +WS_LOG = logging.getLogger("lens.mqtt.websockets") +WS_LOG.setLevel(logging.WARNING) class ProtocolError(Exception): @@ -224,7 +229,7 @@ def publish(self, topic, payload, retain=False, origin=None): for s in targets: s.send(data) - def _connect(self, f, sock, address): + def _connect(self, f, sock, address, authenticate): kind, _flags, body = read_packet(f, self.max_bytes) if kind != CONNECT: raise ProtocolError("the first packet must be CONNECT") @@ -258,7 +263,7 @@ def _connect(self, f, sock, address): sock.sendall(packet(CONNACK, 0, bytes([0, BAD_CLIENT_ID]))) raise ProtocolError("an empty client id needs a clean session") client_id = "lens-" + secrets.token_hex(6) - login = self.authenticate(user, pw, client_id, address) + login = authenticate(user, pw, client_id, address) if not login: sock.sendall(packet(CONNACK, 0, bytes([0, BAD_LOGIN if user else NOT_AUTHORIZED]))) raise ProtocolError(f"refused {user or 'an anonymous client'} from {address[0]}") @@ -273,12 +278,12 @@ def _connect(self, f, sock, address): session.send(packet(CONNACK, 0, bytes([0, ACCEPTED]))) return session, keepalive - def _serve(self, sock, address): + def _serve(self, sock, address, authenticate=None): sock.settimeout(10) f = sock.makefile("rb") session = None try: - session, keepalive = self._connect(f, sock, address) + session, keepalive = self._connect(f, sock, address, authenticate or self.authenticate) sock.settimeout(keepalive * 1.5 if keepalive else None) while True: kind, flags, body = read_packet(f, self.max_bytes) @@ -365,6 +370,92 @@ def _subscribe(self, session, body): session.send(publish_packet(topic, payload, retain=True)) +# ---------- over WebSockets ---------- +class _WebSocket: + """A WebSocket made to look like the socket the broker reads and writes. MQTT packets ride in binary messages, as + many to a message, or split across messages, as the client likes (MQTT 3.1.1, section 6).""" + + def __init__(self, ws): + self.ws, self.buf, self.timeout = ws, bytearray(), None + + def makefile(self, _mode="rb"): + return self + + def read(self, n): + from websockets.exceptions import ConnectionClosed + + while len(self.buf) < n: + try: + msg = self.ws.recv(timeout=self.timeout) # TimeoutError (socket.timeout) when the client went quiet + except ConnectionClosed: + break + if isinstance(msg, str): + raise ProtocolError("MQTT over WebSockets comes in binary messages, not text") + self.buf += msg + data = bytes(self.buf[:n]) + del self.buf[:n] + return data + + def sendall(self, data): + from websockets.exceptions import ConnectionClosed + + try: + self.ws.send(data) + except ConnectionClosed as e: + raise OSError("the WebSocket closed") from e + + def settimeout(self, seconds): + self.timeout = seconds + + def shutdown(self, _how): + pass + + def close(self): + self.ws.close() + + +def _subprotocol(_conn, offered): + """MQTT clients ask for "mqtt" (3.1.1) or "mqttv3.1"; one asking for neither is taken too.""" + return next((p for p in ("mqtt", "mqttv3.1") if p in offered), None) + + +class WebSockets: + """The broker over WebSockets, for MQTT clients in a browser and for those reaching Lens through the web app, which + relays /mqtt here (nextjs-frontend/lens-relay.js). Clients share the broker's topics, retained messages and + subscribers with those on TCP. `authenticate(username, password, client_id, address)` is checked in place of the + broker's; `address_of(headers, peer)` says where a connection comes from (the visitor behind the web app).""" + + def __init__(self, broker, host, port, authenticate, address_of=None): + from websockets.sync.server import serve + + def handler(ws): + peer = ws.remote_address[:2] + address = address_of(ws.request.headers, peer) if address_of else peer + broker._serve(_WebSocket(ws), address, authenticate) + + self.server = serve( + handler, + host, + port, + select_subprotocol=_subprotocol, + compression=None, + max_size=broker.max_bytes + 1024, # a message holds a packet or a few: the broker's own limit stands + server_header=None, + logger=WS_LOG, + family=socket.AF_INET6 if ":" in host else socket.AF_INET, + ) + self.port = self.server.socket.getsockname()[1] + self.thread = None + + def start(self): + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True, name=f"mqtt-ws-{self.port}") + self.thread.start() + return self + + def stop(self): + self.server.shutdown() + + # ---------- a client, for bridges to another broker ---------- class Client: """A minimal MQTT 3.1.1 subscriber: connect, subscribe to filters and hand each message to `on_message(topic, diff --git a/fastapi_backend/app/domain/sensors.py b/fastapi_backend/app/domain/sensors.py index 6920d84e7..ec4ecd966 100644 --- a/fastapi_backend/app/domain/sensors.py +++ b/fastapi_backend/app/domain/sensors.py @@ -41,7 +41,7 @@ import threading import time -from . import auth, mqtt, settings, sources, store, syslog +from . import auth, ipgroups, mqtt, settings, sources, store, syslog R = store.R STREAM_TYPES = { @@ -1037,9 +1037,9 @@ def __init__(self, db, cfg_fn, log=None, name=None): self.db, self.cfg_fn, self.log = db, cfg_fn, log or (lambda *a: None) self.name = name or f"{socket.gethostname()}-{id(self) % 10000}" self.intake = intake(db, cfg_fn, self.log) - self.broker = self.listener = None - self.state = {"mqtt": {}, "syslog": {}} - self.retry = {"mqtt": 0.0, "syslog": 0.0} + self.broker = self.listener = self.web = None + self.state = {"mqtt": {}, "syslog": {}, "mqtt_web": {}} + self.retry = {"mqtt": 0.0, "syslog": 0.0, "mqtt_web": 0.0} self.bridges: dict = {} # sensor id -> {sig, stop, thread} def _on_mqtt(self, topic, payload, login, address): @@ -1094,8 +1094,50 @@ def apply(self): self.retry[what] = time.monotonic() + RETRY_SECONDS if self.listener is not None and want["syslog"]: self.listener.networks = list(want["syslog"][2]) + web = want["mqtt"] and s.get("mqtt_web") and self.broker is not None + self._web((bind, int(s.get("mqtt_web_port", 9883))) if web else None) self._bridges(on) + # ---------- MQTT over WebSockets ---------- + def _web(self, want): + """MQTT over WebSockets beside the broker (sensors.mqtt_web), which the web app relays at /mqtt.""" + have = self.state["mqtt_web"].get("want") + if want == have and (self.web is not None or want is None or time.monotonic() < self.retry["mqtt_web"]): + return # as it should be, or it failed and waits to try again + self._stop_web() + if want is None: + return + try: + self.web = mqtt.WebSockets(self.broker, want[0], want[1], self._web_login, self._visitor).start() + self.state["mqtt_web"] = {"want": want, "running": True, "port": self.web.port, "since": store.now()} + self.log(f"sensors: MQTT over WebSockets listening on port {self.web.port}") + except OSError as e: + self.state["mqtt_web"] = {"want": want, "running": False, "error": f"port {want[1]}: {e.strerror or e}"} + self.retry["mqtt_web"] = time.monotonic() + RETRY_SECONDS + + def _web_login(self, username, password, _client_id, address): + """A client over WebSockets always signs in with a hub login (anonymous clients stay on the local TCP port), and + is turned away for a while after too many wrong ones from its address, as sign-in is.""" + key = f"mqtt-web:{address[0]}" + if not username or auth.throttled(key): + return None + login = authenticate(self.db, self.cfg_fn(), username, password) + if not login: + auth.hit(key) + return login + + def _visitor(self, headers, peer): + return ipgroups.visitor(self.cfg_fn(), peer[0], peer[1], headers.get("x-forwarded-for", "")), peer[1] + + def _stop_web(self): + if self.web is not None: + try: + self.web.stop() + except Exception: # noqa: BLE001 + pass + self.web = None + self.state["mqtt_web"] = {} + # ---------- bridges to other brokers ---------- def _bridges(self, on): """Run a connection for every bridge sensor that isn't paused, one process each (the first to claim it).""" @@ -1180,6 +1222,8 @@ def on_connect(): wait = min(wait * 2, BRIDGE_WAIT[1]) def _stop(self, what): + if what == "mqtt": + self._stop_web() # it serves this broker thing = self.broker if what == "mqtt" else self.listener if thing is not None: try: @@ -1204,6 +1248,7 @@ def heartbeat(self): "at": store.now(), "mqtt": {k: v for k, v in self.state["mqtt"].items() if k != "want"}, "syslog": {k: v for k, v in self.state["syslog"].items() if k != "want"}, + "mqtt_web": {k: v for k, v in self.state["mqtt_web"].items() if k != "want"}, "bridges": sorted(self.bridges), } if self.broker is not None: @@ -1253,6 +1298,7 @@ def hub_status(db, cfg): "enabled": bool(s.get("enabled")), "mqtt": bool(s.get("mqtt")), "mqtt_port": s.get("mqtt_port"), + "mqtt_web": bool(s.get("mqtt_web")), "syslog": bool(s.get("syslog")), "syslog_port": s.get("syslog_port"), "processes": rows, diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index 8b3d580c4..92e07120b 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -74,6 +74,8 @@ "mqtt", "mqtt_port", "mqtt_anonymous", + "mqtt_web", + "mqtt_web_port", "syslog", "syslog_port", "syslog_networks", @@ -1186,6 +1188,7 @@ def _notify_setting(key, value): SENSOR_RANGES = { "mqtt_port": (1, 65535), + "mqtt_web_port": (1, 65535), "syslog_port": (1, 65535), "max_payload_kb": (1, 16384), "raw_days": (1, 36500), @@ -1196,7 +1199,7 @@ def _notify_setting(key, value): def _sensor_setting(key, value): - if key in ("enabled", "mqtt", "mqtt_anonymous", "syslog", "triage"): + if key in ("enabled", "mqtt", "mqtt_anonymous", "mqtt_web", "syslog", "triage"): if not isinstance(value, bool): raise ValueError(f"sensors.{key} is true or false") return value diff --git a/fastapi_backend/app/domain/shares.py b/fastapi_backend/app/domain/shares.py index 6320bf3fc..e7b9e056e 100644 --- a/fastapi_backend/app/domain/shares.py +++ b/fastapi_backend/app/domain/shares.py @@ -33,7 +33,6 @@ # protocol -> the path the web app relays it at (nextjs-frontend/lens-relay.js) WEB_PATHS = {"s3": "/s3", "webdav": "/dav"} -LOOPBACK = ("127.0.0.0/8", "::1/128") def _row(proto): @@ -86,12 +85,9 @@ def remote(cfg): def visitor(handler, cfg): - """The address a request comes from, for throttles: what the web app (and proxies it trusts: server.trusted_proxies, - LENS_TRUSTED_PROXY_HOSTS, this machine) report in X-Forwarded-For, else the connection's peer.""" + """The address a request comes from, for throttles (ipgroups.visitor).""" peer, port = handler.client_address[0], handler.client_address[1] - trusted = tuple((cfg.get("server") or {}).get("trusted_proxies") or ()) + ipgroups.proxy_hosts() + LOOPBACK - addr = ipgroups.client_address(peer, port or 1, handler.headers.get("x-forwarded-for", ""), trusted) - return str(addr) if addr else peer + return ipgroups.visitor(cfg, peer, port, handler.headers.get("x-forwarded-for", "")) def wanted(cfg): diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 0f9fdc8f6..82e1a92a6 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -559,13 +559,16 @@ def labels(): # sensors (sensors.py, docs/sensors.md): off until an admin turns them on. Then the MQTT hub and the syslog listener # run in the process that runs routines (`lens worker`), on these ports; syslog is taken only from syslog_networks. # store, raw_days, rollup_days, important_days and max_per_minute are what a stream sensor gets unless it has its own. - # bind is a startup setting only. + # bind is a startup setting only. mqtt_web: the broker over WebSockets too, on mqtt_web_port, which the web app + # relays at /mqtt (so Remote access and HTTPS reach it), for clients with a hub login only. "sensors": { "enabled": False, "bind": "0.0.0.0", "mqtt": True, "mqtt_port": 1883, "mqtt_anonymous": False, + "mqtt_web": False, + "mqtt_web_port": 9883, "syslog": True, "syslog_port": 5514, "syslog_networks": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8", "fc00::/7", "::1/128"], diff --git a/fastapi_backend/app/schemas/sensors.py b/fastapi_backend/app/schemas/sensors.py index 17079aedf..556ada8fa 100644 --- a/fastapi_backend/app/schemas/sensors.py +++ b/fastapi_backend/app/schemas/sensors.py @@ -96,6 +96,7 @@ class HubStatus(ResponseModel): enabled: bool mqtt: bool mqtt_port: int | None = None + mqtt_web: bool = Field(False, description="the broker over WebSockets too, which the web app relays at /mqtt") syslog: bool syslog_port: int | None = None processes: list[dict[str, Any]] = Field(description="the processes running the hub now, and what they listen on") diff --git a/fastapi_backend/tests/api/test_sensors.py b/fastapi_backend/tests/api/test_sensors.py index 9cebe1d2a..b770a6ad2 100644 --- a/fastapi_backend/tests/api/test_sensors.py +++ b/fastapi_backend/tests/api/test_sensors.py @@ -11,7 +11,7 @@ import pytest -from app.domain import mqtt, routines, sensors, sources, store, syslog +from app.domain import auth, mqtt, routines, sensors, sources, store, syslog from tests.helpers import login, make_user R = store.R @@ -29,8 +29,18 @@ def _cfg_fn(cfg): # ---------- a tiny MQTT client for the tests ---------- class Device: - def __init__(self, port, client_id="dev", user=None, password=None, level=4): - self.sock = socket.create_connection(("127.0.0.1", port), timeout=5) + """`web`: over WebSockets (with these request headers, when a dict), as a browser or the web app's /mqtt connects.""" + + def __init__(self, port, client_id="dev", user=None, password=None, level=4, web=None): + if web is None: + self.sock = socket.create_connection(("127.0.0.1", port), timeout=5) + else: + from websockets.sync.client import connect + + ws = connect(f"ws://127.0.0.1:{port}/mqtt", subprotocols=["mqtt"], additional_headers=web or None, open_timeout=5) + assert ws.subprotocol == "mqtt" + self.sock = mqtt._WebSocket(ws) + self.sock.settimeout(5) self.f = self.sock.makefile("rb") flags = 0x02 | (0x80 if user else 0) | (0x40 if password else 0) body = mqtt.string("MQTT") + bytes([level, flags]) + struct.pack("!H", 30) + mqtt.string(client_id) @@ -226,6 +236,55 @@ def test_bridge_client_reads_another_broker(): b.stop() +def test_broker_over_websockets(): + from websockets.sync.client import connect + + got, seen = [], [] + b = mqtt.Broker("127.0.0.1", 0, lambda *a: {"tcp": 1}, lambda t, p, lg, a: got.append((t, p, lg, a[0]))).start() + + def where(headers, peer): + seen.append(headers.get("x-forwarded-for")) + return ("198.51.100.7", peer[1]) + + w = mqtt.WebSockets(b, "127.0.0.1", 0, lambda u, p, cid, a: {"web": u} if (u, p) == ("dev", "secret12") else None, where).start() + try: + assert Device(w.port, "x", "dev", "wrong", web={}).code == mqtt.BAD_LOGIN + assert Device(w.port, "y", web={}).code == mqtt.NOT_AUTHORIZED # its own check, not the broker's + sub = Device(b.port, "sub", "u", "p") # on TCP + assert sub.subscribe("home/#") == [0] + pub = Device(w.port, "pub", "dev", "secret12", web={"X-Forwarded-For": "203.0.113.5"}) + pub.publish("home/door", "open", qos=1, retain=True) + assert sub.receive() == ("home/door", b"open", False) + _wait(lambda: got) + assert got == [("home/door", b"open", {"web": "dev"}, "198.51.100.7")] + assert "203.0.113.5" in seen + web_sub = Device(w.port, "wsub", "dev", "secret12", web={}) + assert web_sub.subscribe("home/#") == [0] + assert web_sub.receive() == ("home/door", b"open", True) # retained, across transports + Device(b.port, "tcp-pub", "u", "p").publish("home/hall", "21") + assert web_sub.receive() == ("home/hall", b"21", False) + assert _wait(lambda: b.clients() == 3) # tcp-pub's goodbye reaches the broker a moment later + + # packets split across messages, and two in one message, as clients may send them + raw = connect(f"ws://127.0.0.1:{w.port}/mqtt", open_timeout=5) # asking for no subprotocol is fine too + body = mqtt.string("MQTT") + bytes([4, 0xC2]) + struct.pack("!H", 30) + mqtt.string("raw") + body += mqtt.string("dev") + mqtt.string("secret12") + connect_packet = mqtt.packet(mqtt.CONNECT, 0, body) + raw.send(connect_packet[:3]) + raw.send(connect_packet[3:]) + assert raw.recv(timeout=5) == mqtt.packet(mqtt.CONNACK, 0, bytes([0, mqtt.ACCEPTED])) + raw.send(mqtt.publish_packet("home/a", b"1") + mqtt.publish_packet("home/b", b"2")) + assert web_sub.receive() == ("home/a", b"1", False) + assert web_sub.receive() == ("home/b", b"2", False) + raw.send("text frames aren't MQTT") # closed for it + assert _wait(lambda: b.clients() == 3) + for d in (sub, pub, web_sub): + d.close() + finally: + w.stop() + b.stop() + + # ---------- the hub ---------- @pytest.fixture def hub(db, cfg): @@ -302,6 +361,48 @@ def test_hub_logins_and_anonymous(client, admin, db, cfg, hub): assert Device(port, "e", "router", "long-enough").code == mqtt.BAD_LOGIN +def test_mqtt_over_websockets_takes_hub_logins_only(client, admin, db, cfg, hub): + ns = store.ns_id(db, "pods") + sensors.create_login(db, "phone", "phone-pass", space=ns) + assert hub.web is None and hub.state["mqtt_web"] == {} # off by default + cfg["sensors"].update(mqtt_web=True, mqtt_web_port=0, mqtt_anonymous=True) + hub.apply() + assert hub.state["mqtt_web"]["running"] + port = hub.state["mqtt_web"]["port"] + away = {"X-Forwarded-For": "203.0.113.9"} # through the web app, which runs on this machine here + assert Device(port, "a", web=away).code == mqtt.NOT_AUTHORIZED # anonymous clients stay on the TCP port + assert Device(hub.state["mqtt"]["port"], "a").code == 0 + d = Device(port, "phone-1", "phone", "phone-pass", web=away) + assert d.code == 0 + d.publish("owntracks/phone/me", '{"lat": 51.5, "lon": -0.12, "batt": 80}') + d.close() + _wait(lambda: hub.intake.rows) + hub.intake.flush() + row = sensors._by_key(db, "mqtt:owntracks/phone") + assert row and row["space"] == ns # the login's namespace, as on TCP + + # too many wrong passwords from one visitor turn them away for a while; others are still let in + try: + for _ in range(8): + assert Device(port, "x", "phone", "wrong-pass", web=away).code == mqtt.BAD_LOGIN + assert Device(port, "x", "phone", "phone-pass", web=away).code == mqtt.BAD_LOGIN + assert Device(port, "y", "phone", "phone-pass", web={"X-Forwarded-For": "203.0.113.10"}).code == 0 + finally: + auth._FAILS.pop("mqtt-web:203.0.113.9", None) + + hub.heartbeat() + assert sensors.hub_status(db, cfg)["processes"][0]["mqtt_web"]["running"] + cfg["sensors"]["mqtt"] = False # it serves the broker: off with it + hub.apply() + assert hub.web is None and hub.broker is None + cfg["sensors"]["mqtt"] = True + hub.apply() + assert hub.state["mqtt_web"]["running"] + cfg["sensors"]["mqtt_web"] = False + hub.apply() + assert hub.web is None and hub.broker is not None + + def test_syslog_senders_become_sensors(client, admin, db, cfg, hub): port = hub.state["syslog"]["port"] s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) @@ -515,6 +616,12 @@ def test_sensor_settings(client, admin): assert put({"store": "maybe"}).status_code == 400 assert put({"syslog_networks": ["not a network"]}).status_code == 400 assert put({"bind": "0.0.0.0"}).status_code == 400 + assert put({"mqtt_web": True, "mqtt_web_port": 9884}).status_code == 200 + got = client.get("/api/v1/settings", headers=admin).json()["sensors"]["values"] + assert got["mqtt_web"] is True and got["mqtt_web_port"] == 9884 + assert client.get("/api/v1/sensors", headers=admin).json()["hub"]["mqtt_web"] is True + assert put({"mqtt_web": "yes"}).status_code == 400 + assert put({"mqtt_web_port": 0}).status_code == 400 # ---------- log patterns and triage ---------- diff --git a/nextjs-frontend/__tests__/lens-relay.test.ts b/nextjs-frontend/__tests__/lens-relay.test.ts index 6e8212934..2e1bff6a8 100644 --- a/nextjs-frontend/__tests__/lens-relay.test.ts +++ b/nextjs-frontend/__tests__/lens-relay.test.ts @@ -1,9 +1,9 @@ /** @jest-environment node */ import http from "node:http"; -import type { AddressInfo } from "node:net"; +import net, { type AddressInfo } from "node:net"; // eslint-disable-next-line @typescript-eslint/no-require-imports -const { forwardHeaders, route, shareOf, targetOf } = require("../lens-relay.js"); +const { forwardHeaders, route, serverOf, targetOf, upgrade } = require("../lens-relay.js"); type Seen = { method?: string; url?: string; headers: http.IncomingHttpHeaders; body: string }; @@ -23,27 +23,39 @@ function call(port: number, method: string, path: string, headers: Record { - it("finds the share a path is for", () => { - expect(shareOf("/s3")?.name).toBe("S3"); - expect(shareOf("/s3/pods/a%20b.txt?X-Amz-Signature=1")?.name).toBe("S3"); - expect(shareOf("/s3?list-type=2")?.name).toBe("S3"); - expect(shareOf("/dav/")?.name).toBe("WebDAV"); - expect(shareOf("/dav/pods/Collections/")?.name).toBe("WebDAV"); +describe("serverOf", () => { + it("finds the server a path is for", () => { + expect(serverOf("/s3")?.name).toBe("S3 share"); + expect(serverOf("/s3/pods/a%20b.txt?X-Amz-Signature=1")?.name).toBe("S3 share"); + expect(serverOf("/s3?list-type=2")?.name).toBe("S3 share"); + expect(serverOf("/dav/")?.name).toBe("WebDAV share"); + expect(serverOf("/dav/pods/Collections/")?.name).toBe("WebDAV share"); + expect(serverOf("/mqtt")?.name).toBe("MQTT over WebSockets"); }); it("leaves the web app's own paths to Next", () => { - for (const p of ["/", "/s3x", "/s/abc", "/davis", "/api/v1/shares/s3", "/settings?section=dav", "http://x/s3/"]) - expect(shareOf(p)).toBeNull(); + for (const p of [ + "/", + "/s3x", + "/s/abc", + "/davis", + "/mqttx", + "/api/v1/shares/s3", + "/settings?section=dav", + "http://x/s3/", + "/_next/webpack-hmr", + ]) + expect(serverOf(p)).toBeNull(); }); }); describe("targetOf", () => { it("reads where the server is from the environment, else this machine", () => { - const s3 = shareOf("/s3"); + const s3 = serverOf("/s3"); expect(targetOf(s3, {})).toEqual({ host: "127.0.0.1", port: 9555 }); expect(targetOf(s3, { LENS_S3_URL: "http://worker:9555" })).toEqual({ host: "worker", port: 9555 }); - expect(targetOf(shareOf("/dav"), { LENS_WEBDAV_URL: "http://[::1]:7000" })).toEqual({ host: "::1", port: 7000 }); + expect(targetOf(serverOf("/dav"), { LENS_WEBDAV_URL: "http://[::1]:7000" })).toEqual({ host: "::1", port: 7000 }); + expect(targetOf(serverOf("/mqtt"), {})).toEqual({ host: "127.0.0.1", port: 9883 }); }); }); @@ -57,7 +69,7 @@ describe("forwardHeaders", () => { expect: "100-continue", }, "::ffff:203.0.113.5", - shareOf("/s3"), + serverOf("/s3"), ); expect(out).toEqual({ host: "lens.example.com", @@ -66,7 +78,7 @@ describe("forwardHeaders", () => { "x-forwarded-prefix": "/s3", }); expect( - forwardHeaders({ "x-forwarded-for": "198.51.100.1" }, "172.18.0.4", shareOf("/dav"))["x-forwarded-for"], + forwardHeaders({ "x-forwarded-for": "198.51.100.1" }, "172.18.0.4", serverOf("/dav"))["x-forwarded-for"], ).toBe("198.51.100.1, 172.18.0.4"); }); }); @@ -132,9 +144,99 @@ describe("route", () => { try { const r = await call(frontPort, "GET", "/s3/"); expect(r.status).toBe(503); + expect(r.body).toContain("S3 share isn't running here"); expect(r.body).toContain("Settings, File shares"); } finally { delete process.env.LENS_S3_URL; } }); }); + +/** Sends a WebSocket handshake for `path` and gives back what came first, and the socket to go on with. */ +function handshake(port: number, path: string): Promise<{ first: string; socket: net.Socket }> { + return new Promise((resolve, reject) => { + const socket = net.connect({ host: "127.0.0.1", port }, () => + socket.write( + `GET ${path} HTTP/1.1\r\nHost: lens.example.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n` + + "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Protocol: mqtt\r\n\r\n", + ), + ); + socket.once("data", (d) => resolve({ first: d.toString("latin1"), socket })); + socket.on("error", reject); + }); +} + +describe("upgrade", () => { + const seen: { head?: string } = {}; + // stands in for the broker's WebSocket server: answers the handshake, then echoes what it's sent + const broker = net.createServer((s) => { + s.once("data", (d) => { + seen.head = d.toString("latin1"); + s.write("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\r\n"); + s.pipe(s); + }); + }); + const front = http.createServer(); + front.on("upgrade", (req, socket, head) => { + if (!upgrade(req, socket, head)) socket.end("HTTP/1.1 404 Not Found\r\ncontent-length: 4\r\n\r\nnext"); + }); + let frontPort = 0; + + beforeAll(async () => { + const port = await new Promise((resolve) => + broker.listen(0, "127.0.0.1", () => resolve((broker.address() as AddressInfo).port)), + ); + process.env.LENS_MQTT_WS_URL = `http://127.0.0.1:${port}`; + frontPort = await listen(front); + }); + afterAll(() => { + delete process.env.LENS_MQTT_WS_URL; + broker.close(); + front.close(); + }); + + it("passes the handshake on with the visitor and the prefix, then joins the two connections", async () => { + const { first, socket } = await handshake(frontPort, "/mqtt"); + expect(first).toMatch(/^HTTP\/1\.1 101/); + const head = seen.head || ""; + expect(head).toMatch(/^GET \/mqtt HTTP\/1\.1\r\n/); + expect(head).toContain("host: lens.example.com\r\n"); + expect(head).toContain("connection: Upgrade\r\n"); + expect(head).toContain("upgrade: websocket\r\n"); + expect(head).toContain("sec-websocket-protocol: mqtt\r\n"); + expect(head).toContain("x-forwarded-for: 127.0.0.1\r\n"); + expect(head).toContain("x-forwarded-prefix: /mqtt\r\n"); + const frame = Buffer.from([0x82, 0x02, 0x10, 0x00]); // a binary frame, as MQTT rides in + const echoed = new Promise((resolve) => socket.once("data", resolve)); + socket.write(frame); + try { + expect(await echoed).toEqual(frame); + } finally { + socket.destroy(); + } + }); + + it("leaves other upgrades to Next", async () => { + const { first, socket } = await handshake(frontPort, "/_next/webpack-hmr"); + expect(first).toContain("next"); + socket.destroy(); + }); + + it("says so when the broker isn't listening", async () => { + const closed = net.createServer(); + const port = await new Promise((resolve) => + closed.listen(0, "127.0.0.1", () => resolve((closed.address() as AddressInfo).port)), + ); + await new Promise((resolve) => closed.close(resolve)); + const was = process.env.LENS_MQTT_WS_URL; + process.env.LENS_MQTT_WS_URL = `http://127.0.0.1:${port}`; + try { + const { first, socket } = await handshake(frontPort, "/mqtt"); + expect(first).toMatch(/^HTTP\/1\.1 503/); + expect(first).toContain("Settings, Sensors"); + socket.destroy(); + } finally { + process.env.LENS_MQTT_WS_URL = was; + } + }); +}); diff --git a/nextjs-frontend/__tests__/sensors-model.test.ts b/nextjs-frontend/__tests__/sensors-model.test.ts index 7c600b8e2..f3352b47a 100644 --- a/nextjs-frontend/__tests__/sensors-model.test.ts +++ b/nextjs-frontend/__tests__/sensors-model.test.ts @@ -7,6 +7,7 @@ import { handlingText, hubText, lastText, + mqttWebUrl, parseDays, pushUrl, sparkPath, @@ -62,6 +63,10 @@ describe("sensor model", () => { it("tells devices where to send", () => { expect(connectText("mqtt", "lens.local", { mqtt_port: 1883 })).toBe("mqtt://lens.local:1883"); + expect(connectText("mqtt", "lens.example.com", { mqtt_web: true }, null, "https://lens.example.com")).toBe( + "mqtt://lens.example.com:1883 or wss://lens.example.com/mqtt", + ); + expect(mqttWebUrl("http://lens.local:3000/")).toBe("ws://lens.local:3000/mqtt"); expect(connectText("syslog", "lens.local", {})).toBe("lens.local:5514 (UDP or TCP)"); expect(pushUrl("https://lens.example/", "abc")).toBe("https://lens.example/api/v1/sensors/push/abc"); expect(pushUrl("lens.example", "abc")).toBe("https://lens.example/api/v1/sensors/push/abc"); @@ -99,5 +104,15 @@ describe("sensor model", () => { expect( hubText({ ...base, enabled: true, processes: [{ mqtt: { running: false, error: "port 1883: in use" } }] }), ).toEqual({ tone: "error", title: "The hub couldn’t start everything.", body: "MQTT: port 1883: in use" }); + expect( + hubText({ + ...base, + enabled: true, + processes: [{ mqtt: { running: true, port: 1883 }, mqtt_web: { running: true, port: 9883 } }], + }).body, + ).toBe("MQTT on port 1883 · MQTT over WebSockets on port 9883 (/mqtt)"); + expect( + hubText({ ...base, enabled: true, processes: [{ syslog: { running: false, error: "port 5514: in use" } }] }).body, + ).toBe("Syslog: port 5514: in use"); }); }); diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 3a9e31a83..ceeb93a6a 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -9067,6 +9067,12 @@ export type HubStatus = { * Mqtt Port */ mqtt_port?: number | null; + /** + * Mqtt Web + * + * the broker over WebSockets too, which the web app relays at /mqtt + */ + mqtt_web?: boolean; /** * Syslog */ diff --git a/nextjs-frontend/components/sensors/parts.tsx b/nextjs-frontend/components/sensors/parts.tsx index b99e12e8a..8d3782f95 100644 --- a/nextjs-frontend/components/sensors/parts.tsx +++ b/nextjs-frontend/components/sensors/parts.tsx @@ -503,7 +503,10 @@ export function AddSensorDialog({ {types[type]?.help &&

Use this for {types[type].help}.

} {type !== "webhook" && type !== "bridge" && ( - Point it at {connectText(type, host, hub)} + Point it at{" "} + + {connectText(type, host, hub, null, typeof window === "undefined" ? undefined : window.location.origin)} + {!hub.enabled && ". The hub is off: turn it on in Settings → Sensors"}. )} diff --git a/nextjs-frontend/components/sensors/sensor-detail.tsx b/nextjs-frontend/components/sensors/sensor-detail.tsx index 162fedf98..3858d7f6a 100644 --- a/nextjs-frontend/components/sensors/sensor-detail.tsx +++ b/nextjs-frontend/components/sensors/sensor-detail.tsx @@ -474,6 +474,8 @@ function SettingsTab({ s }: { s: Detail }) { s.type as StreamType, typeof window === "undefined" ? "lens" : window.location.hostname, hub ?? {}, + null, + typeof window === "undefined" ? undefined : window.location.origin, )} . diff --git a/nextjs-frontend/components/sensors/sensor-model.ts b/nextjs-frontend/components/sensors/sensor-model.ts index 8d2991001..44ff0ea59 100644 --- a/nextjs-frontend/components/sensors/sensor-model.ts +++ b/nextjs-frontend/components/sensors/sensor-model.ts @@ -111,16 +111,24 @@ export function parseDays(text: string): number | null | "bad" { return n <= 36500 ? n : "bad"; } -/** Where a device sends to, for the add dialog and a sensor's page. */ +/** The broker over WebSockets at the web app's address (`origin`, the page's): wss:// under https. */ +export function mqttWebUrl(origin: string): string { + return `${origin.replace(/^http/, "ws").replace(/\/+$/, "")}/mqtt`; +} + +/** Where a device sends to, for the add dialog and a sensor's page. `origin`: the page's, for MQTT over WebSockets. */ export function connectText( type: StreamType, host: string, - hub: { mqtt_port?: number | null; syslog_port?: number | null }, + hub: { mqtt_port?: number | null; syslog_port?: number | null; mqtt_web?: boolean }, token?: string | null, + origin?: string, ): string { switch (type) { - case "mqtt": - return `mqtt://${host}:${hub.mqtt_port ?? 1883}`; + case "mqtt": { + const tcp = `mqtt://${host}:${hub.mqtt_port ?? 1883}`; + return hub.mqtt_web && origin ? `${tcp} or ${mqttWebUrl(origin)}` : tcp; + } case "syslog": return `${host}:${hub.syslog_port ?? 5514} (UDP or TCP)`; case "webhook": @@ -223,10 +231,11 @@ export function hubText(hub: { const errors: string[] = []; const listening: string[] = []; for (const p of hub.processes) { - for (const what of ["mqtt", "syslog"] as const) { + for (const what of ["mqtt", "mqtt_web", "syslog"] as const) { const st = p[what] as { running?: boolean; port?: number; error?: string } | undefined; - if (st?.running) listening.push(`${what === "mqtt" ? "MQTT" : "syslog"} on port ${st.port}`); - else if (st?.error) errors.push(`${what === "mqtt" ? "MQTT" : "Syslog"}: ${st.error}`); + const name = { mqtt: "MQTT", mqtt_web: "MQTT over WebSockets", syslog: "syslog" }[what]; + if (st?.running) listening.push(`${name} on port ${st.port}${what === "mqtt_web" ? " (/mqtt)" : ""}`); + else if (st?.error) errors.push(`${name[0].toUpperCase()}${name.slice(1)}: ${st.error}`); } } if (!hub.processes.length) diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index 2898d6be7..8d3de32d4 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -1444,6 +1444,10 @@ function SensorsBody({ ctx }: { ctx: BodyCtx }) { +
+ + +
Syslog
diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 4a2254ffc..e3bb136a4 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -2655,6 +2655,22 @@ export const FIELDS: FieldSpec[] = [ kind: "switch", hint: "Off: devices sign in with a hub login (Sensors → Hub logins)", }, + { + section: "sensors", + key: "mqtt_web", + label: "MQTT at the web app’s address too", + kind: "switch", + hint: "Over WebSockets at /mqtt wherever the web app opens, Remote access and HTTPS included, for devices with a hub login. Off: only on the MQTT port", + }, + { + section: "sensors", + key: "mqtt_web_port", + label: "MQTT over WebSockets port", + kind: "int", + min: 1, + max: 65535, + hint: "Where the web app passes /mqtt on to (LENS_MQTT_WS_URL)", + }, { section: "sensors", key: "syslog", label: "Syslog", kind: "switch" }, { section: "sensors", diff --git a/nextjs-frontend/lens-relay.js b/nextjs-frontend/lens-relay.js index 522fae06f..d321acdcc 100644 --- a/nextjs-frontend/lens-relay.js +++ b/nextjs-frontend/lens-relay.js @@ -1,23 +1,42 @@ /** * Relays servers Lens runs besides the API at the web app's own address, so their clients reach them wherever the web - * app is reached, Settings › Remote access and Settings › HTTPS included. Today, the file shares - * (docs/file-tree.md#at-the-web-apps-address): /s3 to the S3 server and /dav to the WebDAV server, both run by the - * worker (fastapi_backend/app/domain/shares.py). + * app is reached, Settings › Remote access and Settings › HTTPS included. All three run in the worker: * - * Requests go to the share servers before Next sees them: Next answers only the usual methods (WebDAV lists folders - * with PROPFIND) and redirects paths that end in a slash. Each keeps its path and Host (S3 clients sign both) and - * carries X-Forwarded-Prefix, which tells the share server it came this way: it turns these away until an admin turns - * on "At the web app's address" in Settings › File shares (shares.remote). The servers are at LENS_S3_URL and - * LENS_WEBDAV_URL (http://worker:9555 and http://worker:9556 in the Docker Compose files), else this machine's ports. + * - /s3 to the S3 server and /dav to the WebDAV server, the file shares (docs/file-tree.md#at-the-web-apps-address, + * fastapi_backend/app/domain/shares.py); + * - /mqtt to the sensor hub's MQTT broker over WebSockets (docs/sensors.md#at-the-web-apps-address, + * fastapi_backend/app/domain/mqtt.py). + * + * Requests go to these servers before Next sees them: Next answers only the usual methods (WebDAV lists folders with + * PROPFIND) and redirects paths that end in a slash. Each keeps its path and Host (S3 clients sign both) and carries + * X-Forwarded-For and X-Forwarded-Prefix. The share servers turn requests that came this way away until an admin turns + * on "At the web app's address" in Settings › File shares (shares.remote); the broker listens on WebSockets only once + * an admin turns on "MQTT at the web app's address" in Settings › Sensors (sensors.mqtt_web). The servers are at + * LENS_S3_URL, LENS_WEBDAV_URL and LENS_MQTT_WS_URL (http://worker:9555, :9556 and :9883 in the Docker Compose files), + * else this machine's ports. * * lens-server.js uses it; `node --require ./lens-relay.js` (in NODE_OPTIONS) adds it to `next dev` and `next start`. */ /* eslint-disable @typescript-eslint/no-require-imports */ const http = require("node:http"); +const net = require("node:net"); -const SHARES = [ - { prefix: "/s3", env: "LENS_S3_URL", fallback: "http://127.0.0.1:9555", name: "S3" }, - { prefix: "/dav", env: "LENS_WEBDAV_URL", fallback: "http://127.0.0.1:9556", name: "WebDAV" }, +const SERVERS = [ + { prefix: "/s3", env: "LENS_S3_URL", fallback: "http://127.0.0.1:9555", name: "S3 share", where: "File shares" }, + { + prefix: "/dav", + env: "LENS_WEBDAV_URL", + fallback: "http://127.0.0.1:9556", + name: "WebDAV share", + where: "File shares", + }, + { + prefix: "/mqtt", + env: "LENS_MQTT_WS_URL", + fallback: "http://127.0.0.1:9883", + name: "MQTT over WebSockets", + where: "Sensors", + }, ]; // Hop-by-hop headers stay with each connection; Expect was answered already (Node sends 100 Continue itself). const HOP = new Set([ @@ -32,34 +51,37 @@ const HOP = new Set([ "expect", ]); -/** The share a request target is for ("/dav/pods/" is WebDAV's), or null. */ -function shareOf(url) { +/** The server a request target is for ("/dav/pods/" is WebDAV's), or null. */ +function serverOf(url) { const path = (url || "").split("?")[0]; - return SHARES.find((s) => path === s.prefix || path.startsWith(`${s.prefix}/`)) || null; + return SERVERS.find((s) => path === s.prefix || path.startsWith(`${s.prefix}/`)) || null; } -/** Where a share's server is: its host and port. */ -function targetOf(share, env = process.env) { - const u = new URL(env[share.env] || share.fallback); +/** Where a server is: its host and port. */ +function targetOf(server, env = process.env) { + const u = new URL(env[server.env] || server.fallback); return { host: u.hostname.replace(/^\[(.*)\]$/, "$1"), port: Number(u.port) || 80 }; } /** The headers sent on: the request's own (Host too), without hop-by-hop ones, with X-Forwarded-For and -Prefix. */ -function forwardHeaders(headers, peer, share) { +function forwardHeaders(headers, peer, server) { const out = {}; for (const [k, v] of Object.entries(headers)) if (!HOP.has(k)) out[k] = v; const from = (peer || "").replace(/^::ffff:/, ""); out["x-forwarded-for"] = out["x-forwarded-for"] ? `${out["x-forwarded-for"]}, ${from}` : from; - out["x-forwarded-prefix"] = share.prefix; + out["x-forwarded-prefix"] = server.prefix; return out; } -/** Answers a request for a share by relaying it; false (nothing sent) when the request isn't for one. */ +const offText = (server) => + `Lens's ${server.name} isn't running here: an admin turns it on in Settings, ${server.where}.\n`; + +/** Answers a request for one of the servers by relaying it; false (nothing sent) when the request isn't for one. */ function route(req, res) { - const share = shareOf(req.url); - if (!share) return false; - const { host, port } = targetOf(share); - const headers = forwardHeaders(req.headers, req.socket && req.socket.remoteAddress, share); + const server = serverOf(req.url); + if (!server) return false; + const { host, port } = targetOf(server); + const headers = forwardHeaders(req.headers, req.socket && req.socket.remoteAddress, server); const up = http.request({ host, port, method: req.method, path: req.url, headers }, (r) => { const out = {}; for (const [k, v] of Object.entries(r.headers)) if (!HOP.has(k)) out[k] = v; @@ -69,7 +91,7 @@ function route(req, res) { up.on("error", () => { if (res.headersSent) return res.destroy(); res.writeHead(503, { "content-type": "text/plain; charset=utf-8", "retry-after": "30" }); - res.end(`Lens's ${share.name} share isn't running here: an admin turns it on in Settings, File shares.\n`); + res.end(offText(server)); }); res.on("close", () => { if (!res.writableFinished) up.destroy(); @@ -78,15 +100,53 @@ function route(req, res) { return true; } +/** + * Answers a WebSocket (an HTTP upgrade) for one of the servers by joining the two connections once the request is + * passed on; false when it isn't for one (Next's own, such as its development reloads, go on to Next). + */ +function upgrade(req, socket, head) { + const server = serverOf(req.url); + if (!server) return false; + const { host, port } = targetOf(server); + const headers = forwardHeaders(req.headers, req.socket && req.socket.remoteAddress, server); + headers.connection = "Upgrade"; + headers.upgrade = req.headers.upgrade || "websocket"; + const lines = [`${req.method} ${req.url} HTTP/1.1`]; + for (const [k, v] of Object.entries(headers)) for (const one of [].concat(v)) lines.push(`${k}: ${one}`); + let joined = false; + const up = net.connect({ host, port }, () => { + joined = true; + up.write(`${lines.join("\r\n")}\r\n\r\n`); + if (head && head.length) up.write(head); + up.pipe(socket); + socket.pipe(up); + }); + up.on("error", () => { + if (joined) return socket.destroy(); + const text = offText(server); + socket.end( + "HTTP/1.1 503 Service Unavailable\r\ncontent-type: text/plain; charset=utf-8\r\nretry-after: 30\r\n" + + `content-length: ${Buffer.byteLength(text)}\r\nconnection: close\r\n\r\n${text}`, + ); + }); + up.on("close", () => { + if (joined) socket.destroy(); + }); + socket.on("error", () => up.destroy()); + socket.on("close", () => up.destroy()); + return true; +} + /** Relays these requests on every HTTP server this process starts (for `node --require ./lens-relay.js`). */ function install() { const emit = http.Server.prototype.emit; http.Server.prototype.emit = function (event, req, res, ...rest) { if (event === "request" && route(req, res)) return true; + if (event === "upgrade" && upgrade(req, res, rest[0])) return true; return emit.call(this, event, req, res, ...rest); }; } if (module.parent && module.parent.id === "internal/preload") install(); -module.exports = { shareOf, targetOf, forwardHeaders, route, install }; +module.exports = { serverOf, targetOf, forwardHeaders, route, upgrade, install }; diff --git a/nextjs-frontend/lens-server.js b/nextjs-frontend/lens-server.js index c47cb25b1..3ceb5fa7d 100644 --- a/nextjs-frontend/lens-server.js +++ b/nextjs-frontend/lens-server.js @@ -7,7 +7,8 @@ * known to set it: any peer with TRUST_PROXY_HEADERS=true (a reverse proxy in front), or one of the hosts named in * LENS_WEB_PROXY_HOSTS (the containers running the Cloudflare tunnel), looked up every 30 seconds. * - * Requests for the file shares (/s3 and /dav) then go to the share servers rather than to Next (lens-relay.js). + * Requests for the file shares (/s3 and /dav) and the MQTT broker (/mqtt, over WebSockets) then go to those servers + * rather than to Next (lens-relay.js). */ /* eslint-disable @typescript-eslint/no-require-imports */ const http = require("node:http"); @@ -48,9 +49,10 @@ function start() { } const emit = http.Server.prototype.emit; http.Server.prototype.emit = function (event, req, res, ...rest) { - if (event === "request") { + if (event === "request" || event === "upgrade") { scrubForwardedFor(req.headers, req.socket && req.socket.remoteAddress, trustAll, proxies); - if (relay.route(req, res)) return true; + if (event === "request" && relay.route(req, res)) return true; + if (event === "upgrade" && relay.upgrade(req, res, rest[0])) return true; } return emit.call(this, event, req, res, ...rest); }; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index ceb1c37bf..d08481e1d 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -46404,6 +46404,12 @@ ], "title": "Mqtt Port" }, + "mqtt_web": { + "type": "boolean", + "title": "Mqtt Web", + "description": "the broker over WebSockets too, which the web app relays at /mqtt", + "default": false + }, "syslog": { "type": "boolean", "title": "Syslog" diff --git a/nextjs-frontend/proxy.ts b/nextjs-frontend/proxy.ts index 659021c60..528d617db 100644 --- a/nextjs-frontend/proxy.ts +++ b/nextjs-frontend/proxy.ts @@ -33,11 +33,11 @@ export default auth((req) => { export const config = { // Skip Auth.js, the backend paths (proxied by route handlers, see lib/api/backend-proxy.ts; /.well-known is OAuth // discovery, /mcp the MCP server, /metrics the Prometheus scrape endpoint, /id and /ns the - // archive's linked-data URIs and vocabulary), the file shares (/s3 and /dav, relayed before Next by lens-relay.js), - // Next's files and assets. + // archive's linked-data URIs and vocabulary), the file shares and MQTT over WebSockets (/s3, /dav and /mqtt, relayed + // before Next by lens-relay.js), Next's files and assets. // Under /iiif only the backend's IIIF resources are skipped (collection, discovery, auth, /iiif//…); the app's own // IIIF pages (/iiif, /iiif/collections/…, /iiif/import, /iiif/metadata/…) need the session like any other page. matcher: [ - "/((?!api/|\\.well-known/|mcp(?:/|$)|s3(?:/|$)|dav(?:/|$)|metrics$|_next/|embed/|s/|id/|ns$|iiif/(?:collection(?!s)|discovery|auth|\\d)|reports/|static/|fonts/|favicon\\.ico|icon\\.svg|apple-icon\\.png|manifest\\.webmanifest|safari-pinned-tab\\.svg|pwa/|robots\\.txt).*)", + "/((?!api/|\\.well-known/|mcp(?:/|$)|s3(?:/|$)|dav(?:/|$)|mqtt(?:/|$)|metrics$|_next/|embed/|s/|id/|ns$|iiif/(?:collection(?!s)|discovery|auth|\\d)|reports/|static/|fonts/|favicon\\.ico|icon\\.svg|apple-icon\\.png|manifest\\.webmanifest|safari-pinned-tab\\.svg|pwa/|robots\\.txt).*)", ], }; diff --git a/packaging/qnap/qpkg/shared/docker-compose.yml b/packaging/qnap/qpkg/shared/docker-compose.yml index 04921de30..c92534a72 100644 --- a/packaging/qnap/qpkg/shared/docker-compose.yml +++ b/packaging/qnap/qpkg/shared/docker-compose.yml @@ -84,9 +84,11 @@ services: AUTH_TRUST_HOST: "true" # the containers running the Cloudflare tunnel: their X-Forwarded-For names the visitor (lens-server.js) LENS_WEB_PROXY_HOSTS: backend,worker - # the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares + # the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares, + # and MQTT over WebSockets at /mqtt, once allowed in Settings › Sensors LENS_S3_URL: http://worker:9555 LENS_WEBDAV_URL: http://worker:9556 + LENS_MQTT_WS_URL: http://worker:9883 # no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel) ports: - "${LENS_PORT:-3000}:3000" diff --git a/packaging/synology/package/compose/compose.yaml.in b/packaging/synology/package/compose/compose.yaml.in index a36e00970..4f197c2b5 100644 --- a/packaging/synology/package/compose/compose.yaml.in +++ b/packaging/synology/package/compose/compose.yaml.in @@ -94,9 +94,11 @@ services: AUTH_TRUST_HOST: "true" # the containers running the Cloudflare tunnel: their X-Forwarded-For names the visitor (lens-server.js) LENS_WEB_PROXY_HOSTS: backend,worker - # the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares + # the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares, + # and MQTT over WebSockets at /mqtt, once allowed in Settings › Sensors LENS_S3_URL: http://worker:9555 LENS_WEBDAV_URL: http://worker:9556 + LENS_MQTT_WS_URL: http://worker:9883 # no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel) ports: - "@PORT@:3000" diff --git a/threat-model.json b/threat-model.json index bd128d9d7..3152f5261 100644 --- a/threat-model.json +++ b/threat-model.json @@ -555,6 +555,15 @@ "provenance": "documented", "source": "docs/smb.md \"What keeps it safe\"" }, + { + "entry_point": "web app /mqtt and port 9883 (MQTT over WebSockets, relayed by lens-relay.js ahead of Next)", + "parameter": "the WebSocket handshake, MQTT packets in binary messages, hub logins", + "attacker_controllable": "yes", + "component": "optional", + "caller_must_enforce": "admin enables sensors.mqtt_web; a hub login on every connection, never anonymous; failed logins throttled per visitor from X-Forwarded-For as trusted proxies report it", + "provenance": "documented", + "source": "docs/sensors.md \"At the web app's address\"" + }, { "entry_point": "web app /s3 and /dav (file shares relayed by lens-relay.js, ahead of Next)", "parameter": "any method, path, Host and body, passed to the share servers with X-Forwarded-Prefix and X-Forwarded-For", @@ -644,7 +653,7 @@ "Expiry uses wall-clock time with no skew handling.", "Several API processes and workers may run at once; throttle counters and the setup code live in each process's memory.", "Child processes (rclone, ffmpeg, LibreOffice, Chromium, Tesseract, poppler, cloudflared, llama-server, code tools, uv/pip) run as argument lists.", - "Listening sockets: API and web app; a loopback netguard proxy during captures; MQTT 1883 and syslog 5514 when sensors are on; llama-server when enabled.", + "Listening sockets: API and web app; a loopback netguard proxy during captures; MQTT 1883 and syslog 5514 when sensors are on, MQTT over WebSockets on 9883 with sensors.mqtt_web; llama-server when enabled.", "Files live under data_dir, including secret.key when ARCHIVE_SECRET_KEY is unset; the server log carries the setup code and, without mail, sign-in links." ], "does_not": [ @@ -793,8 +802,8 @@ "source": "docs/configuration.md \"runs with its sandbox where it can\"" }, { - "name": "sensors.enabled, sensors.mqtt_anonymous", - "default": "off (bind 0.0.0.0); off", + "name": "sensors.enabled, sensors.mqtt_anonymous, sensors.mqtt_web", + "default": "off (bind 0.0.0.0); off; off", "effect": "narrows P-SENSORS", "discouraged": false, "provenance": "documented", @@ -1209,7 +1218,7 @@ "property": "P-SENSORS", "violation_symptom": "integrity-bypass", "severity_tier": "security", - "conditions": "sensors are off until an admin enables them; MQTT needs a hub login unless mqtt_anonymous is on; syslog hears only listed networks", + "conditions": "sensors are off until an admin enables them; MQTT needs a hub login unless mqtt_anonymous is on, and always over WebSockets (/mqtt, sensors.mqtt_web); syslog hears only listed networks", "provenance": "documented", "source": "docs/sensors.md \"Sensors are off until an admin turns them on\"" }, diff --git a/threat-model.md b/threat-model.md index cb11b37c0..bd8303972 100644 --- a/threat-model.md +++ b/threat-model.md @@ -210,7 +210,7 @@ install, no personal content) are off by default and there is no update check *( | Optional ML extras | some download weights on first use on their own (Hugging Face, ModelScope) | *(inferred, Q19)* | | Env proxies | plain `urllib` callers honour `HTTP(S)_PROXY`; the guarded openers ignore it | *(assumption, Q19)* | | Child processes | rclone, ffmpeg/ffprobe, LibreOffice, Chromium, Tesseract, poppler tools, antiword/catdoc, cloudflared, llama-server, `unshare` + Python for code tools, uv/pip; all as argument lists | *(assumption, Q20)* | -| Listening sockets | API and web app; a `netguard` proxy on loopback during captures; MQTT 1883 and syslog 5514 on `sensors.bind` (default `0.0.0.0`) when sensors are on; the file shares' S3 server on 9555 and WebDAV server on 9556 (`shares.bind`) when file shares are on; llama-server on its port (0.0.0.0 in containers, with its own API key) | *(assumption, Q19)* | +| Listening sockets | API and web app; a `netguard` proxy on loopback during captures; MQTT 1883 and syslog 5514 on `sensors.bind` (default `0.0.0.0`) when sensors are on, and MQTT over WebSockets on 9883 when `sensors.mqtt_web` is on too; the file shares' S3 server on 9555 and WebDAV server on 9556 (`shares.bind`) when file shares are on; llama-server on its port (0.0.0.0 in containers, with its own API key) | *(assumption, Q19)* | | Logs | the setup code and a `/setup?code=` link at WARNING; sign-in and reset links when mail is not set | *(documented, docs/authentication.md "the API prints in its log")* | | Files | everything under `data_dir`, including `secret.key` (mode 600) when `ARCHIVE_SECRET_KEY` is unset; temp dirs for imports, rclone config, LibreOffice and Chromium profiles | *(inferred, Q19)* | | Process-wide state | `rdflib` `SPARQL_LOAD_GRAPHS = False` on each SPARQL call; `sys.path` gains `data_dir/python` (components); a signal handler in the CLI | *(assumption, Q19)* | @@ -242,6 +242,7 @@ machine requires HTTPS, and a changed DB credential when SurrealDB runs as its o | Chromium sandbox | off as root or without user namespaces | the `netguard` proxy is the only barrier | Yes | *(documented, docs/configuration.md "runs with its sandbox where it can")* | | `sensors.enabled`, `sensors.mqtt_anonymous` | off (`bind` `0.0.0.0`); off | opens MQTT and syslog listeners; anonymous MQTT publish | Yes; anonymous is the admin's choice | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | | `shares.enabled` | off (`bind` `0.0.0.0`) | opens the read-only S3 and WebDAV servers for the asset tree | Yes | *(documented, docs/file-tree.md "Off until an admin turns it on")* | +| `sensors.mqtt_web` | off | the broker also takes MQTT over WebSockets on `sensors.mqtt_web_port` (9883), which the web app relays at `/mqtt` (`nextjs-frontend/lens-relay.js`), so the tunnel and HTTPS reach the broker from the internet; there it takes hub logins only, never anonymous clients, and throttles failed logins per visitor | Yes | *(documented, docs/sensors.md "At the web app's address")* | | `shares.remote` | off | the web app relays the file shares at its own address (`/s3`, `/dav`; `nextjs-frontend/lens-relay.js`), so the tunnel and HTTPS reach them from the internet; until it's on, the share servers refuse requests carrying `X-Forwarded-Prefix` before any sign-in | Yes | *(documented, docs/file-tree.md "At the web app's address")* | | `tunnel.mode`, `local_llm.enabled`, `telemetry`, `analytics.enabled` | off | outbound tunnel (Cloudflare terminates TLS, Q30); local model server; OTLP export; GA4 Measurement Protocol export of daily counts | Yes | *(documented, docs/remote-access.md "It's off until an admin turns it on")* | | `https.enabled` (and install.sh's `LENS_HTTPS=1`, which Settings › HTTPS can also ask the upgrade agent for) | off | Caddy, fetched at run time, terminates TLS on ports 443/80 in the worker and proxies to the web app; the DNS provider's credentials are written to `data_dir/https/caddy.json` (mode 600), not a command line | Yes | *(documented, docs/https.md "Off by default")* | @@ -303,6 +304,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | NFS share (port 12049 on the host, 2049 in the worker) | NFS and MOUNT RPCs, client address | yes, once enabled, from `nfs.allowed_networks` | data, x-network-source | admin enables and picks the account; operator firewalls | *(documented, docs/file-shares.md "NFS has no login")* | | SMB share (port 1445 on the host, 4445 in the worker) | SMB2 requests, NTLM sign-in with a file access key, client address | yes, once enabled, from `smb.allowed_networks` | data, x-credential, x-network-source | admin enables; operator firewalls | *(documented, docs/smb.md "What keeps it safe")* | | S3 9555 and WebDAV 9556 (file shares) | signed S3 requests (SigV4 header or presigned link); WebDAV with an access key as Basic sign-in | yes, once enabled | x-credential, resource-name | admin enables; each request carries a person's own access key; read-only; failed WebDAV sign-ins throttled per address | *(documented, docs/file-tree.md "Off until an admin turns it on")* | +| web app `/mqtt` and port 9883 (MQTT over WebSockets, relayed by `lens-relay.js` ahead of Next) | the WebSocket handshake, MQTT packets in binary messages, hub logins | yes, once `sensors.enabled`, `sensors.mqtt` and `sensors.mqtt_web`; from the internet through the tunnel or HTTPS | data, size/rate, x-credential, x-network-source | admin enables; a hub login on every connection (anonymous refused whatever `mqtt_anonymous` says); eight failed logins from one visitor in 15 minutes turn it away, the visitor read from `X-Forwarded-For` as trusted proxies report it; the login rides inside MQTT, not in a cookie, so another site's page can't borrow it | *(documented, docs/sensors.md "At the web app's address")* | | web app `/s3` and `/dav` (file shares relayed by `lens-relay.js`, ahead of Next) | any method, path, `Host` and body passed to the share servers with `X-Forwarded-Prefix` and `X-Forwarded-For` | yes, once `shares.enabled` and `shares.remote`; from the internet through the tunnel or HTTPS | x-credential, resource-name, x-network-source | admin enables both; the same per-person access keys, read-only; failed WebDAV sign-ins throttled per visitor from `X-Forwarded-For` as trusted proxies report it (`server.trusted_proxies`, `LENS_TRUSTED_PROXY_HOSTS`, loopback); `X-Forwarded-Prefix` sent straight to a share port only gets the sender refused or signed and linked under that path | *(documented, docs/file-tree.md "At the web app's address")* | | `.env`, `archive.yaml`, `LENS_*`, `ARCHIVE_SECRET_KEY`, `RCLONE_BINARY` | process configuration | no (operator) | x-env-config | keep unreadable to others | *(maintainer, 2026-10)* | | workers and DB peers | DB credential, shared database rows | no (trusted peer) | x-peer | run peers on trusted hosts | *(inferred, Q9)* | @@ -482,7 +484,7 @@ Deployment context: one instance, exposed to the internet behind HTTPS or kept o | Viewer, editor or owner of a namespace | yes | their role; uploads, extensions, notification targets (owners), custom nodes | admin actions; other namespaces; host code | cross-namespace access, SSRF, code execution | *(maintainer, 2026-10)* | | API token, OAuth app, MCP agent | yes | the person's roles, limited by scope | administration (apps); writes with a read token | act beyond scope | *(documented, docs/authentication.md "Tokens act as the person")* | | Content author (uploaded file, transcript, web page, email, calendar, storage remote, IIIF server) | yes | the bytes Lens parses or feeds to a model; the redirects a remote server sends | a role in the namespace; choosing which tools the person has | parser compromise, SSRF through redirects, prompt injection | *(inferred, Q26)* | -| LAN host or device (sensors on) | yes | traffic to MQTT and syslog ports | a hub login (unless `mqtt_anonymous`); a listed syslog network | inject or read readings | *(documented, docs/sensors.md "Only senders on")* | +| LAN host or device (sensors on); any internet client at `/mqtt` with `sensors.mqtt_web` | yes | traffic to MQTT and syslog ports; MQTT over WebSockets through the web app | a hub login (unless `mqtt_anonymous`, which never applies at `/mqtt`); a listed syslog network | inject or read readings | *(documented, docs/sensors.md "Only senders on")* | | Network attacker on the path | yes, on exposed HTTPS deployments | observe and alter traffic outside TLS | breaking TLS; plain-HTTP exposed hosts count as out (§1.6) | steal sessions | *(maintainer, 2026-10)* | | Instance admin | no (trusted) | everything, including host code | nothing: trusted | out of model | *(maintainer, 2026-10)* | | Host operator, log readers, and people the operator gave a backup to | no (trusted) | `.env`, `archive.yaml`, `data_dir`, logs | nothing: trusted. A thief or third party holding a copy of a backup or `data_dir` is not this actor; such a report escalates under N-KEY-WITH-DATA | out of model | *(maintainer, 2026-10)* | @@ -540,7 +542,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-ENCRYPTION | With `encryption.files` on, files are encrypted per namespace. A file cut short, edited or reordered fails to open. A vault has no recovery code, password or admin override, and separate workers never get its key. | info-leak, integrity-bypass | security-critical | *(documented, docs/encryption.md "There is no recovery code, password or admin override")* | Voided by: `encryption.files` off — `domain/keyring.py:545`; shipped default off (`domain/store.py:402`) until a new archive's first start or `lens encrypt`. Unlocked vault keys stay in the API for `encryption.vault_minutes`. `secret.key` placement (Q14). | | P-TELEMETRY | Telemetry is off unless turned on, has no built-in destination, and never sends content, names, addresses or error messages. | info-leak | security-critical | *(documented, docs/telemetry.md "Never sent")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-MODEL-CHECKSUM | Catalog GGUF models are pinned to a revision and checked against their SHA-256. | integrity-bypass | security-critical | *(documented, docs/local-models.md "checked against their SHA-256")* | Voided by: `hf:` references have no pin or hash — `domain/local_llm.py:138-139`. | -| P-SENSORS | Sensors are off until an admin enables them. MQTT needs a hub login unless `mqtt_anonymous` is on. Syslog hears only listed networks. | integrity-bypass | security-critical | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | Voided by: `sensors.mqtt_anonymous` — `domain/sensors.py:984`; `sensors.syslog_networks`, `sensors.bind`. | +| P-SENSORS | Sensors are off until an admin enables them. MQTT needs a hub login unless `mqtt_anonymous` is on, and always over WebSockets (`/mqtt`, `sensors.mqtt_web`). Syslog hears only listed networks. | integrity-bypass | security-critical | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | Voided by: `sensors.mqtt_anonymous` — `domain/sensors.py:984`; `sensors.syslog_networks`, `sensors.bind`; `sensors.mqtt_web` with the web app on plain HTTP sends hub passwords in clear. | | P-NFS | The NFS share is off until an admin turns it on. It is read-only, shows only what one chosen account sees in the web app, and turns away every client outside `nfs.allowed_networks` before rclone reads a byte. | info-leak | security-critical | *(documented, docs/file-shares.md "NFS has no login")* | Voided by: `nfs.allowed_networks` set wide, or a client address spoofed on the local network (NFS has no login) — `domain/share_relay.py`; Docker publishing that rewrites client addresses. | | P-SMB | The SMB share is off until an admin turns it on. It is read-only, signs people in with their own file access keys and shows each only what they see in the web app, refuses guests and SMB1 sessions, gives every connection its own NTLM challenge, and closes connections from outside `smb.allowed_networks` before reading a byte. | info-leak | security-critical | *(documented, docs/smb.md "What keeps it safe")* | Voided by: `smb.allowed_networks` set wide; SMB 2.0 has no encryption on the wire, so a key's NTLM exchange and the files read can be watched on the network — `domain/smb_share.py`. | | P-SHARES | The file shares (S3 and WebDAV) are off until an admin enables them. Every request is signed (S3: AWS Signature Version 4) or signed in (WebDAV: Basic) with an access key someone made while signed in to the web app; it reads as them, sees only what the web app shows them, and never writes. Keys expire as API keys do, stop with a disabled account, and are revoked at once. | info-leak, integrity-bypass | security-critical | *(documented, docs/file-tree.md "Off until an admin turns it on")* | Through the web app (`/s3`, `/dav`) only with `shares.remote` on as well. Voided by: `shares.enabled` with `LENS_BIND=0.0.0.0` sends files, S3 signatures and WebDAV's Basic secret in clear over plain HTTP on the network, as does `shares.remote` when the web app itself is reached over plain HTTP; key secrets are sealed with the server key (`secret.key`), not hashed, since SigV4 signs with the secret itself — `domain/s3.py`, `domain/webdav.py`. | diff --git a/threat-model.yaml b/threat-model.yaml index f85cb4661..ef8f780cc 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:9c0063b128c9082c99f58c1acb64e303f33dfad2a8c520ef0294a25b587a165e" +prose_version: "threat-model.md@sha256:b05cb02c6053092b86edca1b831debc2bf64b2fec529ccae3f59b1d0af8adb42" model_status: under-review triage_policy: strict confidence: {documented: 204, maintainer: 46, inferred: 91, assumption: 8} @@ -113,7 +113,7 @@ host_side_effects: - effect: network-sockets stance: conditional components: [web, outbound, optional] - conditions: "API and web app listen; a netguard proxy on loopback during captures; MQTT 1883 and syslog 5514 on sensors.bind (default 0.0.0.0) when sensors are on; llama-server on its port (0.0.0.0 in containers, with its own API key)" + conditions: "API and web app listen; a netguard proxy on loopback during captures; MQTT 1883 and syslog 5514 on sensors.bind (default 0.0.0.0) when sensors are on, and MQTT over WebSockets on 9883 with sensors.mqtt_web; llama-server on its port (0.0.0.0 in containers, with its own API key)" provenance: {kind: assumption, question_id: Q19} - effect: x-log-credentials stance: present @@ -486,6 +486,15 @@ entry_points: obligation_id: enable-and-limit-smb caller_must_enforce: "admin enables; operator firewalls" provenance: {kind: documented, source: 'docs/smb.md "What keeps it safe"'} + - id: "web app /mqtt and port 9883 (MQTT over WebSockets, relayed by lens-relay.js ahead of Next)" + component: optional + parameters: + - name: "the WebSocket handshake, MQTT packets in binary messages, hub logins" + attacker_controllable: true + x-condition: "once sensors.enabled, sensors.mqtt and sensors.mqtt_web; from the internet through the tunnel or HTTPS" + control_kinds: [data, size, rate, x-credential, x-network-source] + caller_must_enforce: "admin enables; a hub login on every connection (never anonymous); failed logins throttled per visitor from X-Forwarded-For as trusted proxies report it; no cookie, so no cross-site WebSocket hijacking" + provenance: {kind: documented, source: 'docs/sensors.md "At the web app''s address"'} - id: "web app /s3 and /dav (file shares relayed by lens-relay.js, ahead of Next)" component: optional parameters: @@ -1204,6 +1213,14 @@ build_flags: affects_properties: - {property_id: P-SENSORS, effect: narrows} provenance: {kind: documented, source: 'docs/sensors.md "Sensors are off until an admin turns them on"'} + - name: "sensors.mqtt_web" + default: "off" + security_relevant: true + maintainer_stance: supported + x-scope: "the broker also takes MQTT over WebSockets on sensors.mqtt_web_port (9883), which the web app relays at /mqtt, so the tunnel and HTTPS reach it; hub logins only there, failed ones throttled per visitor" + affects_properties: + - {property_id: P-SENSORS, effect: narrows} + provenance: {kind: documented, source: 'docs/sensors.md "At the web app''s address"'} - name: "shares.enabled" default: "off" security_relevant: true @@ -1639,9 +1656,9 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "sensors are off until an admin enables them; MQTT needs a hub login unless mqtt_anonymous is on; syslog hears only listed networks" + conditions: "sensors are off until an admin enables them; MQTT needs a hub login unless mqtt_anonymous is on, and always over WebSockets (/mqtt, sensors.mqtt_web); syslog hears only listed networks" violation_symptoms: [integrity-bypass] - x-voided-by: "sensors.mqtt_anonymous (domain/sensors.py:984); sensors.syslog_networks, sensors.bind" + x-voided-by: "sensors.mqtt_anonymous (domain/sensors.py:984); sensors.syslog_networks, sensors.bind; sensors.mqtt_web with the web app on plain HTTP (hub passwords in clear)" provenance: {kind: documented, source: 'docs/sensors.md "Sensors are off until an admin turns them on"'} - id: P-NFS kind: confidentiality From b245668a977a7b6a60c177303ea7e1459bbb11db Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 17 Oct 2026 09:26:56 +0000 Subject: [PATCH 2/2] fix(threat-model): keep MQTT over WebSockets' entries after merging main The merge of main took main's whole threat-model.yaml, dropping the /mqtt entry point, the sensors.mqtt_web flag and P-SENSORS' conditions. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01CXpr1EWCfUD5tb1SmSwfDa --- threat-model.yaml | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/threat-model.yaml b/threat-model.yaml index 567079594..e7dd1d08d 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -116,7 +116,7 @@ host_side_effects: - effect: network-sockets stance: conditional components: [web, outbound, optional] - conditions: "API and web app listen; a netguard proxy on loopback during captures; MQTT 1883 and syslog 5514 on sensors.bind (default 0.0.0.0) when sensors are on; llama-server on its port (0.0.0.0 in containers, with its own API key)" + conditions: "API and web app listen; a netguard proxy on loopback during captures; MQTT 1883 and syslog 5514 on sensors.bind (default 0.0.0.0) when sensors are on, and MQTT over WebSockets on 9883 with sensors.mqtt_web; llama-server on its port (0.0.0.0 in containers, with its own API key)" provenance: {kind: assumption, question_id: Q19} - effect: x-log-credentials stance: present @@ -508,6 +508,15 @@ entry_points: obligation_id: enable-and-limit-smb caller_must_enforce: "admin enables; operator firewalls" provenance: {kind: documented, source: 'docs/smb.md "What keeps it safe"'} + - id: "web app /mqtt and port 9883 (MQTT over WebSockets, relayed by lens-relay.js ahead of Next)" + component: optional + parameters: + - name: "the WebSocket handshake, MQTT packets in binary messages, hub logins" + attacker_controllable: true + x-condition: "once sensors.enabled, sensors.mqtt and sensors.mqtt_web; from the internet through the tunnel or HTTPS" + control_kinds: [data, size, rate, x-credential, x-network-source] + caller_must_enforce: "admin enables; a hub login on every connection (never anonymous); failed logins throttled per visitor from X-Forwarded-For as trusted proxies report it; no cookie, so no cross-site WebSocket hijacking" + provenance: {kind: documented, source: 'docs/sensors.md "At the web app''s address"'} - id: "web app /s3 and /dav (file shares relayed by lens-relay.js, ahead of Next)" component: optional parameters: @@ -1245,6 +1254,14 @@ build_flags: affects_properties: - {property_id: P-SENSORS, effect: narrows} provenance: {kind: documented, source: 'docs/sensors.md "Sensors are off until an admin turns them on"'} + - name: "sensors.mqtt_web" + default: "off" + security_relevant: true + maintainer_stance: supported + x-scope: "the broker also takes MQTT over WebSockets on sensors.mqtt_web_port (9883), which the web app relays at /mqtt, so the tunnel and HTTPS reach it; hub logins only there, failed ones throttled per visitor" + affects_properties: + - {property_id: P-SENSORS, effect: narrows} + provenance: {kind: documented, source: 'docs/sensors.md "At the web app''s address"'} - name: "shares.enabled" default: "off" security_relevant: true @@ -1689,9 +1706,9 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "sensors are off until an admin enables them; MQTT needs a hub login unless mqtt_anonymous is on; syslog hears only listed networks" + conditions: "sensors are off until an admin enables them; MQTT needs a hub login unless mqtt_anonymous is on, and always over WebSockets (/mqtt, sensors.mqtt_web); syslog hears only listed networks" violation_symptoms: [integrity-bypass] - x-voided-by: "sensors.mqtt_anonymous (domain/sensors.py:984); sensors.syslog_networks, sensors.bind" + x-voided-by: "sensors.mqtt_anonymous (domain/sensors.py:984); sensors.syslog_networks, sensors.bind; sensors.mqtt_web with the web app on plain HTTP (hub passwords in clear)" provenance: {kind: documented, source: 'docs/sensors.md "Sensors are off until an admin turns them on"'} - id: P-NFS kind: confidentiality