Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **The graph as files over SMB.** Settings → SMB share serves the file tree read-only to Finder, Linux and (on port
445) Windows: each person signs in with a file access key and sees what the web app shows them, from the networks
an admin allows. Off by default; the SMB server (impacket) is fetched when it's turned on ([SMB share](docs/smb.md)).
- **A spending limit from the start.** The setup wizard has a Budget step, and the web installer asks for one with a
cloud provider: a scale for a hard limit on everything Lens spends on AI, with a soft limit at 80% that warns. At
the hard limit runs wait (or are skipped) and paid model calls stop until it goes up; models on your own computer
Expand All @@ -16,7 +19,7 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
inside their broader topics, entities by type. Each folder holds its assets' files, with a `.json` sidecar of
metadata and a `.txt` of the text; everyone sees what the web app shows them. Browse and read it through the API
(`/api/v1/tree`) and the MCP tools `list_files` and `read_file` ([The graph as files](docs/file-tree.md)).
SMB and NFS shares are in progress.
An NFS share is in progress.

- **HTTPS with automatic certificates.** Lens can serve itself at `https://` your domain with a certificate it gets
and renews on its own from Let's Encrypt or ZeroSSL (or any ACME authority), proving the domain over HTTP, TLS-ALPN
Expand Down
8 changes: 8 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ x-backend: &backend
LENS_TUNNEL_ORIGIN: http://frontend:3000
# the web app's container (and the one running the tunnel): their X-Forwarded-For names the visitor, for throttles
LENS_TRUSTED_PROXY_HOSTS: frontend,backend,worker
# the SMB share of the file tree (docs/smb.md): listens on 4445 in the container; LENS_SMB_PORT is where it's
# published, for the address Settings shows
LENS_SMB_LISTEN: 0.0.0.0
LENS_SMB_PORT: ${LENS_SMB_PORT:-1445}
# Fedora (docs/fedora.md): empty keeps it off; with the fedora profile, http://fedora:8080/fcrepo/rest
LENS_FEDORA_URL: ${LENS_FEDORA_URL:-}
LENS_FEDORA_USER: ${LENS_FEDORA_USER:-}
Expand Down Expand Up @@ -108,6 +112,10 @@ services:
- "${LENS_SENSOR_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_MQTT_PORT:-1883}:1883"
- "${LENS_SENSOR_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_SYSLOG_PORT:-5514}:5514/udp"
- "${LENS_SENSOR_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_SYSLOG_PORT:-5514}:5514/tcp"
# the file tree over SMB (docs/smb.md), served only once it's turned on in Settings → SMB share, and then only
# to the networks allowed there. 1445 on this machine so an SMB server the host runs on 445 is never in the way
# (Windows only connects to 445: install with LENS_SMB_PORT=445 for it; install.sh keeps it).
- "${LENS_SHARE_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_SMB_PORT:-1445}:4445"
# the graph as files over S3 and WebDAV (docs/file-tree.md): read-only, listened on only once File shares are turned
# on in the app; published where the web app is. Something else using 9555 or 9556? Set LENS_S3_PORT / LENS_WEBDAV_PORT.
- "${LENS_SHARE_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_S3_PORT:-9555}:9555"
Expand Down
5 changes: 5 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ x-backend: &backend
LENS_TUNNEL_ORIGIN: http://frontend:3000
# the web app's container (and the one running the tunnel): their X-Forwarded-For names the visitor, for throttles
LENS_TRUSTED_PROXY_HOSTS: frontend,backend,worker
# the SMB share of the file tree (docs/smb.md): listens on 4445 in the container; LENS_SMB_PORT is where it's
# published, for the address Settings shows
LENS_SMB_LISTEN: 0.0.0.0
LENS_SMB_PORT: ${LENS_SMB_PORT:-1445}
CORS_ORIGINS: '["http://localhost:3000"]'
OPENAPI_OUTPUT_FILE: ./shared-data/openapi.json
# the volume shared with the matterbridge service (its config, written from Settings → Chat rooms)
Expand Down Expand Up @@ -87,6 +91,7 @@ services:
- "${LENS_SENSOR_BIND:-127.0.0.1}:${LENS_MQTT_PORT:-1883}:1883"
- "${LENS_SENSOR_BIND:-127.0.0.1}:${LENS_SYSLOG_PORT:-5514}:5514/udp"
- "${LENS_SENSOR_BIND:-127.0.0.1}:${LENS_SYSLOG_PORT:-5514}:5514/tcp"
- "${LENS_SHARE_BIND:-127.0.0.1}:${LENS_SMB_PORT:-1445}:4445" # the file tree over SMB, once turned on
# the graph as files over S3 and WebDAV (docs/file-tree.md), once File shares are on; LENS_S3_PORT / LENS_WEBDAV_PORT
- "${LENS_SHARE_BIND:-127.0.0.1}:${LENS_S3_PORT:-9555}:9555"
- "${LENS_SHARE_BIND:-127.0.0.1}:${LENS_WEBDAV_PORT:-9556}:9556"
Expand Down
3 changes: 2 additions & 1 deletion docs/file-tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ Lens without learning its API.
| --- | --- |
| The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built |
| An [S3 server](#s3) and a [WebDAV server](#webdav) (read-only, off until turned on) | built |
| SMB and NFS shares | in progress |
| An [SMB share](smb.md) (read-only, off until turned on) | built |
| An NFS share | in progress |

Every way in serves the same tree, built by `app/domain/filetree.py`.

Expand Down
68 changes: 68 additions & 0 deletions docs/smb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# SMB share

Lens can serve [the file tree](file-tree.md) over SMB, the protocol Finder, Windows Explorer and Linux file managers
use for network drives. Namespaces, collections, topics and entities are folders; assets are files inside them, with
their metadata and text beside them. Everyone signs in as themselves and sees exactly what the web app shows them.

*Status: built. Read-only.*

## Turn it on

Settings → SMB share → **Share the file tree over SMB**. It's off until an admin turns it on. The first time, a worker
fetches the SMB server (impacket, about 15 MB), then starts serving within seconds; Settings says when it's running
and shows the address.

Sign in with a **file access key** (the same keys the S3 share takes): the key's id is the user name and its secret
the password. Make one in Settings, keep the secret, and revoke the key to shut its access off: open sessions end
within 30 seconds.

| Where | How |
| --- | --- |
| macOS | Finder → Go → Connect to Server → `smb://<lens-machine>:1445/LENS` |
| Linux | `sudo mount -t cifs -o port=1445,username=<key id>,ro,vers=2.0 //<lens-machine>/LENS /mnt/lens`, or `smb://<lens-machine>:1445/LENS` in Files |
| Windows | Explorer only connects to port 445: install with `LENS_SMB_PORT=445` in front of the install line (kept for later upgrades), then open `\\<lens-machine>\LENS` |

| Setting | Default | What it does |
| --- | --- | --- |
| `smb.enabled` | off | Serve the share |
| `smb.allowed_networks` | private networks (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`, `100.64.0.0/10` for Tailscale, `fc00::/7`, `::1/128`) | Machines anywhere else are turned away before they can sign in; changes apply without a restart |
| `smb.bind`, `smb.port` | `127.0.0.1`, `4445` | Startup settings only: where the worker listens (`LENS_SMB_LISTEN`, `LENS_SMB_LISTEN_PORT`). In Docker it listens on `0.0.0.0:4445` inside the worker container |

With Docker, the worker container publishes the share on port **1445** of the host (`LENS_SMB_PORT=<port>` in front of
the install line picks another), so an SMB server the host already runs on 445 is never in the way. Like the web app, it's reachable from
this machine only unless `LENS_BIND=0.0.0.0` (what `install.sh` sets on a server); `LENS_SHARE_BIND` opens the file
shares on their own while the web app stays local.

Agents read the share's state at `GET /api/v1/settings/smb/status` and turn it on or off with
`PUT /api/v1/settings/smb` (`{"enabled": true}`), like every other setting.

## What keeps it safe

- **Off by default**, and nothing listens until an admin turns it on.
- **Read-only.** Every call that would write, rename, delete or make a folder is refused before it reaches the tree.
- **Per-person views.** A key opens its owner's view of the tree: the namespaces and collections they have a role in,
nothing else. A disabled account or a revoked key stops working within 30 seconds, mid-session too.
- **No guests.** Anonymous and guest sessions get nothing; a share is only handed to a signed-in key.
- **Allowed networks only.** A connection from anywhere else is closed before a byte of SMB is read.
- **Hardened server.** Each connection gets its own random NTLM challenge; SMB1 is refused past the first
negotiation (clients move on to SMB 2); the NTLM responses impacket logs are kept out of Lens's log.
- **Only the tree.** The server's file system is a stand-in that answers from the tree: no path reaches the disk.

SMB 2.0 has no encryption on the wire. Keep the share on networks you trust (or a VPN such as Tailscale), and keep its
port off the internet.

## How it works

The worker runs impacket's SMB server (pure Python; impacket's license is a modified Apache 1.1, and it's fetched only
when the share is turned on, as the `smb` extra). impacket serves folders through Python's `os` module; in Lens, its
`os` is a small read-only stand-in (`domain/smb_share.py`) that looks files up in the tree for the signed-in key's
owner, and reads them as they stream (files in vaults are decrypted on the way; files in storage connections are
fetched as they're read). One person's view of the tree is reused for a few seconds, so the many look-ups a file
manager makes after listing a folder don't each ask the database again.

## Refine later

- SMB 3 with encryption on the wire (impacket speaks SMB 2.0.2).
- Finder's Bonjour discovery, so the share shows up on its own.
- Packaging for the Synology and QNAP compose files, which don't publish the port yet.
- Writing: dropping a file into a collection's folder to import it.
8 changes: 8 additions & 0 deletions fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
monitoring,
semantic,
settings,
smb_share,
sources,
speech,
store,
Expand All @@ -55,6 +56,7 @@
MailTestResult,
Removed,
SemanticStatus,
SmbStatus,
SpeechTestResult,
Started,
TelemetryStatus,
Expand Down Expand Up @@ -178,6 +180,12 @@ def remove_local_model(user: AdminWriter, cfg: Cfg, db: Db, model: str = Query(.
return Removed(removed=removed)


@router.get("/settings/smb/status")
def smb_status(user: AdminReader, cfg: Cfg, db: Db) -> SmbStatus:
"""The file tree over SMB (Settings › SMB share): whether it's serving, to which networks, and how to connect."""
return SmbStatus.model_validate(smb_share.status(db, cfg))


@router.get("/settings/tunnel/status")
def tunnel_status(user: AdminReader, cfg: Cfg, db: Db) -> TunnelStatus:
"""How the Cloudflare tunnel (Settings › Remote access) is doing: its address, whether it's connected, and
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,7 @@ def platform_main(argv, config):
sensors,
settings,
shares,
smb_share,
sources,
telemetry,
tunnel,
Expand Down Expand Up @@ -380,6 +381,7 @@ def password():
if not (a.steps or a.no_schedule): # and scans watched folders and runs routines, as `lens watch` does
routines.start(db, C, stop, log=print)
sensors.start(db, C, stop, log=print, name=wk.name) # and the sensor hub, while sensors are on
smb_share.start(db, C, stop, name=wk.name, log=print) # and the SMB share of the file tree, while it's on
shares.start(db, C, stop, log=print, name=wk.name) # and the file shares (S3, WebDAV), while they're on
bridge.start(db, C, stop, lambda: cfg, name=wk.name, log_fn=print) # and the chat-room bridge, while it's on
tunnel.start(db, C, stop, name=wk.name, log=print) # and the Cloudflare tunnel, while it's on
Expand Down
14 changes: 14 additions & 0 deletions fastapi_backend/app/domain/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -529,6 +529,19 @@ def fetch(self, cfg, m, say):
pip_install(cfg, "msg", say)


class Smb(Component):
"""impacket, for the SMB share of the file tree (smb_share.py): fetched when the share is turned on."""

def needed(self, cfg, m):
return bool((cfg.get("smb") or {}).get("enabled"))

def present(self, cfg):
return importable("impacket.smbserver")

def fetch(self, cfg, m, say):
pip_install(cfg, "smb", say)


class Laya(Component):
"""laya-mlx and the chosen Laya model, for local decisions (decide.py) on Apple Silicon."""

Expand Down Expand Up @@ -682,6 +695,7 @@ def detail(self, cfg):
),
OllamaModel("llm-model", "Chat model", "the LLM provider's model, pulled on Ollama", section="llm"),
OllamaModel("embedding-model", "Embedding model", "search by meaning's model, pulled on Ollama", section="embeddings", steps={"embed"}),
Smb("smb", "SMB server", "shares the file tree over SMB, read-only (smb.enabled; impacket)", size_mb=15, license="Apache-1.1"),
Msg("msg", "Outlook .msg emails", "reads Outlook .msg files (extract-msg)", optional=True, license="GPL-3.0", size_mb=5),
]
BY_ID = {c.id: c for c in COMPONENTS}
Expand Down
20 changes: 20 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@
"tunnel": None,
"https": None,
# bind is a startup setting only
# the SMB share of the file tree (smb_share.py): bind and port are startup settings only (Docker publishes the port)
"smb": ("enabled", "allowed_networks"),
"sensors": (
"enabled",
"mqtt",
Expand Down Expand Up @@ -431,6 +433,8 @@ def _check(section, key, value, default):
return _share_setting(key, value)
if section == "sensors":
return _sensor_setting(key, value)
if section == "smb":
return _smb_setting(key, value)
if section == "components":
return _component_setting(key, value)
if section == "mail" and key != "security":
Expand Down Expand Up @@ -918,6 +922,22 @@ def _sensor_setting(key, value):
return value


def _smb_setting(key, value):
if key == "enabled":
if not isinstance(value, bool):
raise ValueError("smb.enabled is true or false")
return value
if not isinstance(value, list) or len(value) > 100:
raise ValueError("smb.allowed_networks is a list of networks like 192.168.1.0/24")
out = []
for v in value:
try:
out.append(str(ipaddress.ip_network(str(v).strip(), strict=False)))
except ValueError:
raise ValueError(f"smb.allowed_networks: {v} isn't a network like 192.168.1.0/24") from None
return list(dict.fromkeys(out))


def _share_setting(key, value):
if key in ("enabled", "s3", "webdav"):
if not isinstance(value, bool):
Expand Down
Loading
Loading