From b410e097c64e4e9a3b54532a95b031b5b5c5d511 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Thu, 15 Oct 2026 11:46:09 +0000 Subject: [PATCH] feat(smb): import files added over the SMB share A file copied into the SMB share is written to the uploads disk while it's open and imported when it's closed (share_import.Adder.begin and finish), with the same permission and checks as an upload. MKCOL-style folder creates under Collections make collections; renaming a file kept aside imports it; deleting one removes it. Files of a type Lens doesn't take are refused at create, files over the upload limit are cut off as they're written and dropped, and files deleted on close or left open when the connection ends are never imported. smb.add_files (on) makes the share writable; off, it's read-only as before. Adder.check now checks a file's type even before its size is known (S3 multipart, chunked WebDAV, SMB creates). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g --- CHANGELOG.md | 7 + README.md | 4 +- docs/file-tree.md | 14 +- docs/requirements.md | 2 +- docs/smb.md | 37 ++- fastapi_backend/app/domain/settings.py | 6 +- fastapi_backend/app/domain/share_import.py | 70 ++++- fastapi_backend/app/domain/smb_share.py | 267 ++++++++++++++++-- fastapi_backend/app/domain/store.py | 1 + fastapi_backend/app/schemas/admin.py | 1 + .../tests/domain/test_smb_share.py | 6 + .../tests/domain/test_smb_writes.py | 175 ++++++++++++ .../app/openapi-client/types.gen.ts | 6 + nextjs-frontend/components/settings/model.ts | 11 +- nextjs-frontend/openapi.json | 6 + threat-model.json | 4 +- threat-model.md | 6 +- threat-model.yaml | 10 +- 18 files changed, 579 insertions(+), 54 deletions(-) create mode 100644 fastapi_backend/tests/domain/test_smb_writes.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 288cf5803..27fa867f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,6 +101,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version only the version showing. In the web app, Regenerate (and Regenerate with… another model) replaces Try another model, and `‹ 2 / 3 ›` under a question or an answer moves between its versions. Existing conversations are upgraded on the next start (`chat-tree`) ([Versions](docs/assistant.md)). +- **Files added over SMB are imported.** Copy files into a collection's folder on the SMB share and each is imported + when it's closed, as an upload is; a new folder under `Collections` is a new collection. Files of a type Lens doesn't + take are refused as they're made, files over the upload limit are cut off as they're written, and a file deleted as + it's closed or left open when its client goes away isn't imported. The system's own files (`.DS_Store`, `Thumbs.db`, + Office's `~$` files) are kept aside, and renaming a file saved under a temporary name imports it. **Import files + people add** in Settings, SMB share (`smb.add_files`, on) turns it off and makes the share read-only again ([Adding + files](docs/smb.md#adding-files)). - **The assistant searches the web.** With web search on (off by default; Settings → AI assistant → Web search), the assistant gets `web_search` and `web_read`: it searches with DuckDuckGo (no key), your own SearXNG, Brave Search or Tavily, and reads pages and online PDFs, public addresses only, through the same guard as captured pages. Pages are diff --git a/README.md b/README.md index 22da01f9d..f3f346d48 100644 --- a/README.md +++ b/README.md @@ -71,8 +71,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p completions API for any app, cited from that namespace alone, each off until its owner turns it on, with API keys that open that one namespace and nothing else. The same folders and files can be mounted over S3, WebDAV, NFS or SMB (S3 and WebDAV also at the web app's address, through remote access), so file managers and scripts read Lens too, and a - file copied in over S3 or WebDAV is imported where it was put. The web app runs on the same REST API, so an agent can - do what a person can. + file copied in over S3, WebDAV or SMB is imported where it was put. The web app runs on the same REST API, so an agent + can do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app), [API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md); diff --git a/docs/file-tree.md b/docs/file-tree.md index 892b93472..d30928a6e 100644 --- a/docs/file-tree.md +++ b/docs/file-tree.md @@ -3,15 +3,15 @@ Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read -Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3 or WebDAV -is imported there ([Adding files](#adding-files)). +Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3, WebDAV or +SMB is imported there ([Adding files](#adding-files)). | Where | Status | | --- | --- | | The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built | | An [S3 server](#s3) and a [WebDAV server](#webdav) (off until turned on); files added through them are imported | built | | An [NFS share](file-shares.md) (read-only, off until turned on, allowed networks only) | built | -| An [SMB share](smb.md) (read-only, off until turned on) | built | +| An [SMB share](smb.md) (off until turned on); files added through it are imported | built | Every way in serves the same tree, built by `app/domain/filetree.py`. @@ -47,7 +47,7 @@ Names are made safe for every file system: `/ \ : * ? " < > |` and control chara Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces they have a role in, and in a namespace where they only have roles on some collections, those collections and the topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The API only -reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers. +reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers and the [SMB share](smb.md). ## The API @@ -118,7 +118,7 @@ behind TLS on a network you don't trust. ## Adding files -A file copied into the tree over S3 or WebDAV is imported, as an upload in the web app is: the same checks of its type +A file copied into the tree over S3, WebDAV or [SMB](smb.md) is imported, as an upload in the web app is: the same checks of its type and size (Settings, Uploads), the namespace's pipeline, and the same file twice is added once. Where it's put decides where it goes: @@ -143,8 +143,8 @@ after they last change, seen only by you, and never imported; renaming one to an kept on the uploads disk, in memory per worker, so a restart forgets them. Adding files is on whenever the shares are; turn it off in Settings, File shares (**Import files people add**, -`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The -[NFS](file-shares.md) and [SMB](smb.md) shares stay read-only for now. +`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The SMB +share has its own switch (`smb.add_files`, on). The [NFS](file-shares.md) share stays read-only for now. ## At the web app's address diff --git a/docs/requirements.md b/docs/requirements.md index 6b817e880..2ea8ee958 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -66,7 +66,7 @@ request, to another check that proves the same thing; loosening a requirement or | API keys that open one namespace and nothing else | [test_namespace_keys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_namespace_keys.py) `test_a_key_for_one_namespace_opens_its_servers_alone`, `test_it_has_the_persons_role_there_and_no_more` | | The graph's folders and files mounted read-only over S3, WebDAV, NFS and SMB | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_people_read_their_namespaces_over_s3`, `test_what_s3_refuses`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_mount_and_read`, `test_what_webdav_refuses`; [test_nfs_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_nfs_share.py) `test_the_runner_follows_the_setting`; [test_smb_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_share.py) `test_reads_the_tree_as_its_key_sees_it`, `test_nothing_can_be_written` | | S3 and WebDAV at the web app's address, through remote access, once an admin allows it | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`, `test_through_the_web_app_each_visitor_has_its_own_throttle` | -| A file copied in over S3 or WebDAV is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused` | +| A file copied in over S3, WebDAV or SMB is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused`; [test_smb_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_writes.py) `test_a_file_copied_into_a_collection_is_imported_there`, `test_new_folders_under_collections_are_new_collections`, `test_what_may_not_be_added_is_refused`, `test_the_share_is_read_only_while_add_files_is_off` | | The web app runs on the same REST API, through a client generated from its OpenAPI schema | [ci.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/ci.yml) `OpenAPI schema is up to date` | ## Ask, and get cited answers diff --git a/docs/smb.md b/docs/smb.md index ff078c8d2..14780b10b 100644 --- a/docs/smb.md +++ b/docs/smb.md @@ -2,9 +2,10 @@ Lens can serve [the file tree](file-tree.md) over SMB, the protocol Finder, Windows Explorer and Linux file managers use for network drives. Namespaces, collections, topics and entities are folders; assets are files inside them, with -their metadata and text beside them. Everyone signs in as themselves and sees exactly what the web app shows them. +their metadata and text beside them. Everyone signs in as themselves and sees exactly what the web app shows them, and +a file they copy into a collection's folder is imported there ([Adding files](file-tree.md#adding-files)). -*Status: built. Read-only.* +*Status: built.* ## Turn it on @@ -19,12 +20,13 @@ within 30 seconds. | Where | How | | --- | --- | | macOS | Finder → Go → Connect to Server → `smb://:1445/LENS` | -| Linux | `sudo mount -t cifs -o port=1445,username=,ro,vers=2.0 ///LENS /mnt/lens`, or `smb://:1445/LENS` in Files | +| Linux | `sudo mount -t cifs -o port=1445,username=,vers=2.0 ///LENS /mnt/lens` (add `ro` to only read), or `smb://:1445/LENS` in Files | | Windows | Explorer only connects to port 445: install with `LENS_SMB_PORT=445` in front of the install line (kept for later upgrades), then open `\\\LENS` | | Setting | Default | What it does | | --- | --- | --- | | `smb.enabled` | off | Serve the share | +| `smb.add_files` | on | Import the files people copy in (**Import files people add**); off, the share is read-only. Turning it on or off restarts the share, which ends open sessions | | `smb.allowed_networks` | private networks (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`, `100.64.0.0/10` for Tailscale, `fc00::/7`, `::1/128`) | Machines anywhere else are turned away before they can sign in; changes apply without a restart | | `smb.bind`, `smb.port` | `127.0.0.1`, `4445` | Startup settings only: where the worker listens (`LENS_SMB_LISTEN`, `LENS_SMB_LISTEN_PORT`). In Docker it listens on `0.0.0.0:4445` inside the worker container | @@ -36,10 +38,29 @@ shares on their own while the web app stays local. Agents read the share's state at `GET /api/v1/settings/smb/status` and turn it on or off with `PUT /api/v1/settings/smb` (`{"enabled": true}`), like every other setting. +## Adding files + +Copy files into a collection's folder (with a file manager, `cp` onto a mount, or `smbclient`'s `put`) and each is +imported when it's closed, as an upload is: the same checks of its type and size, the namespace's pipeline, +and the same file twice is added once. A new folder under `Collections` is a new collection; a file anywhere else in a +namespace goes to its default collection. It takes the import permission (or an editor's role on the collection), as +the web app does; [Adding files](file-tree.md#adding-files) has the rest. + +A file is refused as it's made when its type can't be uploaded, and cut off as it's written once it's larger than +uploads take (`uploads.max_mb`); a file that was cut off, deleted as it was closed, or still open when its client went +away isn't imported. Files the system writes for itself (`.DS_Store`, `._` files, `Thumbs.db`, `desktop.ini`, Office's +`~$` files) and files saved under a temporary name are kept aside for a while, seen only by you; renaming one to an +ordinary name imports it, which is how apps that save to a temporary file and rename it work. What Lens made can't be +written over, renamed or deleted. + ## What keeps it safe - **Off by default**, and nothing listens until an admin turns it on. -- **Read-only.** Every call that would write, rename, delete or make a folder is refused before it reaches the tree. +- **Adds, never changes.** A file is imported only where its owner may import, checked as an upload is. Nothing Lens + made can be written over, renamed or deleted, and namespaces can't be made. With `smb.add_files` off, every call that + would write, rename, delete or make a folder is refused before it reaches the tree. +- **Files arrive on Lens's disk, not in memory.** A file being written goes to the uploads folder, cut off at the + upload size limit, and is imported or dropped when it's closed. - **Per-person views.** A key opens its owner's view of the tree: the namespaces and collections they have a role in, nothing else. A disabled account or a revoked key stops working within 30 seconds, mid-session too. - **No guests.** Anonymous and guest sessions get nothing; a share is only handed to a signed-in key. @@ -55,8 +76,9 @@ port off the internet. The worker runs impacket's SMB server (pure Python; impacket's license is a modified Apache 1.1, and it's fetched only when the share is turned on, as the `smb` extra). impacket serves folders through Python's `os` module; in Lens, its -`os` is a small read-only stand-in (`domain/smb_share.py`) that looks files up in the tree for the signed-in key's -owner, and reads them as they stream (files in vaults are decrypted on the way; files in storage connections are +`os` is a small stand-in (`domain/smb_share.py`) that looks files up in the tree for the signed-in key's +owner, writes files people add to the uploads folder and hands them to the importer (`domain/share_import.py`) as +they're closed, and reads files as they stream (files in vaults are decrypted on the way; files in storage connections are fetched as they're read). One person's view of the tree is reused for a few seconds, so the many look-ups a file manager makes after listing a folder don't each ask the database again. @@ -65,4 +87,5 @@ manager makes after listing a folder don't each ask the database again. - SMB 3 with encryption on the wire (impacket speaks SMB 2.0.2). - Finder's Bonjour discovery, so the share shows up on its own. - Packaging for the Synology and QNAP compose files, which don't publish the port yet. -- Writing: dropping a file into a collection's folder to import it. +- Delete, rename and move assets through the share (only files just added can be today). +- Free space as the uploads disk has it (impacket reports a fixed size). diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index bba6483eb..21183d8b1 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -68,7 +68,7 @@ "nfs": ("enabled", "port", "allowed_networks", "account"), # bind is a startup setting only # the SMB share of the file tree (smb_share.py): bind and port are startup settings only (Docker publishes the port) - "smb": ("enabled", "allowed_networks"), + "smb": ("enabled", "allowed_networks", "add_files"), "sensors": ( "enabled", "mqtt", @@ -1306,9 +1306,9 @@ def _nfs_setting(key, value): def _smb_setting(key, value): - if key == "enabled": + if key in ("enabled", "add_files"): if not isinstance(value, bool): - raise ValueError("smb.enabled is true or false") + raise ValueError(f"smb.{key} is true or false") return value if not isinstance(value, list) or len(value) > 100: raise ValueError("smb.allowed_networks is a list of networks like 192.168.1.0/24") diff --git a/fastapi_backend/app/domain/share_import.py b/fastapi_backend/app/domain/share_import.py index 56bdc84e7..da211132c 100644 --- a/fastapi_backend/app/domain/share_import.py +++ b/fastapi_backend/app/domain/share_import.py @@ -332,9 +332,9 @@ def check(self, path, size=None): raise Rejected(f"collections go at most {hierarchy.MAX_DEPTH} deep") elif not self.may_import(where.space, self._target(where)): raise Refused("adding files here needs the import permission (an editor's role)") - if not junk(where.name) and size: + if not junk(where.name) and size != 0: # an empty file is kept aside: its bytes may follow (Finder, Explorer) try: - uploads.check(self.cfg, where.name, size) + uploads.check(self.cfg, where.name, size if size is not None else 1) # no size yet: its type except uploads.TooLarge as e: raise TooLarge(str(e)) from None except ValueError as e: @@ -395,8 +395,8 @@ def folder(self, path): return True def touch(self, path, modified): - """Set the time of a file just added (an S3 client copying it onto itself with new metadata); False when it - isn't one.""" + """Set the time of a file just added (an S3 client copying it onto itself with new metadata, an SMB client + setting it); False when it isn't one. A file kept aside is imported with it.""" parts = filetree.Tree.split(path) if not parts or _seconds(modified) is None: return False @@ -405,6 +405,8 @@ def touch(self, path, modified): if it is None: return False it.mtime = str(modified) + if it.kind == "pending": + it.modified = _iso(_seconds(modified)) return True def put(self, path, src, modified=None, md5=None): @@ -460,11 +462,17 @@ def delete(self, path): _remove(it.path) def move(self, src, dst): - """Rename a file kept aside: to an ordinary name, it's imported then. Everything else is Refused.""" + """Rename a file kept aside: to an ordinary name, it's imported then (once it's closed, when it's still being + written: finish). Everything else is Refused.""" it = self.pending(src) if it is None or not it.path: raise Refused("files can be added here, not moved or renamed") parts = filetree.Tree.split(src) + if it.holds: + where = self.check(dst) + REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) + REGISTRY.put(self.account["id"], where.folder, where.name, it) + return {"kind": "pending"} moving = spool(self.cfg) os.replace(it.path, moving) REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) @@ -475,3 +483,55 @@ def move(self, src, dst): os.replace(moving, it.path) REGISTRY.put(self.account["id"], tuple(parts[:-1]), parts[-1], it) raise + + # ---------- written in place (SMB) ---------- + def begin(self, path): + """A file a client makes at `path` and writes in place (SMB), its size not known yet: checked as check() + checks it, then a new, empty file on the uploads disk (returned) that shows under its name, kept aside and + held, until finish().""" + where = self.check(path) + src = spool(self.cfg) + src.touch(mode=0o600) + item = Item("pending", time.monotonic() + PENDING_SECONDS, path=str(src), modified=_iso(), holds=1) + REGISTRY.put(self.account["id"], where.folder, where.name, item) + return src + + def reopen(self, path): + """A file kept aside at `path`, opened to write again: its file on the uploads disk, held until finish().""" + it = self.pending(path) + if it is None or not it.path: + raise Refused("files can be added here, not changed") + parts = filetree.Tree.split(path) + REGISTRY.hold(self.account["id"], tuple(parts[:-1]), parts[-1], 1) + return pathlib.Path(it.path) + + def _held(self, path, src): + it = self.pending(path) + if it is None or it.path != str(src): + return None # deleted, or written over by another file, meanwhile + parts = filetree.Tree.split(path) + REGISTRY.hold(self.account["id"], tuple(parts[:-1]), parts[-1], -1) + return it + + def finish(self, path, src): + """A file written in place (begin, reopen) at `path` was closed: it's imported now when it's an ordinary file + with something in it and nobody else has it open; else it stays kept aside. put()'s answer, or None when it + was deleted or written over meanwhile.""" + it = self._held(path, src) + parts = filetree.Tree.split(path) + if it is None: + return None + if it.holds or junk(parts[-1]) or os.path.getsize(src) == 0: + return {"kind": "pending"} + REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) # put() files it again + try: + return self.put(path, src, it.mtime) + except BaseException: + if os.path.exists(src): # kept aside again: the client can delete it, or rename it and try again + REGISTRY.put(self.account["id"], tuple(parts[:-1]), parts[-1], it) + raise + + def abandon(self, path, src): + """A file written in place that won't be finished (its client went away, or deleted it as it closed it): it + stays kept aside, not imported, until it expires or is deleted.""" + self._held(path, src) diff --git a/fastapi_backend/app/domain/smb_share.py b/fastapi_backend/app/domain/smb_share.py index 4306b515e..c3a57795b 100644 --- a/fastapi_backend/app/domain/smb_share.py +++ b/fastapi_backend/app/domain/smb_share.py @@ -1,8 +1,10 @@ -"""Lens as a read-only SMB share: Finder, Explorer and Linux mount the file tree of the graph like any network drive. +"""Lens as an SMB share: Finder, Explorer and Linux mount the file tree of the graph like any network drive. What the share holds is the shared file tree (filetree.py, docs/file-tree.md): graph nodes are folders, nested by their links, with the -assets as files at the leaves. Each person sees what they may see in Lens, nothing more, and nobody can write: every -call that would change something is refused before it reaches the tree. +assets as files at the leaves. Each person sees what they may see in Lens, nothing more. While `smb.add_files` is on, +a file someone copies in is imported where they put it (share_import.py: the same checks and permissions as an +upload), and a new folder under Collections is a new collection; nothing Lens made can be changed or deleted. Off, +every call that would change something is refused before it reaches the tree. The server is impacket's SMB server (Apache-style license, pure Python), run inside the worker while Settings › File sharing › SMB is on. impacket serves folders on disk through `os`; here its module-level `os` is a small stand-in @@ -22,6 +24,7 @@ from __future__ import annotations +import contextlib import errno import ipaddress import itertools @@ -119,8 +122,35 @@ def nt_hash(secret: str) -> bytes: # ---------- the file system impacket sees ---------- +@dataclass +class _Writing: + """A file open to write: whose, where in the tree (`rel`), its real file on the uploads disk (`src`) and fd.""" + + account: str + rel: str + src: object + real: int + adder: object + failed: bool = False # a write was refused (too large, the disk full): it's dropped as it's closed + + +def _refused(e, p): + """share_import's refusals as the OSError impacket turns into an SMB status.""" + from . import filetree, share_import + + if isinstance(e, filetree.NotFound): + return FileNotFoundError(errno.ENOENT, "no such folder", p) + if isinstance(e, share_import.TooLarge): + return OSError(errno.EFBIG, str(e), p) + if isinstance(e, share_import.Refused | share_import.Rejected): + return PermissionError(errno.EACCES, str(e), p) + return e + + class VirtualOS(types.ModuleType): - """Just enough of `os` for impacket.smbserver, answered from the tree: read-only, and nothing outside ROOT. + """Just enough of `os` for impacket.smbserver, answered from the tree, and nothing outside ROOT. Files people add + are written to real files on the uploads disk while they're open, and imported when they're closed + (share_import.Adder: begin, finish). Paths are ROOT//; the key says whose view of the tree it is (and is checked again as it ages).""" @@ -132,6 +162,8 @@ def __init__(self, tree, keys: Keys): self.getpid, self.urandom, self.fspath, self.environ = _os.getpid, _os.urandom, _os.fspath, {} self.tree, self.keys = tree, keys self.fds: dict[int, list] = {} + self.writing: dict[int, _Writing] = {} + self.dropping: set[int] = set() # fds closed with delete-on-close: not imported self.lock = threading.Lock() self.next_fd = itertools.count(1 << 20) self.path = _Path(self) @@ -169,9 +201,49 @@ def listdir(self, p): raise NotADirectoryError(errno.ENOTDIR, "not a folder", p) return [c.name for c in self.tree.children(account, rel)] + def _adder(self, account, p): + adder = self.tree.adder(account) + if adder is None: + raise PermissionError(errno.EROFS, "the share is read-only", p) + return adder + + def _open_to_write(self, p, flags): + """An fd writing a file someone adds, or None when the file is opened to read after all (a folder, or a file + that's there opened without changing it).""" + from . import filetree, share_import + + account, rel = self._where(p) + e = self.tree.entry(account, rel) + if e is not None and e.is_dir: + return None + if e is None and not flags & _os.O_CREAT: + raise FileNotFoundError(errno.ENOENT, "no such file or folder", p) + if e is not None and not flags & (_os.O_WRONLY | _os.O_RDWR | _os.O_TRUNC | _os.O_APPEND): + return None + adder = self._adder(account, p) + added = getattr(e.source, "added", None) if e is not None else None + try: + if added == "pending": + src = adder.reopen(rel) + real = _os.open(src, _os.O_RDWR | (flags & _os.O_TRUNC)) + elif e is None or flags & _os.O_TRUNC: + src = adder.begin(rel) # written over: a new file (Refused for what Lens made) + real = _os.open(src, _os.O_RDWR) + else: + return None # opened to change in place: it reads, and writing to it is refused + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as x: + raise _refused(x, p) from None + with self.lock: + fd = next(self.next_fd) + self.fds[fd] = [None, e or Entry(posixpath.basename(rel), False), 0] + self.writing[fd] = _Writing(account, rel, src, real, adder) + return fd + def open(self, p, flags, mode=0o777, **k): if flags & _WRITE: - raise PermissionError(errno.EROFS, "the share is read-only", p) + fd = self._open_to_write(p, flags) + if fd is not None: + return fd account, rel, e = self._entry(p) if account is None: raise PermissionError(errno.EACCES, "not a share", p) @@ -188,11 +260,17 @@ def _fd(self, fd): return f def lseek(self, fd, pos, how): + w = self.writing.get(fd) + if w is not None: + return _os.lseek(w.real, pos, how) f = self._fd(fd) f[2] = max(0, pos if how == _os.SEEK_SET else (f[2] + pos if how == _os.SEEK_CUR else int(f[1].size or 0) + pos)) return f[2] def read(self, fd, n): + w = self.writing.get(fd) + if w is not None: + return _os.read(w.real, int(n)) f = self._fd(fd) if f[0] is None: raise IsADirectoryError(errno.EISDIR, "is a folder") @@ -205,19 +283,125 @@ def read(self, fd, n): f[2] += len(data) return data + def write(self, fd, data): + w = self.writing.get(fd) + if w is None: + raise PermissionError(errno.EROFS, "only files added here can be written") + most = int(w.adder.cfg["uploads"]["max_mb"]) + try: + if _os.lseek(w.real, 0, _os.SEEK_CUR) + len(data) > most * (1 << 20): + raise OSError(errno.EFBIG, f"files up to {most} MB") + return _os.write(w.real, data) + except OSError: + w.failed = True + raise + def close(self, fd): + from . import filetree, share_import + with self.lock: f = self.fds.pop(fd, None) + w = self.writing.pop(fd, None) + dropped = fd in self.dropping + self.dropping.discard(fd) if f and f[0] is not None: f[0].close() + if w is None: + return + _os.close(w.real) + if dropped: + return w.adder.abandon(w.rel, w.src) + if w.failed: # part of a file: not imported, and not kept + w.adder.abandon(w.rel, w.src) + with contextlib.suppress(Exception): + w.adder.delete(w.rel) + return None + try: + w.adder.finish(w.rel, w.src) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as x: + raise _refused(x, w.rel) from None + finally: + self.tree.drop(w.account) + + def drop(self, fd): + """The client deletes this file as it closes it: don't import it then.""" + with self.lock: + if fd in self.writing: + self.dropping.add(fd) + + def abandon_all(self, fds): + """A connection ended with these files still open: close them, and import none of them.""" + for fd in fds: + self.drop(fd) + try: + self.close(fd) + except OSError: + pass def fsync(self, fd): return None + def mkdir(self, p, mode=0o777): + from . import filetree, share_import + + account, rel = self._where(p) + adder = self._adder(account, p) + e = self.tree.entry(account, rel) + if e is not None and e.is_dir: + return None # impacket makes the folder of any create that may make one, even when it's there + try: + made = adder.folder(rel) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as x: + raise _refused(x, p) from None + finally: + self.tree.drop(account) + if not made: + raise FileExistsError(errno.EEXIST, "there's a folder there already", p) + return None + + def remove(self, p): + from . import share_import + + account, rel = self._where(p) + try: + self._adder(account, p).delete(rel) + except share_import.Refused as x: + raise _refused(x, p) from None + + unlink = remove + + def rename(self, src, dst): + from . import filetree, share_import + + account, rel = self._where(src) + to_account, to = self._where(dst) + if to_account != account: + raise PermissionError(errno.EXDEV, "another person's view", dst) + try: + self._adder(account, src).move(rel, to) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as x: + raise _refused(x, src) from None + finally: + self.tree.drop(account) + with self.lock: + for w in self.writing.values(): + if w.account == account and w.rel == rel: + w.rel = to + + replace = rename + + def utime(self, p, times=None, **k): + """impacket passes (LastWriteTime, ChangeTime): the first is the file's own time.""" + account, rel = self._where(p) + adder = self._adder(account, p) + when = times[0] if times else time.time() + if not adder.touch(rel, when): + raise PermissionError(errno.EACCES, "only files just added can have their time set", p) + def _read_only(self, *a, **k): raise PermissionError(errno.EROFS, "the share is read-only") - write = remove = unlink = mkdir = makedirs = rmdir = rename = replace = utime = chmod = _read_only + makedirs = rmdir = chmod = _read_only def setreuid(self, *a): return None @@ -278,20 +462,40 @@ def __init__(self, db, cfg_fn): self.trees: dict[str, tuple[float, object]] = {} self.lock = threading.Lock() - def _tree(self, account): + def _view(self, account): + """(their Tree, their account row).""" from . import auth, filetree with self.lock: hit = self.trees.get(account) if hit and time.monotonic() - hit[0] < TREE_SECONDS: - return hit[1] + return hit[1], hit[2] row = auth.get_account(self.db, account) if not row or row.get("disabled"): raise PermissionError(errno.EACCES, "no such account") t = filetree.Tree(self.db, self.cfg_fn(), filetree.Viewer.of(self.db, row)) with self.lock: - self.trees[account] = (time.monotonic(), t) - return t + self.trees[account] = (time.monotonic(), t, row) + return t, row + + def _tree(self, account): + return self._view(account)[0] + + def adder(self, account): + """Someone adding files (share_import.Adder, with their view of the tree), or None while the share doesn't + take files (smb.add_files).""" + from . import share_import + + cfg = self.cfg_fn() + if not share_import.accepting(cfg, "smb"): + return None + tree, row = self._view(account) + return share_import.Adder(self.db, cfg, row, "smb", tree) + + def drop(self, account): + """Their view of the tree afresh from the next call (after they added a file or made a folder).""" + with self.lock: + self.trees.pop(account, None) @staticmethod def _entry(e): @@ -406,7 +610,7 @@ def search_share(conn_id, share, server): class Server: """One SMB server on (bind, port), serving `tree` to holders of file access keys, in its own threads.""" - def __init__(self, bind, port, tree, keys: Keys, networks=None, name="Lens", log=None): + def __init__(self, bind, port, tree, keys: Keys, networks=None, name="Lens", log=None, add_files=False): smbserver = _impacket() self.log = log or (lambda *a: None) self.networks = list(networks if networks is not None else store.DEFAULTS["smb"]["allowed_networks"]) @@ -451,6 +655,15 @@ def processRequest(self, conn_id, data): # noqa: N802 raise ConnectionAbortedError("SMB1 is not served") return super().processRequest(conn_id, data) + def removeConnection(self, name): # noqa: N802 + """A connection ended: files it still had open to write aren't imported (they're incomplete).""" + try: + opened = self.getConnectionData(name, False).get("OpenedFiles") or {} + vos.abandon_all([f.get("FileHandle") for f in opened.values() if f.get("FileHandle") in vos.writing]) + except Exception: # noqa: BLE001 - it was never set up + pass + super().removeConnection(name) + creds = Credentials(keys) cfg = smbserver.configparser.ConfigParser() cfg.add_section("global") @@ -465,13 +678,15 @@ def processRequest(self, conn_id, data): # noqa: N802 "SMB2Support": "True", }.items(): cfg.set("global", k, v) - for share, path, comment, kind in (("IPC$", "", "", "3"), (SHARE, ROOT, "Lens: the graph as files (read-only)", "0")): + comment = "Lens: the graph as files" + ("; files added are imported" if add_files else " (read-only)") + for share, path, note, kind, ro in (("IPC$", "", "", "3", "yes"), (SHARE, ROOT, comment, "0", "no" if add_files else "yes")): cfg.add_section(share) - for k, v in {"comment": comment, "read only": "yes", "share type": kind, "path": path}.items(): + for k, v in {"comment": note, "read only": ro, "share type": kind, "path": path}.items(): cfg.set(share, k, v) self.server = LensSMB((bind, int(port)), config_parser=cfg, ipv6=":" in bind) self.server.keys = keys self.server.processConfigFile() + plain_close = self.server.hookSmb2Command(smbserver.smb2.SMB2_CLOSE, lambda c, s, r: _closing(vos, plain_close, c, s, r)) self.port = self.server.server_address[1] self.extras = [smbserver.SRVSServer(), smbserver.WKSTServer()] # share lists for Explorer and Finder for x in self.extras: @@ -509,6 +724,23 @@ def stop(self): pass +def _closing(vos, plain, conn_id, server, packet): + """SMB2 CLOSE: a file marked delete-on-close is deleted as it's closed, so it isn't imported first.""" + try: + from impacket import smb3structs as smb2 + + conn = server.getConnectionData(conn_id, False) + fid = smb2.SMB2Close(packet["Data"])["FileID"].getData() + if fid == b"\xff" * 16: + fid = conn["LastRequest"]["SMB2_CREATE"]["FileID"] + f = conn["OpenedFiles"].get(fid) or {} + if f.get("DeleteOnClose"): + vos.drop(f.get("FileHandle")) + except Exception: # noqa: BLE001 - impacket answers a malformed close itself + pass + return plain(conn_id, server, packet) + + # ---------- following the settings ---------- def conf(cfg): return {**store.DEFAULTS["smb"], **(cfg.get("smb") or {})} @@ -534,7 +766,7 @@ def __init__(self, db, cfg_fn, name=None, log=None, tree=None, keys=None): def apply(self): s = conf(self.cfg_fn()) - want = (s["bind"], int(s["port"])) if s.get("enabled") else None + want = (s["bind"], int(s["port"]), bool(s.get("add_files"))) if s.get("enabled") else None if self.server is not None: self.server.networks = list(s.get("allowed_networks") or []) if want == self.want and (self.server is not None or want is None or time.monotonic() < self.retry): @@ -546,8 +778,8 @@ def apply(self): try: self.tree = self.tree or FileTree(self.db, self.cfg_fn) self.keys = self.keys or _keys(self.db, self.cfg_fn) - self.server = Server(want[0], want[1], self.tree, self.keys, s.get("allowed_networks"), log=self.log).start() - self.log(f"smb: serving the file tree on port {self.server.port} (read-only)") + self.server = Server(want[0], want[1], self.tree, self.keys, s.get("allowed_networks"), log=self.log, add_files=want[2]).start() + self.log(f"smb: serving the file tree on port {self.server.port}" + ("" if want[2] else " (read-only)")) except (OSError, ImportError) as e: self.error = ( f"port {want[1]}: {e.strerror or e}" if isinstance(e, OSError) else "impacket isn't installed (pip install impacket)" @@ -625,5 +857,6 @@ def status(db, cfg): "process": running[0].get("process") if running else None, "installed": installed, "address": f"smb://{host}{'' if port == 445 else f':{port}'}/{SHARE}", - "mount": f"sudo mount -t cifs -o port={port},username=,ro,vers=2.0 //{host}/{SHARE} /mnt/lens", + "mount": f"sudo mount -t cifs -o port={port},username=,{'' if s.get('add_files') else 'ro,'}vers=2.0 //{host}/{SHARE} /mnt/lens", + "add_files": bool(s.get("add_files")), } diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index d6766befe..960d44cd5 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -545,6 +545,7 @@ def labels(): "bind": os.environ.get("LENS_SMB_LISTEN") or "127.0.0.1", "port": int(os.environ.get("LENS_SMB_LISTEN_PORT") or 4445), "allowed_networks": ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8", "100.64.0.0/10", "fc00::/7", "::1/128"], + "add_files": True, # files people copy in are imported (share_import.py); off: read-only }, # the assistant's code sandbox (sandbox.py, docs/assistant.md#code-sandbox): off until an admin turns it on. Then # each conversation's Python and bash run in a sandbox of its own on the OpenShell gateway at `gateway`, made from diff --git a/fastapi_backend/app/schemas/admin.py b/fastapi_backend/app/schemas/admin.py index 7afaac2bb..9664ffb23 100644 --- a/fastapi_backend/app/schemas/admin.py +++ b/fastapi_backend/app/schemas/admin.py @@ -58,6 +58,7 @@ class SmbStatus(ResponseModel): installed: bool = Field(description="whether the SMB server (impacket) is installed here; it's fetched when the share is turned on") address: str = Field(description="the address Finder and Linux file managers open (the machine's name filled in by the caller)") mount: str = Field(description="the command that mounts it on Linux") + add_files: bool = Field(False, description="whether files people copy in are imported (smb.add_files); off: read-only") class SlackStatus(ResponseModel): diff --git a/fastapi_backend/tests/domain/test_smb_share.py b/fastapi_backend/tests/domain/test_smb_share.py index 9c4802a0c..55325d198 100644 --- a/fastapi_backend/tests/domain/test_smb_share.py +++ b/fastapi_backend/tests/domain/test_smb_share.py @@ -41,6 +41,12 @@ def children(self, account, path): names = {p[len(pre) :].split("/")[0] for p in [*self.FILES[account], *self._dirs(account)] if p.startswith(pre) and p[len(pre) :]} return [self.entry(account, pre + n) for n in sorted(names)] + def adder(self, account): + return None # files can't be added (smb.add_files off) + + def drop(self, account): + pass + def open(self, account, e): data = self.FILES[account][e.source] return smb_share.Reader(lambda start, end: (data[i : min(i + 1000, end + 1)] for i in range(start, end + 1, 1000)), len(data)) diff --git a/fastapi_backend/tests/domain/test_smb_writes.py b/fastapi_backend/tests/domain/test_smb_writes.py new file mode 100644 index 000000000..822cd0396 --- /dev/null +++ b/fastapi_backend/tests/domain/test_smb_writes.py @@ -0,0 +1,175 @@ +"""Adding files over SMB (docs/smb.md, docs/file-tree.md#adding-files), end to end with impacket's own SMB client and +the real file tree: a file copied in is imported where it's put, as an upload is; what may not be added is refused.""" + +from __future__ import annotations + +import io +import time + +import pytest + +pytest.importorskip("impacket") + +from impacket.smb3structs import FILE_DELETE_ON_CLOSE, FILE_NON_DIRECTORY_FILE, FILE_OVERWRITE_IF, GENERIC_ALL # noqa: E402 +from impacket.smbconnection import SessionError, SMBConnection # noqa: E402 + +from app.domain import hierarchy, share_import, smb_share, store # noqa: E402 +from tests.helpers import make_user, seed, text_pdf # noqa: E402 + +PDF = text_pdf([["Harbour minutes", "The lamp was lit at dusk."]]) +S = smb_share.SHARE + + +@pytest.fixture +def env(db, cfg, folder): + a, _, _ = seed(db, cfg, folder) + pods = store.ns_id(db, "pods") + talks = hierarchy.create(db, pods, "Talks") + users = { + "lked": make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}), + "lkvi": make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}), + } + keys = smb_share.Keys(lambda kid: {"account": users[kid], "secret": f"{kid}-secret"} if kid in users else None) + conf = {"cfg": {**cfg, "smb": {**cfg.get("smb", {}), "add_files": True}}} + share_import.REGISTRY.items.clear() + s = smb_share.Server("127.0.0.1", 0, smb_share.FileTree(db, lambda: conf["cfg"]), keys, ["127.0.0.0/8"], add_files=True).start() + yield {"port": s.port, "pods": pods, "talks": talks, "asset": a, "conf": conf, "server": s} + s.stop() + + +def _connect(env, who="lked"): + c = SMBConnection("127.0.0.1", "127.0.0.1", sess_port=env["port"]) + c.login(who, f"{who}-secret") + return c + + +def _names(c, path): + return sorted(f.get_longname() for f in c.listPath(S, path + "\\*") if f.get_longname() not in (".", "..")) + + +def _recording(db, title): + return db.one("SELECT record::id(id) AS id, collection, source, size FROM recording WHERE title = $t", t=title) + + +def test_a_file_copied_into_a_collection_is_imported_there(db, env): + c = _connect(env) + c.putFile(S, "pods\\Collections\\Talks\\minutes.pdf", io.BytesIO(PDF).read) + rec = _recording(db, "minutes") + assert rec["collection"] == env["talks"] and rec["source"] == "document" and rec["size"] == len(PDF) + assert db.one("SELECT VALUE id FROM job WHERE recording = $r LIMIT 1", r=rec["id"]) # the namespace's pipeline + audit = db.one("SELECT * FROM audit_log WHERE action = 'share.import'") + assert audit["detail"]["via"] == "smb" and audit["detail"]["path"] == "pods/Collections/Talks/minutes.pdf" + # the client finds it under the name it gave, beside the asset's own files, and reads it back + names = _names(c, "pods\\Collections\\Talks") + assert "minutes.pdf" in names and f"minutes [{rec['id']}].pdf" in names + out = io.BytesIO() + c.getFile(S, "pods\\Collections\\Talks\\minutes.pdf", out.write) + assert out.getvalue() == PDF + # only for them: others see the asset as Lens names it + assert "minutes.pdf" not in _names(_connect(env, "lkvi"), "pods\\Collections\\Talks") + # the same file again (Finder's "Replace") is the same asset + c.putFile(S, "pods\\Collections\\Talks\\minutes.pdf", io.BytesIO(PDF).read) + assert db.values("SELECT VALUE record::id(id) FROM recording WHERE title = 'minutes'") == [rec["id"]] + # a file at the top of the namespace goes to its default collection + c.putFile(S, "pods\\notes.pdf", io.BytesIO(PDF + b"\n%2").read) + assert _recording(db, "notes")["collection"] == store.default_collection(db, env["pods"]) + + +def test_new_folders_under_collections_are_new_collections(db, env): + c = _connect(env) + c.createDirectory(S, "pods\\Collections\\Harbour") + c.createDirectory(S, "pods\\Collections\\Harbour\\2026") + assert "Harbour" in _names(c, "pods\\Collections") and _names(c, "pods\\Collections\\Harbour") == ["2026"] + c.putFile(S, "pods\\Collections\\Harbour\\2026\\minutes.pdf", io.BytesIO(PDF).read) + rec = _recording(db, "minutes") + made = db.one("SELECT record::id(id) AS id, parent FROM collection WHERE name = '2026'") + harbour = db.one("SELECT VALUE record::id(id) FROM collection WHERE name = 'Harbour'") + assert rec["collection"] == made["id"] and made["parent"] == harbour + with pytest.raises(SessionError): # it's there already + c.createDirectory(S, "pods\\Collections\\Harbour") + with pytest.raises(SessionError): # folders go under Collections only + c.createDirectory(S, "pods\\Topics\\New topic") + assert db.values("SELECT VALUE action FROM audit_log WHERE action = 'collection.create'") == ["collection.create"] * 2 + + +def test_an_apps_own_files_and_temporary_names_are_kept_aside(db, env): + c = _connect(env) + c.putFile(S, "pods\\Collections\\Talks\\.DS_Store", io.BytesIO(b"\0\0\0\1Bud1").read) + c.putFile(S, "pods\\Collections\\Talks\\empty.pdf", io.BytesIO(b"").read) + c.putFile(S, "pods\\Collections\\Talks\\draft.tmp", io.BytesIO(PDF).read) + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + names = _names(c, "pods\\Collections\\Talks") + assert {".DS_Store", "empty.pdf", "draft.tmp"} <= set(names) + c.deleteFile(S, "pods\\Collections\\Talks\\.DS_Store") + assert ".DS_Store" not in _names(c, "pods\\Collections\\Talks") + # renamed to an ordinary name, it's imported (an app saving under a temporary name) + c.rename(S, "pods\\Collections\\Talks\\draft.tmp", "pods\\Collections\\Talks\\draft.pdf") + assert _recording(db, "draft")["collection"] == env["talks"] + # a file deleted as it's closed is never imported + tid = c.connectTree(S) + fid = c.createFile(tid, "pods\\Collections\\Talks\\scratch.pdf", GENERIC_ALL, creationOption=FILE_NON_DIRECTORY_FILE | FILE_DELETE_ON_CLOSE, + creationDisposition=FILE_OVERWRITE_IF) # fmt: skip + c.writeFile(tid, fid, PDF) + c.closeFile(tid, fid) + assert not _recording(db, "scratch") and "scratch.pdf" not in _names(c, "pods\\Collections\\Talks") + # nor is a file still open when its client goes away + other = _connect(env) + tid = other.connectTree(S) + fid = other.createFile(tid, "pods\\Collections\\Talks\\half.pdf") + other.writeFile(tid, fid, PDF[:100]) + other.getSMBServer().get_socket().close() + for _ in range(50): # the server notices in its own time + held = [i.holds for names in share_import.REGISTRY.items.values() for n, i in names.items() if n == "half.pdf"] + if held == [0]: + break + time.sleep(0.05) + assert held == [0] and not _recording(db, "half") + + +def test_what_may_not_be_added_is_refused(db, env): + c = _connect(env) + with pytest.raises(SessionError): # a viewer + _connect(env, "lkvi").putFile(S, "pods\\Collections\\Talks\\minutes.pdf", io.BytesIO(PDF).read) + with pytest.raises(SessionError): # a type Lens doesn't take: refused as it's made + c.putFile(S, "pods\\Collections\\Talks\\tool.exe", io.BytesIO(b"MZ").read) + asset = f"pods\\Collections\\General\\ep1 [{env['asset']}].txt" + with pytest.raises(SessionError): # what Lens made can't be written over, deleted or renamed + c.putFile(S, asset, io.BytesIO(b"x").read) + with pytest.raises(SessionError): + c.deleteFile(S, asset) + with pytest.raises(SessionError): + c.rename(S, asset, "pods\\Collections\\General\\mine.txt") + with pytest.raises(SessionError): # namespaces are made in the web app + c.createDirectory(S, "new-namespace") + env["conf"]["cfg"] = {**env["conf"]["cfg"], "uploads": {**env["conf"]["cfg"]["uploads"], "max_mb": 1}} + with pytest.raises(SessionError): # over uploads.max_mb: cut off as it's written + c.putFile(S, "pods\\Collections\\Talks\\big.pdf", io.BytesIO(PDF + b"\0" * (2 << 20)).read) + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + # off: read-only again, whatever the share says + env["conf"]["cfg"] = {**env["conf"]["cfg"], "smb": {**env["conf"]["cfg"]["smb"], "add_files": False}} + with pytest.raises(SessionError): + c.putFile(S, "pods\\Collections\\Talks\\minutes.pdf", io.BytesIO(PDF).read) + assert not _recording(db, "minutes") + + +def test_the_share_is_read_only_while_add_files_is_off(db, cfg): + live = {**cfg, "smb": {**cfg.get("smb", {}), "enabled": True, "bind": "127.0.0.1", "port": 0, "add_files": False}} + r = smb_share.Runner(db, lambda: live, "w1") + r.apply() + try: + assert r.server is not None and smb_share.status(db, live)["add_files"] is False + first = r.server + live["smb"]["add_files"] = True # turning it on restarts the server with a writable share + r.apply() + assert r.server is not first and smb_share.status(db, live)["add_files"] is True + assert ",ro," not in smb_share.status(db, live)["mount"] + finally: + r.close() + + +def test_smb_add_files_is_checked(): + from app.domain import settings + + assert settings._smb_setting("add_files", False) is False + with pytest.raises(ValueError): + settings._smb_setting("add_files", "no") diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index f3443449f..fb7f35103 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -20667,6 +20667,12 @@ export type SmbStatus = { * the command that mounts it on Linux */ mount: string; + /** + * Add Files + * + * whether files people copy in are imported (smb.add_files); off: read-only + */ + add_files?: boolean; [key: string]: unknown; }; diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index b9419a77b..eae6daec0 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -348,7 +348,7 @@ export const SECTIONS: SectionSpec[] = [ label: "SMB share", backend: ["smb"], description: - "Off unless you turn it on. The graph as folders and files over SMB, read-only, for Finder, Explorer and Linux: each person signs in with a file access key and sees what they see in the web app.", + "Off unless you turn it on. The graph as folders and files over SMB, for Finder, Explorer and Linux: each person signs in with a file access key, sees what they see in the web app, and files they copy into a collection’s folder are imported there.", }, { id: "shares", @@ -2642,7 +2642,14 @@ export const FIELDS: FieldSpec[] = [ key: "enabled", label: "Share the file tree over SMB", kind: "switch", - hint: "Read-only. Off: nothing listens", + hint: "Off: nothing listens", + }, + { + section: "smb", + key: "add_files", + label: "Import files people add", + kind: "switch", + hint: "A file copied into a collection’s folder is imported there, as an upload; a new folder under Collections is a new collection. Off: read-only", }, { section: "smb", diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 328e0ce6a..df3340465 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -66703,6 +66703,12 @@ "type": "string", "title": "Mount", "description": "the command that mounts it on Linux" + }, + "add_files": { + "type": "boolean", + "title": "Add Files", + "description": "whether files people copy in are imported (smb.add_files); off: read-only", + "default": false } }, "additionalProperties": true, diff --git a/threat-model.json b/threat-model.json index 8375d6f6f..89dcce9b8 100644 --- a/threat-model.json +++ b/threat-model.json @@ -1242,9 +1242,9 @@ }, { "property": "P-SMB", - "violation_symptom": "info-leak", + "violation_symptom": "info-leak, integrity-bypass", "severity_tier": "security", - "conditions": "the SMB share is off until an admin turns it on; read-only; per-person file access keys and views; no guests or SMB1; a random NTLM challenge per connection; allowed networks only", + "conditions": "the SMB share is off until an admin turns it on; files added are imported only where the key's owner may import, checked as uploads, never over what Lens made (smb.add_files off: read-only); per-person file access keys and views; no guests or SMB1; a random NTLM challenge per connection; allowed networks only", "provenance": "documented", "source": "docs/smb.md \"What keeps it safe\"" }, diff --git a/threat-model.md b/threat-model.md index c933291a5..5382237e7 100644 --- a/threat-model.md +++ b/threat-model.md @@ -303,7 +303,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /automations/hooks/{token}` (an automation's webhook) | body (JSON, a form's text fields or text), query, Standard Webhooks headers, token in the path | yes, anyone given the address | data, size/rate, x-credential, x-prompt-injection | no hook exists until an automation with a webhook trigger is saved; the token is 256 random bits kept only as a hash, and revoked with its trigger; 64 KB and 60 calls a minute per hook; a signature, when sent (or required by a `signed` hook), must match the hook's sealed secret within five minutes; a retried message id is logged once; what was sent is outside text (`lens.untrusted`), wrapped as data in prompts | *(documented, docs/automations.md "No webhook exists until an automation with such a trigger is saved")* | | automations' params and prompts, `POST /automations/preview` and `/automations/variables` | template text, a draft automation, an activity id | yes, anyone signed in (preview, the picker); the namespace editors who manage an automation (its params) | callback/code (declarative), data, x-prompt-injection | rendered in the Jinja sandbox (2 MB of output, `*` and `**` guarded; Q23); previews read only activities, namespaces and automations the person may see; prompts mark outside text as data | *(documented, docs/automations.md "The variable picker and preview")* | | NFS share (port 12049 on the host, 2049 in the worker) | NFS and MOUNT RPCs, client address | yes, once enabled, from `nfs.allowed_networks` | data, x-network-source | admin enables and picks the account; operator firewalls | *(documented, docs/file-shares.md "NFS has no login")* | -| SMB share (port 1445 on the host, 4445 in the worker) | SMB2 requests, NTLM sign-in with a file access key, client address | yes, once enabled, from `smb.allowed_networks` | data, x-credential, x-network-source | admin enables; operator firewalls | *(documented, docs/smb.md "What keeps it safe")* | +| SMB share (port 1445 on the host, 4445 in the worker) | SMB2 requests, NTLM sign-in with a file access key, client address | yes, once enabled, from `smb.allowed_networks` | data, x-credential, x-network-source | admin enables; operator firewalls; files added are imported only where that key's owner may import (`smb.add_files` off: read-only), written to the uploads folder and cut off at the upload size limit | *(documented, docs/smb.md "What keeps it safe")* | | S3 9555 and WebDAV 9556 (file shares) | signed S3 requests (SigV4 header or presigned link); WebDAV with an access key as Basic sign-in | yes, once enabled | x-credential, resource-name | admin enables; each request carries a person's own access key; files added are imported only where that person may import (`shares.add_files` off: read-only), checked as uploads, size-capped while streamed, never over Lens's own files; failed WebDAV sign-ins throttled per address | *(documented, docs/file-tree.md "Off until an admin turns it on")* | | web app `/s3` and `/dav` (file shares relayed by `lens-relay.js`, ahead of Next) | any method, path, `Host` and body passed to the share servers with `X-Forwarded-Prefix` and `X-Forwarded-For` | yes, once `shares.enabled` and `shares.remote`; from the internet through the tunnel or HTTPS | x-credential, resource-name, x-network-source | admin enables both; the same per-person access keys, files added imported as at the ports (`shares.add_files`); failed WebDAV sign-ins throttled per visitor from `X-Forwarded-For` as trusted proxies report it (`server.trusted_proxies`, `LENS_TRUSTED_PROXY_HOSTS`, loopback); `X-Forwarded-Prefix` sent straight to a share port only gets the sender refused or signed and linked under that path | *(documented, docs/file-tree.md "At the web app's address")* | | `.env`, `archive.yaml`, `LENS_*`, `ARCHIVE_SECRET_KEY`, `RCLONE_BINARY` | process configuration | no (operator) | x-env-config | keep unreadable to others | *(maintainer, 2026-10)* | @@ -416,7 +416,7 @@ Every row is claimed, disclaimed, N/A with a reason, or unresolved. Family short | Optional | Callback / collaborator execution | claimed | tunnel token and llama-server key stay off command lines | P-CHILD-SECRETS | *(documented, docs/remote-access.md "in its environment, not on its command line")* | | Optional | Reference lifecycle | claimed | each service off until an admin turns it on | P-SENSORS | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | | Optional | Confidentiality | claimed | NFS share off until an admin turns it on; read-only; one account's view; allowed networks only | P-NFS | *(documented, docs/file-shares.md "NFS has no login")* | -| Optional | Confidentiality | claimed | SMB share off until an admin turns it on; read-only; each key sees its owner's view; allowed networks only | P-SMB | *(documented, docs/smb.md "What keeps it safe")* | +| Optional | Confidentiality | claimed | SMB share off until an admin turns it on; adds files only as its owner may import, never changes what Lens made (`smb.add_files` off: read-only); each key sees its owner's view; allowed networks only | P-SMB | *(documented, docs/smb.md "What keeps it safe")* | | Optional | Concurrency; resource cost | disclaimed | MQTT and syslog connection counts, one thread per connection | N-VOLUME | *(documented, SECURITY.md "denial of service by sheer volume")* | ## 1.8 Assumptions and guarantees about outputs @@ -546,7 +546,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-MODEL-CHECKSUM | Catalog GGUF models are pinned to a revision and checked against their SHA-256. | integrity-bypass | security-critical | *(documented, docs/local-models.md "checked against their SHA-256")* | Voided by: `hf:` references have no pin or hash — `domain/local_llm.py:138-139`. | | P-SENSORS | Sensors are off until an admin enables them. MQTT needs a hub login unless `mqtt_anonymous` is on. Syslog hears only listed networks. | integrity-bypass | security-critical | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | Voided by: `sensors.mqtt_anonymous` — `domain/sensors.py:984`; `sensors.syslog_networks`, `sensors.bind`. | | P-NFS | The NFS share is off until an admin turns it on. It is read-only, shows only what one chosen account sees in the web app, and turns away every client outside `nfs.allowed_networks` before rclone reads a byte. | info-leak | security-critical | *(documented, docs/file-shares.md "NFS has no login")* | Voided by: `nfs.allowed_networks` set wide, or a client address spoofed on the local network (NFS has no login) — `domain/share_relay.py`; Docker publishing that rewrites client addresses. | -| P-SMB | The SMB share is off until an admin turns it on. It is read-only, signs people in with their own file access keys and shows each only what they see in the web app, refuses guests and SMB1 sessions, gives every connection its own NTLM challenge, and closes connections from outside `smb.allowed_networks` before reading a byte. | info-leak | security-critical | *(documented, docs/smb.md "What keeps it safe")* | Voided by: `smb.allowed_networks` set wide; SMB 2.0 has no encryption on the wire, so a key's NTLM exchange and the files read can be watched on the network — `domain/smb_share.py`. | +| P-SMB | The SMB share is off until an admin turns it on. It adds files only where the key's owner may import, checked as uploads (type, size cut off while written), never writes over, renames or deletes what Lens made, and with `smb.add_files` off is read-only. It signs people in with their own file access keys and shows each only what they see in the web app, refuses guests and SMB1 sessions, gives every connection its own NTLM challenge, and closes connections from outside `smb.allowed_networks` before reading a byte. | info-leak, integrity-bypass | security-critical | *(documented, docs/smb.md "What keeps it safe")* | Voided by: `smb.allowed_networks` set wide; SMB 2.0 has no encryption on the wire, so a key's NTLM exchange and the files read or added can be watched (or files added altered) on the network — `domain/smb_share.py`, `domain/share_import.py`. | | P-SHARES | The file shares (S3 and WebDAV) are off until an admin enables them. Every request is signed (S3: AWS Signature Version 4) or signed in (WebDAV: Basic) with an access key someone made while signed in to the web app; it reads as them and sees only what the web app shows them. A file it writes is imported as an upload by that person would be (the import permission or an editor's role on the collection; making a collection as the web app allows; the upload type and size checks, with the body streamed to disk and cut off at the size limit), never replaces or deletes an asset or a file Lens made, and with `shares.add_files` off nothing is written. Keys expire as API keys do, stop with a disabled account, and are revoked at once. | info-leak, integrity-bypass | security-critical | *(documented, docs/file-tree.md "Off until an admin turns it on")* | Through the web app (`/s3`, `/dav`) only with `shares.remote` on as well. Voided by: `shares.enabled` with `LENS_BIND=0.0.0.0` sends files, S3 signatures and WebDAV's Basic secret in clear over plain HTTP on the network, as does `shares.remote` when the web app itself is reached over plain HTTP; key secrets are sealed with the server key (`secret.key`), not hashed, since SigV4 signs with the secret itself; a leaked key can add files as its owner can — `domain/s3.py`, `domain/webdav.py`, `domain/share_import.py`. | | P-CHILD-SECRETS | The tunnel token reaches cloudflared in its environment, not its command line; llama-server gets its own random API key. | info-leak | correctness-only | *(documented, docs/remote-access.md "in its environment, not on its command line")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-REMOTE-NAMES | An IMAP message is named by mailbox validity and uid, so a rebuilt mailbox never serves another message under an old name. | wrong-output | correctness-only | *(documented, docs/processing.md "an old one never reads another message")* | Voided by: none identified (not required for this symptom; no exhaustive search) | diff --git a/threat-model.yaml b/threat-model.yaml index 3fd9e4843..141a31f56 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:35bef40c8c965ae68f3e5d5f839c22c8528f5526f251c4e5384deb840575ca60" +prose_version: "threat-model.md@sha256:b1a8afaeb189d8ae7b051e70c7095ccb2eb1a9fdc22f51d5c0c2367a0cbba6f7" model_status: under-review triage_policy: strict confidence: {documented: 204, maintainer: 46, inferred: 91, assumption: 8} @@ -713,7 +713,7 @@ contract_dimensions: - {component: optional, dimension: callback-execution, status: claimed, property_id: P-CHILD-SECRETS, conditions: "tunnel token and llama-server key stay off command lines", provenance: {kind: documented, source: 'docs/remote-access.md "in its environment, not on its command line"'}} - {component: optional, dimension: reference-lifecycle, status: claimed, property_id: P-SENSORS, conditions: "each service off until an admin turns it on", provenance: {kind: documented, source: 'docs/sensors.md "Sensors are off until an admin turns them on"'}} - {component: optional, dimension: confidentiality, status: claimed, property_id: P-NFS, conditions: "NFS share off until an admin turns it on; read-only; one account's view; allowed networks only", provenance: {kind: documented, source: 'docs/file-shares.md "NFS has no login"'}} - - {component: optional, dimension: confidentiality, status: claimed, property_id: P-SMB, conditions: "SMB share off until an admin turns it on; read-only; each key sees its owner's view; allowed networks only", provenance: {kind: documented, source: 'docs/smb.md "What keeps it safe"'}} + - {component: optional, dimension: confidentiality, status: claimed, property_id: P-SMB, conditions: "SMB share off until an admin turns it on; adds files only as its owner may import, never changes what Lens made (smb.add_files off: read-only); each key sees its owner's view; allowed networks only", provenance: {kind: documented, source: 'docs/smb.md "What keeps it safe"'}} - {component: optional, dimension: concurrency-reentrancy, status: disclaimed, property_id: N-VOLUME, conditions: "MQTT and syslog connection counts, one thread per connection", provenance: {kind: documented, source: 'SECURITY.md "denial of service by sheer volume"'}} - {component: optional, dimension: resource-complexity, status: disclaimed, property_id: N-VOLUME, conditions: "MQTT and syslog connection counts, one thread per connection", provenance: {kind: documented, source: 'SECURITY.md "denial of service by sheer volume"'}} @@ -1683,9 +1683,9 @@ properties_claimed: kind: confidentiality components: [optional] tier: security-critical - conditions: "the SMB share is off until an admin turns it on; read-only; per-person file access keys and views; no guests or SMB1; a random NTLM challenge per connection; allowed networks only" - violation_symptoms: [info-leak] - x-voided-by: "smb.allowed_networks set wide; SMB 2.0 has no encryption on the wire (domain/smb_share.py)" + conditions: "the SMB share is off until an admin turns it on; files added are imported only where the key's owner may import, checked as uploads, never over what Lens made (smb.add_files off: read-only); per-person file access keys and views; no guests or SMB1; a random NTLM challenge per connection; allowed networks only" + violation_symptoms: [info-leak, integrity-bypass] + x-voided-by: "smb.allowed_networks set wide; SMB 2.0 has no encryption on the wire (domain/smb_share.py, domain/share_import.py)" provenance: {kind: documented, source: 'docs/smb.md "What keeps it safe"'} - id: P-SHARES kind: authentication