From ff828f91e8cf9b1b49ec13866567493d8f2f01ff Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Thu, 15 Oct 2026 11:34:36 +0000 Subject: [PATCH 1/4] feat(shares): import files added over S3 and WebDAV A file written into the file tree over S3 or WebDAV is imported as a web upload is (uploads.take): same type and size checks, the namespace's pipeline, fingerprint dedupe. Where it lands decides where it goes: a collection's folder, a new folder under Collections (a new collection), or the default collection. It needs the import permission or an editor's role on the collection, and never replaces a file Lens made. S3 takes PutObject (plain, chunked, aws-chunked with signed chunks and checksums), multipart uploads, folder markers, rclone's self-copy to set times, and deletes of files kept aside. WebDAV takes PUT, MKCOL, MOVE and DELETE of files kept aside, and answers LOCK/UNLOCK/PROPPATCH as class 2 so Finder and Explorer mount it writable. Expect: 100-continue is held until the write is known to be allowed. shares.add_files (on) turns it off from Settings, File shares. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g --- CHANGELOG.md | 9 + docs/file-tree.md | 66 ++- fastapi_backend/app/domain/filetree.py | 43 +- fastapi_backend/app/domain/http_body.py | 188 +++++++ fastapi_backend/app/domain/s3.py | 396 ++++++++++++++- fastapi_backend/app/domain/settings.py | 4 +- fastapi_backend/app/domain/share_import.py | 477 ++++++++++++++++++ fastapi_backend/app/domain/shares.py | 11 +- fastapi_backend/app/domain/store.py | 5 +- fastapi_backend/app/domain/uploads.py | 40 +- fastapi_backend/app/domain/webdav.py | 231 ++++++++- fastapi_backend/app/schemas/filetree.py | 1 + fastapi_backend/tests/api/test_s3_writes.py | 276 ++++++++++ fastapi_backend/tests/api/test_webdav.py | 20 +- .../tests/api/test_webdav_writes.py | 157 ++++++ .../app/openapi-client/types.gen.ts | 12 + .../components/account/s3-keys.tsx | 7 +- nextjs-frontend/components/settings/model.ts | 13 +- nextjs-frontend/openapi.json | 12 + threat-model.json | 2 +- threat-model.md | 8 +- threat-model.yaml | 8 +- 22 files changed, 1910 insertions(+), 76 deletions(-) create mode 100644 fastapi_backend/app/domain/http_body.py create mode 100644 fastapi_backend/app/domain/share_import.py create mode 100644 fastapi_backend/tests/api/test_s3_writes.py create mode 100644 fastapi_backend/tests/api/test_webdav_writes.py diff --git a/CHANGELOG.md b/CHANGELOG.md index c80f9ecb3..b6d7fe7ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased +- **Files added over S3 and WebDAV are imported.** Copy a file into a collection's folder (`rclone copy`, Finder, + Windows Explorer, any S3 client) and it's imported there as an upload is, with the same checks and the namespace's + pipeline; a new folder under `Collections` is a new collection, and a file anywhere else goes to the namespace's + default collection. It takes the import permission (or an editor's role on the collection), files the system writes + for itself (`.DS_Store`, `Thumbs.db`, half-written files) are never imported, and a re-sync copies nothing twice. + The S3 server now takes streamed and multipart uploads; WebDAV takes PUT, MKCOL and Finder's and Explorer's ways of + copying. This is on whenever the shares are; **Import files people add** in Settings, File shares + (`shares.add_files`) turns it off and makes them read-only again ([Adding files](docs/file-tree.md#adding-files)). + ## 0.10.0 October 15, 2026 {id="0.10.0"} - **The assistant always answers.** When a question used up its tool steps (`ai.max_steps`), the assistant used to stop diff --git a/docs/file-tree.md b/docs/file-tree.md index 26b6cddac..fa2480209 100644 --- a/docs/file-tree.md +++ b/docs/file-tree.md @@ -3,12 +3,13 @@ Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read -Lens without learning its API. +Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3 or WebDAV +is imported there ([Adding files](#adding-files)). | Where | Status | | --- | --- | | The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built | -| An [S3 server](#s3) and a [WebDAV server](#webdav) (read-only, off until turned on) | built | +| An [S3 server](#s3) and a [WebDAV server](#webdav) (off until turned on); files added through them are imported | built | | An [NFS share](file-shares.md) (read-only, off until turned on, allowed networks only) | built | | An [SMB share](smb.md) (read-only, off until turned on) | built | @@ -45,7 +46,8 @@ Names are made safe for every file system: `/ \ : * ? " < > |` and control chara Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces they have a role in, and in a namespace where they only have roles on some collections, those collections and the -topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The tree only reads. +topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The API only +reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers. ## The API @@ -59,7 +61,7 @@ The MCP server has `list_files` and `read_file` ([MCP server](mcp.md)): `read_fi ## S3 Off until an admin turns it on, in Settings, File shares (`shares.enabled`). Then Lens's worker serves the tree -read-only over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the +over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the host), where the web app is: this machine only unless Lens was opened to the network (`LENS_BIND`; `LENS_SHARE_BIND` sets it apart). Each namespace you can read is a bucket; keys are the paths above. @@ -84,10 +86,14 @@ endpoint = http://lens.local:9555 force_path_style = true ``` -then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, or `rclone mount lens: ~/lens`. +then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, `rclone copy ./talks +"lens:pods/Collections/Talks"` ([Adding files](#adding-files)), or `rclone mount lens: ~/lens`. Supported: listing buckets, listing objects (v1 and v2, by folder or recursively, a page at a time), reading objects whole or -in ranges, and presigned links. Anything that writes answers AccessDenied. Requests must be signed with Signature -Version 4. +in ranges, presigned links, and adding files: PutObject (whole, streamed with `aws-chunked` and signed chunks, or with +a checksum), multipart uploads, a key ending in `/` for a folder, a copy of a file just added onto itself (to set its +time, as rclone does), and DeleteObject(s) for files kept aside. A file's `x-amz-meta-mtime` (rclone's) dates it. +Making or deleting buckets and deleting assets answer AccessDenied; copying other objects, NotImplemented. Requests must be signed with +Signature Version 4. The server speaks plain HTTP. On a network you don't trust, keep it on this machine or behind TLS (a reverse proxy), since files travel in clear. @@ -95,20 +101,58 @@ since files travel in clear. ## WebDAV Off until an admin turns it on, with S3 (Settings, File shares; `shares.webdav`, on by default once shares are). -Lens's worker serves the same tree read-only over WebDAV on port 9556 (`shares.webdav_port`; with Docker, +Lens's worker serves the same tree over WebDAV on port 9556 (`shares.webdav_port`; with Docker, `LENS_WEBDAV_PORT`), so it mounts as a drive: in Finder, Go, Connect to Server; in Windows Explorer, Map network drive; in GNOME Files or Dolphin, `dav://lens.local:9556/`; or `rclone` with `type = webdav`. Sign in with an access key: its id as the user name and its secret as the password. The folders at the top are your -namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work; anything that writes answers -403. Eight failed sign-ins from one address within 15 minutes are refused for a while. +namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work, and so does adding files +([Adding files](#adding-files)): PUT (with `Expect: 100-continue`, refused before the file is sent when it can't go +there), MKCOL for a folder, DELETE and MOVE for files kept aside, and PROPPATCH (answered, not kept), so dragging +files and folders in from Finder or Windows Explorer works. LOCK and UNLOCK answer as a class 2 server so Finder mounts +it writable, but they lock nothing. Deleting or moving assets answers 403. Eight failed sign-ins from one address within 15 +minutes are refused for a while. Like S3, it speaks plain HTTP, and Basic sign-in sends the secret with every request: keep it on this machine or behind TLS on a network you don't trust. +## Adding files + +A file copied into the tree over S3 or WebDAV is imported, as an upload in the web app is: the same checks of its type +and size (Settings, Uploads), the namespace's pipeline, and the same file twice is added once. Where it's put decides +where it goes: + +| Put it in | It goes to | +| --- | --- | +| `/Collections//…` | that collection | +| `/Collections//…` | a new collection, made with any missing ones above it (up to 8 deep), as **New collection** makes one | +| `/`, `/Collections/`, or a folder under `Topics/` or `Entities/` | the namespace's default collection; topics and entities come from the analysis | + +Folder names match collections whatever their case. It must be yours to add to: the import permission in the +namespace, or an editor's or admin's role on the collection (making a collection takes what making one in the web app +takes). Otherwise the file is refused before it's sent where the client allows (S3's and WebDAV's `Expect: +100-continue`). Files Lens made (an asset's own file and its `.json`, `.txt` and `.pdf`) can't be written over. Each +file added is audited (`share.import`, with the protocol and path), and each collection made too. + +After a file is imported, the asset shows as ` [<id>]` like every other; the name you gave it stays in its folder, +for you, for 10 minutes, so the copying app finds the file it just wrote and a re-sync sees it's there (with its own +time and MD5, so `rclone copy` and `rclone check` agree). Files an operating system or an app writes for itself +(`.DS_Store`, `._` files, `Thumbs.db`, `desktop.ini`, Office's lock files, `.part`, `.tmp` and other hidden or +half-written files) and empty files (Finder and Explorer make one before they write it) are kept aside for 10 minutes +after they last change, seen only by you, and never imported; renaming one to an ordinary name imports it then. They're +kept on the uploads disk, in memory per worker, so a restart forgets them. + +Adding files is on whenever the shares are; turn it off in Settings, File shares (**Import files people add**, +`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The +[NFS](file-shares.md) and [SMB](smb.md) shares stay read-only for now. + ## Refine later -- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks for clients that insist on them. +- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks that lock. +- Delete, rename and move assets through the shares (only files just added can be today), and copy objects over S3. +- A read-only access key, for a person who may add files but wants a key that can't. +- Keep the names files were added under past 10 minutes and across restarts; a folder copy that hits a file type Lens + doesn't take stops in Finder. - Show supplementary files (captions, translations, attachments) beside each asset. - Cache an asset's text and metadata so large folders list faster. - A sidecar for each collection, topic and entity, and a folder of notes. diff --git a/fastapi_backend/app/domain/filetree.py b/fastapi_backend/app/domain/filetree.py index 79692c413..cc8aece6c 100644 --- a/fastapi_backend/app/domain/filetree.py +++ b/fastapi_backend/app/domain/filetree.py @@ -16,8 +16,9 @@ What someone sees is what the web app shows them: the namespaces they have a role in (admins: all of them), and in a namespace where they only have roles on some collections, those collections and the topics and entities of the -assets in them. The tree only reads; names are made safe for every file system (no `/ \\ : * ? " < > |`) and unique -in their folder, ignoring case. +assets in them. Names are made safe for every file system (no `/ \\ : * ? " < > |`) and unique in their folder, +ignoring case. The tree itself only reads: a file someone adds through a share is imported by share_import.py, which +shows it to them under the name they gave it for a while (overlay). """ from __future__ import annotations @@ -58,17 +59,24 @@ class NotAFolder(ValueError): @dataclass(frozen=True) class Viewer: """Who is looking: an admin sees every namespace; others their namespaces (`roles`) and, in namespaces they have - no role in, the collections they were given a role on, with the ones inside them (`collections`).""" + no role in, the collections they were given a role on, with the ones inside them (`collections`). `account` and + `perms` (their permissions in each namespace, roles.py) are for adding files (share_import.py).""" admin: bool = False roles: dict[int, str] = field(default_factory=dict) collections: dict[int, dict[int, str]] = field(default_factory=dict) + account: int | None = None + perms: dict[int, frozenset[str]] = field(default_factory=dict) @classmethod def of(cls, db, account: dict[str, Any]) -> Viewer: """For an account row ({id, admin}).""" + from . import roles as rl + admin = bool(account.get("admin")) - return cls(admin, auth.roles(db, account), {} if admin else hierarchy.roles_of(db, account["id"])) + perms = auth.permissions(db, account) + roles = {sid: rl.rank(p) for sid, p in perms.items()} # as auth.roles + return cls(admin, roles, {} if admin else hierarchy.roles_of(db, account["id"]), account.get("id"), perms) def spaces(self, db) -> dict[int, set[int] | None]: """{namespace: None (all of it) or the collections seen}.""" @@ -84,7 +92,9 @@ def spaces(self, db) -> dict[int, set[int] | None]: @dataclass class Entry: """A file or a folder. A file's bytes come from `data` (made here) or `path` (stored, maybe encrypted) or a storage - source (`remote`: {source, path, space}).""" + source (`remote`: {source, path, space}). `added` marks a file someone just added through a share, shown to them + under the name they gave it (share_import.py): "arrived" (imported: the asset's own file) or "pending" (kept + aside, not imported).""" name: str folder: bool @@ -95,11 +105,14 @@ class Entry: path: str | None = None encrypted: bool = False remote: dict[str, Any] | None = None + added: str | None = None + md5: str | None = None # of a file just added, as the client sent it (its ETag over S3) + mtime: str | None = None # the time the client gave a file just added, as it wrote it (rclone's x-amz-meta-mtime) @property def version(self) -> str: - """Changes when the file does (for ETags): its name, size and time.""" - return hashlib.sha256(f"{self.name}\0{self.size}\0{self.modified}".encode()).hexdigest()[:24] + """Changes when the file does (for ETags): its name, size and time; a file just added, its MD5.""" + return self.md5 or hashlib.sha256(f"{self.name}\0{self.size}\0{self.modified}".encode()).hexdigest()[:24] def out(self) -> dict[str, Any]: d: dict[str, Any] = {"name": self.name, "folder": self.folder, "modified": self.modified} @@ -203,7 +216,23 @@ def list(self, path: str) -> list[Entry]: raise NotAFolder(path) from None raise + def forget(self) -> None: + """Read the database afresh from now on (after a file was added or a collection made).""" + self._cols.clear() + self._seen.clear() + self._topic_rows.clear() + def _list(self, parts: list[str]) -> list[Entry]: + out = self._listing(parts) + if parts and self.viewer.account is not None: + from . import share_import + + extra = share_import.overlay(self, parts, {e.name for e in out}) + if extra: + out = sorted(out + extra, key=lambda e: e.name.casefold()) + return out + + def _listing(self, parts: list[str]) -> list[Entry]: if not parts: return [_folder(n) for n in sorted(self._names.values(), key=str.casefold)] sid = self._space(parts[0]) diff --git a/fastapi_backend/app/domain/http_body.py b/fastapi_backend/app/domain/http_body.py new file mode 100644 index 000000000..c581374c3 --- /dev/null +++ b/fastapi_backend/app/domain/http_body.py @@ -0,0 +1,188 @@ +"""Request bodies as the file shares receive them (s3.py, webdav.py): streamed to a file, never held in memory, as +Content-Length bytes, HTTP/1.1 chunked transfer coding, or S3's aws-chunked coding (STREAMING-… payloads, whose +chunk signatures are checked when they're signed). Each body is checked against what the client said about it: +its length, the SHA-256 it signed, its Content-MD5, and an x-amz-checksum-crc32 or -sha256. +""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import zlib +from collections.abc import Iterator + +PIECE = 1 << 16 +LINE_MAX = 8192 +EMPTY_SHA256 = hashlib.sha256(b"").hexdigest() + + +class BodyError(ValueError): + """A body that's malformed, cut short, too large, or isn't what was signed. `code` names it as S3 does.""" + + def __init__(self, code: str, message: str, status: int = 400): + super().__init__(message) + self.code, self.message, self.status = code, message, status + + +class _Stream: + """Bytes from an iterator of pieces, read by length or by line.""" + + def __init__(self, pieces: Iterator[bytes]): + self.pieces, self.buf = pieces, b"" + + def _fill(self) -> bool: + piece = next(self.pieces, b"") + self.buf += piece + return bool(piece) + + def readline(self) -> bytes: + while b"\n" not in self.buf: + if len(self.buf) > LINE_MAX: + raise BodyError("IncompleteBody", "a chunk's header line is too long") + if not self._fill(): + raise BodyError("IncompleteBody", "the body ended early") + line, _, self.buf = self.buf.partition(b"\n") + return line.rstrip(b"\r") + + def read(self, n: int) -> Iterator[bytes]: + """Exactly `n` bytes, a piece at a time.""" + while n > 0: + if not self.buf and not self._fill(): + raise BodyError("IncompleteBody", "the body ended early") + out, self.buf = self.buf[:n], self.buf[n:] + n -= len(out) + yield out + + +def _fixed(rfile, length: int) -> Iterator[bytes]: + while length > 0: + piece = rfile.read(min(PIECE, length)) + if not piece: + raise BodyError("IncompleteBody", "the body ended before Content-Length bytes") + length -= len(piece) + yield piece + + +def _chunk_size(line: bytes) -> tuple[int, dict[str, str]]: + head, *params = line.decode("latin-1").split(";") + try: + size = int(head.strip(), 16) + except ValueError: + raise BodyError("IncompleteBody", "a chunk's size isn't a hex number") from None + return size, dict(p.strip().split("=", 1) for p in params if "=" in p) + + +def _http_chunked(rfile) -> Iterator[bytes]: + s = _Stream(iter(lambda: rfile.read1(PIECE) if hasattr(rfile, "read1") else rfile.read(1), b"")) + while True: + size, _ = _chunk_size(s.readline()) + if size == 0: + while s.readline(): # trailers, then the blank line + pass + return + yield from s.read(size) + if s.readline(): + raise BodyError("IncompleteBody", "a chunk isn't followed by CRLF") + + +class Signed: + """What a STREAMING-AWS4-HMAC-SHA256-PAYLOAD body's chunks are signed with: the signing key, the request's time and + scope, and the seed signature (the request's own).""" + + def __init__(self, key: bytes, amz: str, scope: str, seed: str): + self.key, self.amz, self.scope, self.prev = key, amz, scope, seed + + def check(self, sig: str, chunk_sha256: str) -> None: + to_sign = "\n".join(["AWS4-HMAC-SHA256-PAYLOAD", self.amz, self.scope, self.prev, EMPTY_SHA256, chunk_sha256]) + want = hmac.new(self.key, to_sign.encode(), hashlib.sha256).hexdigest() + if not hmac.compare_digest(want, sig or ""): + raise BodyError("SignatureDoesNotMatch", "a chunk's signature doesn't match", 403) + self.prev = want + + +def _aws_chunked(raw: Iterator[bytes], signed: Signed | None, trailer: dict[str, str]) -> Iterator[bytes]: + s = _Stream(raw) + while True: + size, params = _chunk_size(s.readline()) + h = hashlib.sha256() if signed else None + for piece in s.read(size): + if h: + h.update(piece) + yield piece + if signed: + signed.check(params.get("chunk-signature", ""), h.hexdigest()) # type: ignore[union-attr] + if size == 0: + while line := s.readline(): # trailers (a checksum, its signature), then the blank line + k, _, v = line.decode("latin-1").partition(":") + trailer[k.strip().lower()] = v.strip() + return + if s.readline(): + raise BodyError("IncompleteBody", "a chunk isn't followed by CRLF") + + +def receive(rfile, headers: dict[str, str], dest, limit: int, signed: Signed | None = None) -> tuple[int, str]: + """Stream a request's body into the file `dest` (lowercased `headers`): (its size, its MD5 in hex). BodyError when + it's malformed, over `limit` bytes, or not what the client said it is.""" + te = headers.get("transfer-encoding", "").lower() + if "chunked" in te: + raw = _http_chunked(rfile) + elif headers.get("content-length", "").strip().isdigit(): + length = int(headers["content-length"]) + if length > limit and not headers.get("x-amz-content-sha256", "").startswith("STREAMING-"): + raise BodyError("EntityTooLarge", "the file is larger than this server takes") + raw = _fixed(rfile, length) + else: + raise BodyError("MissingContentLength", "send Content-Length, or the body chunked", 411) + payload = headers.get("x-amz-content-sha256", "") + trailer: dict[str, str] = {} + if payload.startswith("STREAMING-"): + if payload.startswith("STREAMING-AWS4-HMAC-SHA256-PAYLOAD") and signed is None: + raise BodyError("InvalidRequest", "a signed streaming body needs a signed request", 403) + body = _aws_chunked(raw, signed if payload.startswith("STREAMING-AWS4-HMAC-SHA256-PAYLOAD") else None, trailer) + else: + body = raw + want = {k: headers[k] for k in ("x-amz-checksum-crc32", "x-amz-checksum-sha256") if headers.get(k)} + named = {t.strip().lower() for t in headers.get("x-amz-trailer", "").split(",") if t.strip()} + sha = hashlib.sha256() if len(payload) == 64 or "x-amz-checksum-sha256" in want or "x-amz-checksum-sha256" in named else None + crc = 0 if "x-amz-checksum-crc32" in want or "x-amz-checksum-crc32" in named else None + md5, size = hashlib.md5(usedforsecurity=False), 0 + with open(dest, "wb") as f: + for piece in body: + size += len(piece) + if size > limit: + raise BodyError("EntityTooLarge", "the file is larger than this server takes") + md5.update(piece) + if sha: + sha.update(piece) + if crc is not None: + crc = zlib.crc32(piece, crc) + f.write(piece) + want.update({k: v for k, v in trailer.items() if k in ("x-amz-checksum-crc32", "x-amz-checksum-sha256")}) + if headers.get("x-amz-decoded-content-length", "").isdigit() and int(headers["x-amz-decoded-content-length"]) != size: + raise BodyError("IncompleteBody", "the body isn't as long as x-amz-decoded-content-length says") + if len(payload) == 64 and sha and not hmac.compare_digest(sha.hexdigest(), payload.lower()): + raise BodyError("XAmzContentSHA256Mismatch", "the body isn't what was signed") + if headers.get("content-md5") and not hmac.compare_digest(base64.b64encode(md5.digest()).decode(), headers["content-md5"].strip()): + raise BodyError("BadDigest", "the body doesn't match its Content-MD5") + if "x-amz-checksum-sha256" in want and sha and base64.b64encode(sha.digest()).decode() != want["x-amz-checksum-sha256"]: + raise BodyError("BadDigest", "the body doesn't match its x-amz-checksum-sha256") + if ( + "x-amz-checksum-crc32" in want + and crc is not None + and base64.b64encode(crc.to_bytes(4, "big")).decode() != want["x-amz-checksum-crc32"] + ): + raise BodyError("BadDigest", "the body doesn't match its x-amz-checksum-crc32") + return size, md5.hexdigest() + + +def drain(rfile, headers: dict[str, str], most: int = 1 << 20) -> bool: + """Read and drop a small body (up to `most` bytes) so the connection can carry on; False when it can't be (too + large, chunked, or the client waits for 100 Continue): close the connection instead.""" + if headers.get("expect", "").lower() == "100-continue" or "chunked" in headers.get("transfer-encoding", "").lower(): + return False + n = int(headers.get("content-length") or 0) if (headers.get("content-length") or "0").isdigit() else most + 1 + if n > most: + return False + rfile.read(n) + return True diff --git a/fastapi_backend/app/domain/s3.py b/fastapi_backend/app/domain/s3.py index e024cc9f0..9773a6451 100644 --- a/fastapi_backend/app/domain/s3.py +++ b/fastapi_backend/app/domain/s3.py @@ -1,14 +1,18 @@ -"""The file tree over S3 (docs/file-tree.md#s3): a read-only S3 server any S3 client mounts or syncs (rclone, -Cyberduck, the AWS CLI, s3fs, boto3), each person's namespaces as buckets. +"""The file tree over S3 (docs/file-tree.md#s3): an S3 server any S3 client mounts or syncs (rclone, Cyberduck, the +AWS CLI, s3fs, boto3), each person's namespaces as buckets. Off until an admin turns it on (`shares.enabled`); then the process that runs routines (`lens worker`) listens on `shares.s3_port`, as the sensor hub does (shares.py). Every request is signed (AWS Signature Version 4, in the Authorization header or a presigned link) with an access key someone made for themselves in the web app; the key acts as them and sees what they see in the web app. Keys' secrets are sealed with the server's key (they must be readable: SigV4 signs -with the secret itself), shown once, and can be revoked. Writes answer AccessDenied. +with the secret itself), shown once, and can be revoked. Path-style addressing (http://host:port/<bucket>/<key>), any region name. Supported: ListBuckets, HeadBucket, -GetBucketLocation, ListObjects (v1 and v2, with a delimiter or recursive), HeadObject and GetObject (with Range). +GetBucketLocation, ListObjects (v1 and v2, with a delimiter or recursive), HeadObject and GetObject (with Range). While +`shares.add_files` is on, a file put into the tree is imported (share_import.py): PutObject (whole, chunked, or +aws-chunked with signed chunks), multipart uploads, a folder marker (a key ending in /) under Collections makes a +collection, and DeleteObject(s) removes only what share_import.py keeps aside. Anything else that writes answers +AccessDenied or NotImplemented. """ from __future__ import annotations @@ -16,17 +20,23 @@ import base64 import datetime as dt import email.utils +import errno import hashlib import hmac import http.server +import json +import os import re import secrets +import shutil import threading +import time import urllib.parse +import xml.etree.ElementTree as ET from typing import Any from xml.sax.saxutils import escape -from . import auth, filetree, keyring, settings, store +from . import auth, filetree, http_body, keyring, settings, share_import, store, uploads R = store.R NS = "http://s3.amazonaws.com/doc/2006-03-01/" @@ -36,6 +46,9 @@ MAX_KEYS = 1000 MAX_PRESIGN_SECONDS = 7 * 24 * 3600 NAME_MAX = 80 +UPLOAD_ID = re.compile(r"^[A-Za-z0-9_-]{32}$") +MAX_PARTS = 10000 +XML_MAX = 1 << 20 # a CompleteMultipartUpload or DeleteObjects body class S3Error(Exception): @@ -193,9 +206,12 @@ def _hmac(key, msg): return hmac.new(key, msg.encode(), hashlib.sha256).digest() +def signing_key(secret, date, region, service): + return _hmac(_hmac(_hmac(_hmac(("AWS4" + secret).encode(), date), region), service), "aws4_request") + + def signature(secret, date, region, service, to_sign): - k = _hmac(_hmac(_hmac(_hmac(("AWS4" + secret).encode(), date), region), service), "aws4_request") - return hmac.new(k, to_sign.encode(), hashlib.sha256).hexdigest() + return hmac.new(signing_key(secret, date, region, service), to_sign.encode(), hashlib.sha256).hexdigest() def _amz_time(s): @@ -205,9 +221,10 @@ def _amz_time(s): raise S3Error(403, "AccessDenied", "the request's date is missing or malformed") from None -def verify(db, cfg, method, raw_path, pairs, headers, now=None): +def verify(db, cfg, method, raw_path, pairs, headers, now=None, chunks=None): """Check a request's signature; the account it acts as. `pairs`: the query string's (name, value) pairs, `headers` - lowercased names to values.""" + lowercased names to values. `chunks`, a dict, gets what a streaming body's chunks are signed with + (http_body.Signed) under "signed".""" now = now or dt.datetime.now(dt.timezone.utc) q = dict(pairs) presigned = "X-Amz-Signature" in q @@ -255,6 +272,9 @@ def verify(db, cfg, method, raw_path, pairs, headers, now=None): if not hmac.compare_digest(signature(secret, date, region, service, to_sign), sig): raise S3Error(403, "SignatureDoesNotMatch", "the signature doesn't match: check the secret key") db.q("UPDATE $r SET last_used_at = $t", r=R("file_key", kid), t=store.now()) + if chunks is not None: + stamp = amz or when.strftime("%Y%m%dT%H%M%SZ") + chunks["signed"] = http_body.Signed(signing_key(secret, date, region, service), stamp, "/".join(scope[1:]), sig) return acct @@ -416,9 +436,66 @@ def do_GET(self): def do_HEAD(self): self._handle(head=True) + def handle_expect_100(self): + """Hold back 100 Continue until the request is known to be allowed (_go_on): a refused upload is never sent.""" + return True + + def _go_on(self): + if self.headers.get("expect", "").lower() == "100-continue": + self.send_response_only(100) + self.end_headers() + def _write(self): - self.rfile.read(min(int(self.headers.get("content-length") or 0), 1 << 20)) # what was sent, to keep the connection - self._fail(S3Error(403, "AccessDenied", "the file tree is read-only")) + raw, _, query = self.path.partition("?") + headers = {k.lower(): v for k, v in self.headers.items()} + self._read = False + try: + pairs = urllib.parse.parse_qsl(query, keep_blank_values=True) + db, cfg = self.server.db, self.server.cfg_fn() + chunks: dict[str, Any] = {} + acct = verify(db, cfg, self.command, raw, pairs, headers, chunks=chunks) + if not share_import.accepting(cfg, "s3"): + raise S3Error(403, "AccessDenied", "the file tree is read-only") + adder = share_import.Adder(db, cfg, acct, "s3") + bucket, _, key = urllib.parse.unquote(raw).lstrip("/").partition("/") + q = dict(pairs) + if not bucket or (not key and self.command != "POST"): + raise S3Error(403, "AccessDenied", "namespaces are made in the web app" if bucket else "name a bucket") + if bucket not in [e.name for e in adder.tree.list("")]: + raise S3Error(404, "NoSuchBucket", "no such bucket, or not yours to read") + Writes(self, db, cfg, acct, adder, bucket, key, q, headers, chunks.get("signed")).run() + except S3Error as e: + self._refuse(e, raw, headers) + except http_body.BodyError as e: + self._refuse(S3Error(e.status, e.code, e.message), raw, headers) + except share_import.TooLarge as e: + self._refuse(S3Error(400, "EntityTooLarge", str(e)), raw, headers) + except share_import.Rejected as e: + self._refuse(S3Error(400, "InvalidArgument", str(e)), raw, headers) + except share_import.Refused as e: + self._refuse(S3Error(403, "AccessDenied", str(e)), raw, headers) + except filetree.NotFound: + self._refuse(S3Error(403, "AccessDenied", "no such folder: files go in a namespace, or in Collections/…"), raw, headers) + except keyring.Locked: + self._refuse(S3Error(403, "AccessDenied", "this namespace is a locked vault: an owner unlocks it in the web app"), raw, headers) + except (BrokenPipeError, ConnectionResetError): + self.close_connection = True + except OSError as e: + if e.errno != errno.ENOSPC: + self.server.log(f"s3: {type(e).__name__}: {e}") + msg = "the server doesn't have room for this file" if e.errno == errno.ENOSPC else "something went wrong on the server" + self._refuse(S3Error(507 if e.errno == errno.ENOSPC else 500, "InternalError", msg), raw, headers) + except Exception as e: # noqa: BLE001 - say so, and keep serving + self.server.log(f"s3: {type(e).__name__}: {e}") + self._refuse(S3Error(500, "InternalError", "something went wrong on the server"), raw, headers) + + def _refuse(self, e, raw, headers): + try: + if not self._read and not http_body.drain(self.rfile, headers): + self.close_connection = True # the body is still coming (or never will): don't read it + self._fail(e, raw) + except (BrokenPipeError, ConnectionResetError): # the client went away mid-file + self.close_connection = True do_PUT = do_POST = do_DELETE = _write # noqa: N815 - http.server's names @@ -442,9 +519,13 @@ def _handle(self, head): return self._send(200, _xml(f'<LocationConstraint xmlns="{NS}"></LocationConstraint>'), head=head) if head: return self._send(200, head=True) - if set(q) & {"acl", "policy", "versioning", "lifecycle", "cors", "tagging", "uploads", "website", "object-lock"}: - raise S3Error(501, "NotImplemented", "the file tree only lists and reads") + if "uploads" in q: + return self._send(200, list_uploads(cfg, acct, bucket)) + if set(q) & {"acl", "policy", "versioning", "lifecycle", "cors", "tagging", "website", "object-lock"}: + raise S3Error(501, "NotImplemented", "the file tree only lists, reads and takes files") return self._send(200, list_objects(tree, bucket, q)) + if "uploadId" in q and not head: + return self._send(200, list_parts(cfg, acct, bucket, key, q["uploadId"])) return self._object(tree, db, cfg, bucket, key, head) except S3Error as e: self._fail(e, raw) @@ -485,6 +566,8 @@ def _object(self, tree, db, cfg, bucket, key, head): self.send_header("Accept-Ranges", "bytes") if status == 206: self.send_header("Content-Range", f"bytes {start}-{end}/{e.size}") + if e.mtime: + self.send_header("x-amz-meta-mtime", e.mtime) # as rclone sent it self.end_headers() if head or end < start: return @@ -492,6 +575,293 @@ def _object(self, tree, db, cfg, bucket, key, head): self.wfile.write(chunk) +# ---------- adding files ---------- +def _mtime(headers): + """The file's own time a client sent (rclone: x-amz-meta-mtime, seconds since 1970, as text), or None.""" + return headers.get("x-amz-meta-mtime") or None + + +def _limit(cfg): + return int(cfg["uploads"]["max_mb"]) * uploads.MB + + +class Writes: + """One request that writes (PUT, POST, DELETE), for an account allowed to add files.""" + + def __init__(self, h, db, cfg, acct, adder, bucket, key, q, headers, signed): + self.h, self.db, self.cfg, self.acct, self.adder = h, db, cfg, acct, adder + self.bucket, self.key, self.q, self.headers, self.signed = bucket, key, q, headers, signed + self.path = f"{bucket}/{key}" + + def run(self): + m, q = self.h.command, self.q + if not self.key: # POST to the bucket + if "delete" in q: + return self.delete_many() + raise S3Error(501, "NotImplemented", "the file tree only lists, reads and takes files") + if "uploadId" in q: + if m == "PUT" and "partNumber" in q: + return self.part() + if m == "POST": + return self.complete() + if m == "DELETE": + return self.abort() + if m == "POST" and "uploads" in q: + return self.start() + if m == "DELETE": + return self.delete() + if m == "PUT" and not set(q) - {"x-id"}: # x-id: the AWS SDKs' operation name + if self.headers.get("x-amz-copy-source"): + return self.retime() + return self.folder() if self.key.endswith("/") else self.put() + raise S3Error(501, "NotImplemented", "copying, tags and the like aren't served: put a file, or upload it in parts") + + def _declared(self): + n = self.headers.get("x-amz-decoded-content-length") or self.headers.get("content-length") or "" + return int(n) if n.isdigit() else None + + def _receive(self, dest): + """The body into `dest`: (size, md5 hex).""" + self.h._go_on() + self.h._read = True + if "chunked" in self.headers.get("transfer-encoding", "").lower(): + self.h.close_connection = True # a chunked body may have been read past its end + return http_body.receive(self.h.rfile, self.headers, dest, _limit(self.cfg), self.signed) + + def put(self): + self.adder.check(self.path, self._declared()) + dest = share_import.spool(self.cfg) + try: + _, md5 = self._receive(dest) + self.adder.put(self.path, dest, _mtime(self.headers), md5) # imported (moved away), or kept aside there + except BaseException: + share_import._remove(dest) + raise + self.h._send(200, headers={"ETag": f'"{md5}"'}) + + def retime(self): + """CopyObject of a file just added onto itself with new metadata: rclone setting its time. Nothing else.""" + source = urllib.parse.unquote(self.headers["x-amz-copy-source"].split("?")[0]).lstrip("/") + replace = self.headers.get("x-amz-metadata-directive", "").upper() == "REPLACE" + if source != self.path or not replace or not self.adder.touch(self.path, _mtime(self.headers)): + raise S3Error(501, "NotImplemented", "copying isn't served: put the file instead") + self.h._read = True + e = self.adder.tree.stat(self.path) + self.h._send(200, _xml(f'<CopyObjectResult xmlns="{NS}"><LastModified>{_iso(e.modified)}</LastModified>' + f"<ETag>"{e.version}"</ETag></CopyObjectResult>")) # fmt: skip + + def folder(self): + if self._declared(): + raise S3Error(400, "InvalidArgument", "a key ending in / is a folder: it holds no bytes") + self.h._read = True + self.adder.folder(self.path.rstrip("/")) + self.h._send(200, headers={"ETag": f'"{hashlib.md5(b"", usedforsecurity=False).hexdigest()}"'}) + + def delete(self): + self.h._read = True + try: + self.adder.delete(self.path) + except share_import.Refused: + try: + self.adder.tree.stat(self.path) + except filetree.NotFound: + return self.h._send(204) # nothing there: S3 says done + raise + self.h._send(204) + + def delete_many(self): + body = self._xml_body() + quiet = (body.findtext("{*}Quiet") or body.findtext("Quiet") or "").lower() == "true" + done, errs = [], [] + for o in body.iter(): + if o.tag.split("}")[-1] != "Object": + continue + k = next((c.text or "" for c in o if c.tag.split("}")[-1] == "Key"), "") + try: + self.adder.delete(f"{self.bucket}/{k}") + done.append(k) + except share_import.Refused as e: + try: + self.adder.tree.stat(f"{self.bucket}/{k}") + errs.append((k, str(e))) + except filetree.NotFound: + done.append(k) + out = [] if quiet else [f"<Deleted><Key>{escape(k)}</Key></Deleted>" for k in done] + out += [f"<Error><Key>{escape(k)}</Key><Code>AccessDenied</Code><Message>{escape(m)}</Message></Error>" for k, m in errs] + self.h._send(200, _xml(f'<DeleteResult xmlns="{NS}">' + "".join(out) + "</DeleteResult>")) + + def _xml_body(self): + n = self._declared() or 0 + if n > XML_MAX: + raise S3Error(400, "MalformedXML", "the request is too large") + dest = share_import.spool(self.cfg) + try: + self._receive(dest) + raw = dest.read_bytes() + finally: + share_import._remove(dest) + if len(raw) > XML_MAX: + raise S3Error(400, "MalformedXML", "the request is too large") + if b"<!DOCTYPE" in raw or b"<!ENTITY" in raw: + raise S3Error(400, "MalformedXML", "the request isn't valid") + try: + return ET.fromstring(raw) + except ET.ParseError: + raise S3Error(400, "MalformedXML", "the request isn't valid XML") from None + + # ---------- in parts ---------- + def start(self): + self.adder.check(self.path) + self.h._read = True + uid = start_upload(self.cfg, self.acct["id"], self.bucket, self.key, _mtime(self.headers)) + self.h._send(200, _xml(f'<InitiateMultipartUploadResult xmlns="{NS}"><Bucket>{escape(self.bucket)}</Bucket>' + f"<Key>{escape(self.key)}</Key><UploadId>{uid}</UploadId></InitiateMultipartUploadResult>")) # fmt: skip + + def part(self): + d, _ = upload_of(self.cfg, self.acct["id"], self.bucket, self.key, self.q["uploadId"]) + try: + n = int(self.q["partNumber"]) + except ValueError: + n = 0 + if not 1 <= n <= MAX_PARTS: + raise S3Error(400, "InvalidArgument", f"part numbers go from 1 to {MAX_PARTS}") + tmp = d / f"{n:05d}.part" + try: + size, md5 = self._receive(tmp) + os.replace(tmp, d / f"{n:05d}") + (d / f"{n:05d}.md5").write_text(md5) + finally: + share_import._remove(tmp) + if sum(p["size"] for p in parts_of(d)) > _limit(self.cfg): + raise S3Error(400, "EntityTooLarge", f"files up to {self.cfg['uploads']['max_mb']} MB") + self.h._send(200, headers={"ETag": f'"{md5}"'}) + + def complete(self): + d, meta = upload_of(self.cfg, self.acct["id"], self.bucket, self.key, self.q["uploadId"]) + body = self._xml_body() + listed = [] + for p in body.iter(): + if p.tag.split("}")[-1] != "Part": + continue + kids = {c.tag.split("}")[-1]: (c.text or "").strip() for c in p} + try: + listed.append((int(kids.get("PartNumber", "")), kids.get("ETag", "").strip('"'))) + except ValueError: + raise S3Error(400, "MalformedXML", "a part's number isn't a number") from None + if not listed: + raise S3Error(400, "MalformedXML", "list the parts to put together") + if [n for n, _ in listed] != sorted({n for n, _ in listed}): + raise S3Error(400, "InvalidPartOrder", "list the parts in order, each once") + have = {p["number"]: p for p in parts_of(d)} + for n, etag in listed: + if n not in have or (etag and etag != have[n]["etag"]): + raise S3Error(400, "InvalidPart", f"part {n} wasn't uploaded, or its ETag differs") + whole = share_import.spool(self.cfg) + md5s = hashlib.md5(usedforsecurity=False) + self.h._read = True + try: + with open(whole, "wb") as out: + for n, _ in listed: + with open(d / f"{n:05d}", "rb") as f: + shutil.copyfileobj(f, out, 1 << 20) + md5s.update(bytes.fromhex(have[n]["etag"])) + self.adder.put(self.path, whole, meta.get("mtime")) + except BaseException: + share_import._remove(whole) + raise + shutil.rmtree(d, ignore_errors=True) + etag = f"{md5s.hexdigest()}-{len(listed)}" + self.h._send(200, _xml(f'<CompleteMultipartUploadResult xmlns="{NS}"><Location>/{escape(self.path)}</Location>' + f"<Bucket>{escape(self.bucket)}</Bucket><Key>{escape(self.key)}</Key>" + f"<ETag>"{etag}"</ETag></CompleteMultipartUploadResult>")) # fmt: skip + + def abort(self): + d, _ = upload_of(self.cfg, self.acct["id"], self.bucket, self.key, self.q["uploadId"]) + self.h._read = True + shutil.rmtree(d, ignore_errors=True) + self.h._send(204) + + +def _uploads_dir(cfg, uid): + if not UPLOAD_ID.match(uid or ""): + raise S3Error(404, "NoSuchUpload", "no such upload") + return uploads.incoming(cfg) / f"s3mp-{uid}" + + +def start_upload(cfg, account, bucket, key, mtime=None): + """A new multipart upload of `key` by an account: its id. Uploads nobody finished within uploads.expire_hours go.""" + sweep_uploads(cfg) + uid = secrets.token_urlsafe(24) + d = _uploads_dir(cfg, uid) + d.mkdir() + meta = {"account": account, "bucket": bucket, "key": key, "created": store.now(), "mtime": mtime} + (d / "meta.json").write_text(json.dumps(meta)) + return uid + + +def upload_of(cfg, account, bucket, key, uid): + """(its folder, its meta) for one of the account's uploads of this key; NoSuchUpload otherwise.""" + d = _uploads_dir(cfg, uid) + try: + meta = json.loads((d / "meta.json").read_text()) + except (OSError, ValueError): + raise S3Error(404, "NoSuchUpload", "no such upload: it was finished, cancelled or expired") from None + if (meta.get("account"), meta.get("bucket"), meta.get("key")) != (account, bucket, key): + raise S3Error(404, "NoSuchUpload", "no such upload") + return d, meta + + +def parts_of(d): + out = [] + for p in sorted(d.glob("[0-9][0-9][0-9][0-9][0-9]")): + try: + out.append({"number": int(p.name), "size": p.stat().st_size, "etag": (d / f"{p.name}.md5").read_text().strip(), + "modified": p.stat().st_mtime}) # fmt: skip + except (OSError, ValueError): + continue + return out + + +def sweep_uploads(cfg): + cut = time.time() - int(cfg["uploads"]["expire_hours"]) * 3600 + for d in uploads.incoming(cfg).glob("s3mp-*"): + try: + if d.stat().st_mtime < cut and all(p.stat().st_mtime < cut for p in d.iterdir()): + shutil.rmtree(d, ignore_errors=True) + except OSError: + pass + + +def _stamp(t): + return dt.datetime.fromtimestamp(t, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.000Z") + + +def list_parts(cfg, acct, bucket, key, uid): + d, _ = upload_of(cfg, acct["id"], bucket, key, uid) + parts = "".join( + f"<Part><PartNumber>{p['number']}</PartNumber><LastModified>{_stamp(p['modified'])}</LastModified>" + f"<ETag>"{p['etag']}"</ETag><Size>{p['size']}</Size></Part>" + for p in parts_of(d) + ) + return _xml(f'<ListPartsResult xmlns="{NS}"><Bucket>{escape(bucket)}</Bucket><Key>{escape(key)}</Key><UploadId>{uid}</UploadId>' + f"<IsTruncated>false</IsTruncated>{parts}</ListPartsResult>") # fmt: skip + + +def list_uploads(cfg, acct, bucket): + """The account's unfinished multipart uploads in a bucket.""" + out = [] + for d in sorted(uploads.incoming(cfg).glob("s3mp-*")): + try: + meta = json.loads((d / "meta.json").read_text()) + except (OSError, ValueError): + continue + if meta.get("account") == acct["id"] and meta.get("bucket") == bucket: + out.append(f"<Upload><Key>{escape(meta['key'])}</Key><UploadId>{d.name[5:]}</UploadId>" + f"<Initiated>{_iso(meta.get('created'))}</Initiated></Upload>") # fmt: skip + return _xml(f'<ListMultipartUploadsResult xmlns="{NS}"><Bucket>{escape(bucket)}</Bucket><IsTruncated>false</IsTruncated>' + + "".join(out) + "</ListMultipartUploadsResult>") # fmt: skip + + def password_account(db, cfg, kid, secret): """The account an access key and its secret sign in as (WebDAV's Basic sign-in), or None.""" found = key_account(db, cfg, kid) diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index 61340331b..6704e5c8b 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -82,7 +82,7 @@ "triage", ), # bind is a startup setting only - "shares": ("enabled", "s3", "s3_port", "webdav", "webdav_port"), + "shares": ("enabled", "s3", "s3_port", "webdav", "webdav_port", "add_files"), "video": ( "sample_seconds", "scene_threshold", @@ -1252,7 +1252,7 @@ def _smb_setting(key, value): def _share_setting(key, value): - if key in ("enabled", "s3", "webdav"): + if key in ("enabled", "s3", "webdav", "add_files"): if not isinstance(value, bool): raise ValueError(f"shares.{key} is true or false") return value diff --git a/fastapi_backend/app/domain/share_import.py b/fastapi_backend/app/domain/share_import.py new file mode 100644 index 000000000..56bdc84e7 --- /dev/null +++ b/fastapi_backend/app/domain/share_import.py @@ -0,0 +1,477 @@ +"""Files people add through the file shares (docs/file-tree.md#adding-files): a file written into the tree over S3, +WebDAV, SMB or NFS is imported as if it had been uploaded in the web app (uploads.take): the same checks of its type +and size, the namespace's pipeline, and the same file twice isn't added twice. + +Where it's written decides where it goes: + + <namespace>/Collections/<collection>/<inside it>/<file> that collection (made, with any above it, when missing) + <namespace>/<file>, <namespace>/Collections/<file> the namespace's default collection + <namespace>/Topics/…/<file>, <namespace>/Entities/…/<file> the default collection too (topics and entities come + from the analysis) + +It must be the person's to add to: the import permission in the namespace, or an editor's or admin's role on the +collection, as an upload in the web app. Making a collection takes what making one in the web app takes. Files Lens +made (an asset's own file, its .json, .txt and .pdf) can't be written over. + +A client that has just written a file looks for it under the name it gave, which the tree doesn't have (the asset +shows as `<title> [<id>]`). So the name stays in its folder, for the person who wrote it, for ARRIVED_SECONDS, +holding the asset's own file. Files an operating system or an app writes for itself (.DS_Store, ._ files, Thumbs.db, +desktop.ini, Office lock files, `.part` and `.tmp` files, anything hidden), and empty files (Finder and Explorer make +one before they write it) are kept aside instead, on the uploads disk for PENDING_SECONDS after they last changed, +and never imported; renaming one to an ordinary name imports it then (an upload under a temporary name). + +Each server process keeps these in memory (they last minutes); it's the person's own view, nobody else sees them. +""" + +from __future__ import annotations + +import datetime as dt +import mimetypes +import os +import pathlib +import re +import secrets +import threading +import time +from dataclasses import dataclass, field +from . import auth, filetree, hierarchy, store, uploads + +ARRIVED_SECONDS = 600 # how long a file's own name stays, after it was imported +PENDING_SECONDS = 600 # how long a file kept aside stays after it last changed +SWEEP_SECONDS = 60 +PROTOCOLS = ("s3", "webdav", "smb", "nfs") +# what an operating system or an app writes for itself, never imported: hidden files (.DS_Store, ._ AppleDouble, +# .~lock files), Office's ~$ owner files, Windows' Thumbs.db and desktop.ini, the Mac's Icon\r, NTFS streams (a:b), +# and files still arriving under a temporary name +JUNK = re.compile( + r"^(?:\..*|~\$.*|thumbs\.db|ehthumbs(?:_vista)?\.db|desktop\.ini|icon\r|.*:.*" + r"|.*\.(?:tmp|temp|part|partial|filepart|crdownload|download|swp|swx|lock))$", + re.IGNORECASE | re.DOTALL, +) + + +class Refused(PermissionError): + """Not theirs to add to, or no file can go there; says why.""" + + +class Rejected(ValueError): + """A file that can't be imported (its type, its size...); says why.""" + + +class TooLarge(Rejected): + """Over uploads.max_mb.""" + + +def accepting(cfg, protocol): + """Whether a share takes files people add: shares.add_files for S3 and WebDAV, smb.add_files, nfs.add_files.""" + section = "shares" if protocol in ("s3", "webdav") else protocol + return bool({**store.DEFAULTS[section], **(cfg.get(section) or {})}.get("add_files")) + + +def junk(name): + """Whether a file name is one an operating system or an app writes for itself (JUNK).""" + return bool(JUNK.match(name or "")) + + +def spool(cfg): + """A new file name on the uploads disk for a file to arrive in (not made yet); take() moves it from there.""" + return uploads.incoming(cfg) / f"share-{secrets.token_hex(12)}" + + +# ---------- what each person has just added ---------- +@dataclass +class Item: + kind: str # "arrived" (imported: `recording`) or "pending" (kept aside: `path`) + expires: float + recording: int | None = None + path: str | None = None + modified: str | None = None + md5: str | None = None # as the client sent it + mtime: str | None = None # the time the client gave it, as it wrote it (seconds since 1970) + holds: int = 0 # writers that have it open (SMB): it doesn't expire meanwhile + + +@dataclass +class Registry: + """{(account, folder's parts): {name: Item}}, in memory, guarded by a lock.""" + + items: dict[tuple[int, tuple[str, ...]], dict[str, Item]] = field(default_factory=dict) + lock: threading.Lock = field(default_factory=threading.Lock) + swept: float = 0.0 + + def folder(self, account, folder): + self.prune() + with self.lock: + return dict(self.items.get((account, tuple(folder)), {})) + + def get(self, account, folder, name): + return self.folder(account, folder).get(name) + + def put(self, account, folder, name, item): + with self.lock: + old = self.items.setdefault((account, tuple(folder)), {}).get(name) + self.items[(account, tuple(folder))][name] = item + if old and old.kind == "pending" and old.path and old.path != item.path: + _remove(old.path) + + def pop(self, account, folder, name): + with self.lock: + got = self.items.get((account, tuple(folder)), {}).pop(name, None) + if not self.items.get((account, tuple(folder)), True): + self.items.pop((account, tuple(folder)), None) + return got + + def hold(self, account, folder, name, by): + with self.lock: + it = self.items.get((account, tuple(folder)), {}).get(name) + if it is not None: + it.holds = max(0, it.holds + by) + it.expires = time.monotonic() + PENDING_SECONDS + + def prune(self): + now = time.monotonic() + gone = [] + with self.lock: + for key, names in list(self.items.items()): + for name, it in list(names.items()): + if it.expires < now and not it.holds: + gone.append(names.pop(name)) + if not names: + self.items.pop(key) + for it in gone: + if it.kind == "pending" and it.path: + _remove(it.path) + + def paths(self): + with self.lock: + return {it.path for names in self.items.values() for it in names.values() if it.path} + + +REGISTRY = Registry() + + +def _remove(path): + try: + os.unlink(path) + except OSError: + pass + + +def sweep(cfg): + """Files kept aside that this process no longer knows (it restarted) go from the uploads disk.""" + REGISTRY.prune() + if time.monotonic() - REGISTRY.swept < SWEEP_SECONDS: + return + REGISTRY.swept = time.monotonic() + keep, cut = REGISTRY.paths(), time.time() - 2 * max(PENDING_SECONDS, ARRIVED_SECONDS) + for p in uploads.incoming(cfg).glob("share-*"): + try: + if str(p) not in keep and p.stat().st_mtime < cut: + p.unlink() + except OSError: + pass + + +def _seconds(t): + """A time in seconds since 1970 (a number or its text) that's believable, or None.""" + try: + t = float(t) if t is not None else None + except ValueError: + return None + return t if t is not None and 0 < t < time.time() + 86400 else None + + +def _iso(t=None): + return dt.datetime.fromtimestamp(t if t is not None else time.time(), dt.timezone.utc).isoformat(timespec="seconds") + + +def overlay(tree, parts, taken): + """The files someone just added to a folder, as the tree shows them to them: named as they gave them, beside the + folder's own files (`taken`: their names, which win).""" + items = REGISTRY.folder(tree.viewer.account, parts) + if not items: + return [] + out = [] + arrived = {it.recording for it in items.values() if it.kind == "arrived"} + recs = ( + { + r["id"]: r + for r in tree.db.rows( + f"SELECT {filetree.REC_FIELDS} FROM recording WHERE id IN $ids", ids=[filetree.R("recording", int(x)) for x in arrived] + ) + } + if arrived + else {} + ) + for name, it in items.items(): + if name in taken: + continue + if it.kind == "arrived": + rec = recs.get(it.recording) + own = tree._primary(rec) if rec else None + if own is None: + continue + own.name, own.added, own.md5, own.mtime = name, "arrived", it.md5, it.mtime + out.append(own) + elif it.path and os.path.isfile(it.path): + ctype = mimetypes.guess_type(name)[0] or "application/octet-stream" + out.append(filetree.Entry(name, False, os.path.getsize(it.path), it.modified, ctype, path=it.path, added="pending", + md5=it.md5, mtime=it.mtime)) # fmt: skip + return out + + +# ---------- adding ---------- +@dataclass +class Place: + """Where a file is being written: its namespace, the collection it goes into (`collection`; None: the default), + or the collections still to make for it (`missing`, inside `parent`), its folder's parts and its name.""" + + space: int + namespace: str + folder: tuple[str, ...] + name: str + collection: int | None = None + parent: int | None = None + missing: list[str] = field(default_factory=list) + + +class Adder: + """One person adding files to the tree over one protocol (PROTOCOLS). `account`: their account row ({id, email, + admin}); `tree`: their view of the tree, when the server has one already.""" + + def __init__(self, db, cfg, account, protocol, tree=None): + self.db, self.cfg, self.account, self.protocol = db, cfg, account, protocol + self.tree = tree or filetree.Tree(db, cfg, filetree.Viewer.of(db, account)) + self.viewer: filetree.Viewer = self.tree.viewer + sweep(cfg) + + # ---------- where ---------- + def place(self, path, folder=False): + """Where a file (or, `folder`, a folder) at `path` goes. NotFound for a namespace or folder this person doesn't + see; Refused for a place nothing can be added.""" + parts = filetree.Tree.split(path) + if len(parts) < 2: + raise Refused("files go inside a namespace" if not folder else "namespaces are made in the web app") + tree = self.tree + sid = tree._space(parts[0]) + *up, name = parts + if name in (".", "..") or len(name.encode()) > 255: + raise Refused("that name can't be used") + top = up[1] if len(up) > 1 else None + where = Place(sid, parts[0], tuple(up), name) + if folder: + if top != filetree.COLLECTIONS: + raise Refused("folders can be made inside Collections only: each one is a collection") + self._walk(where, up[2:] + [name]) + return where + if top == filetree.COLLECTIONS: + self._walk(where, up[2:]) + elif top is not None and not self._view_folder(sid, top, up[2:]): + raise filetree.NotFound(path) + return where + + def _view_folder(self, sid, top, names): + """Whether Topics/… or Entities/… names a folder this person sees.""" + tree = self.tree + if top == filetree.TOPICS: + tid = None + for n in names[: filetree.TOPIC_DEPTH + 1]: + kids = tree._narrower(sid, tid) + if n not in kids: + return False + tid = kids[n] + return len(names) <= filetree.TOPIC_DEPTH + if top == filetree.ENTITIES: + if not names: + return True + types = tree._types(sid) + if names[0] not in types or len(names) > 2: + return False + return len(names) == 1 or names[1] in tree._entities(sid, types[names[0]]) + return False + + def _walk(self, where, names): + """Follow collection folders down from the top: the collection they reach, or the deepest one there is and the + names still to make inside it.""" + cid = None + for i, n in enumerate(names): + kids = self.tree._children(where.space, cid) + hit = kids.get(n) or next((c for k, c in kids.items() if k.casefold() == n.casefold()), None) + if hit is None: + where.parent, where.missing = cid, list(names[i:]) + return + cid = hit["id"] + where.collection = cid + + # ---------- may they ---------- + def may_import(self, sid, cid): + """The import permission in the namespace, or an editor's or admin's role on the collection.""" + if "import" in self.viewer.perms.get(sid, ()): + return True + role = self.viewer.collections.get(sid, {}).get(cid) if cid is not None else None + return hierarchy.ROLES.get(role or "", 0) >= auth.ROLES["editor"] + + def may_make(self, sid, parent): + """Editors of the namespace make collections anywhere in it; an admin of a collection, inside it.""" + return auth.allows(self.viewer.roles, sid, "editor") or ( + parent is not None and self.viewer.collections.get(sid, {}).get(parent) == "admin" + ) + + def _target(self, where): + return where.collection if where.collection is not None or where.missing else store.default_collection(self.db, where.space) + + def check(self, path, size=None): + """Whether this person may write a file at `path` (of `size` bytes, when known), before anything is sent: + the Place, or NotFound, Refused or Rejected.""" + where = self.place(path) + self._not_lens(where) + if where.missing: + if not self.may_make(where.space, where.parent): + raise Refused("making a collection needs editor access to the namespace, or admin of the collection it goes in") + if len(where.missing) + len(hierarchy.path(self.db, where.parent) if where.parent else []) > hierarchy.MAX_DEPTH: + raise Rejected(f"collections go at most {hierarchy.MAX_DEPTH} deep") + elif not self.may_import(where.space, self._target(where)): + raise Refused("adding files here needs the import permission (an editor's role)") + if not junk(where.name) and size: + try: + uploads.check(self.cfg, where.name, size) + except uploads.TooLarge as e: + raise TooLarge(str(e)) from None + except ValueError as e: + raise Rejected(str(e)) from None + return where + + def _not_lens(self, where): + """Files Lens made, and folders, can't be written over. Lens's files are all named `<title> [<id>].<ext>`.""" + if filetree.ASSET_ID.search(where.name): + try: + e = self.tree.stat("/".join([*where.folder, where.name])) + except filetree.NotFound: + return + if not e.added: + raise Refused("Lens made this file: it can't be written over") + return + top = where.folder[1] if len(where.folder) > 1 else None + if top is None: + folders = {filetree.COLLECTIONS, filetree.ENTITIES, filetree.TOPICS} + elif top == filetree.COLLECTIONS and not where.missing: + folders = set(self.tree._children(where.space, where.collection if len(where.folder) > 2 else None)) + else: + folders = set() + if where.name in folders: + raise Refused("that's a folder") + + def _make(self, where): + """The collections still missing on the way to `where`; the one it reaches.""" + cid = where.parent + for n in where.missing: + try: + cid = hierarchy.create(self.db, where.space, n, cid, by=self.account.get("email")) + except ValueError as e: + kids = self.tree._children(where.space, cid) + hit = next((c for k, c in kids.items() if k.casefold() == n.casefold()), None) # made meanwhile + if hit is None: + raise Rejected(str(e)) from None + cid = hit["id"] + else: + auth.audit(self.db, self._by(), "collection.create", f"collection:{cid}", {"name": n, "namespace": where.namespace, + "parent": where.parent, "via": self.protocol}) # fmt: skip + self.tree.forget() + where.collection, where.missing = cid, [] + return cid + + def _by(self): + return {"id": self.account["id"], "email": self.account.get("email")} + + # ---------- doing it ---------- + def folder(self, path): + """Make the collection a folder at `path` stands for (and any above it); whether one was made.""" + where = self.place(path, folder=True) + if not where.missing: + return False + if not self.may_make(where.space, where.parent): + raise Refused("making a collection needs editor access to the namespace, or admin of the collection it goes in") + self._make(where) + return True + + def touch(self, path, modified): + """Set the time of a file just added (an S3 client copying it onto itself with new metadata); False when it + isn't one.""" + parts = filetree.Tree.split(path) + if not parts or _seconds(modified) is None: + return False + with REGISTRY.lock: + it = REGISTRY.items.get((self.account["id"], tuple(parts[:-1])), {}).get(parts[-1]) + if it is None: + return False + it.mtime = str(modified) + return True + + def put(self, path, src, modified=None, md5=None): + """The file at `src` (on the uploads disk: spool(); it's moved away) was written at `path`: import it, or keep it + aside (junk or empty). `modified`: the file's own time in seconds since 1970 (a number, or the text the client + sent, which is shown back to it as it was), when the client said; `md5`: its MD5 (hex), when the server worked + it out. Returns {kind: "arrived", recording, duplicate} or {kind: "pending"}.""" + src = pathlib.Path(src) + stamp = str(modified) if modified is not None else None + modified = _seconds(modified) + where = self.check(path, src.stat().st_size) + key = (self.account["id"], where.folder, where.name) + if junk(where.name) or src.stat().st_size == 0: + if modified: + os.utime(src, (modified, modified)) + item = Item("pending", time.monotonic() + PENDING_SECONDS, path=str(src), modified=_iso(modified), md5=md5, mtime=stamp) + REGISTRY.put(*key, item) + return {"kind": "pending"} + # check() saw they may import there; whoever may make a collection may import into it + cid = self._make(where) if where.missing else self._target(where) + try: + up = uploads.take(self.db, self.cfg, where.namespace, str(src), where.name, self._by(), collection=cid, + modified=int(modified * 1000) if modified else None) # fmt: skip + except uploads.TooLarge as e: + raise TooLarge(str(e)) from None + except ValueError as e: + raise Rejected(str(e)) from None + finally: + _remove(src) # take() moved it unless it refused it + self.tree.forget() + rid = up.get("recording") + REGISTRY.put(*key, Item("arrived", time.monotonic() + ARRIVED_SECONDS, recording=rid, md5=md5, mtime=stamp)) + auth.audit(self.db, self._by(), "share.import", f"recording:{rid}", + {"via": self.protocol, "path": "/".join([*where.folder, where.name]), "duplicate": bool(up.get("duplicate"))}) # fmt: skip + return {"kind": "arrived", "recording": rid, "duplicate": bool(up.get("duplicate"))} + + def pending(self, path): + """A file kept aside at `path` for this person: its Item, or None.""" + parts = filetree.Tree.split(path) + if not parts: + return None + it = REGISTRY.get(self.account["id"], tuple(parts[:-1]), parts[-1]) + return it if it and it.kind == "pending" else None + + def delete(self, path): + """Delete a file kept aside (an app tidying up after itself); everything else is Refused.""" + parts = filetree.Tree.split(path) + it = self.pending(path) + if it is None: + raise Refused("files can be added here, not deleted") + REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) + if it.path: + _remove(it.path) + + def move(self, src, dst): + """Rename a file kept aside: to an ordinary name, it's imported then. Everything else is Refused.""" + it = self.pending(src) + if it is None or not it.path: + raise Refused("files can be added here, not moved or renamed") + parts = filetree.Tree.split(src) + moving = spool(self.cfg) + os.replace(it.path, moving) + REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) + try: + return self.put(dst, moving, os.stat(moving).st_mtime) + except BaseException: + if moving.exists(): # back where it was, so the client can try again + os.replace(moving, it.path) + REGISTRY.put(self.account["id"], tuple(parts[:-1]), parts[-1], it) + raise diff --git a/fastapi_backend/app/domain/shares.py b/fastapi_backend/app/domain/shares.py index 64fc2a314..30b293a59 100644 --- a/fastapi_backend/app/domain/shares.py +++ b/fastapi_backend/app/domain/shares.py @@ -1,5 +1,6 @@ -"""File shares (docs/file-tree.md): the servers that share the graph as files (filetree.py) on the network, read-only, with -people's own access keys (s3.py): S3 (s3.py) and WebDAV (webdav.py). +"""File shares (docs/file-tree.md): the servers that share the graph as files (filetree.py) on the network, with +people's own access keys (s3.py): S3 (s3.py) and WebDAV (webdav.py). Files people add through them are imported +(share_import.py) while `shares.add_files` is on. Off until an admin turns them on (`shares.enabled`, then `s3` and `webdav`); the process that runs routines (`lens worker`) follows the settings as they change, as the sensor hub does, and says what each server is doing in @@ -126,10 +127,12 @@ def start(db, cfg_fn, stop, log=None, name=None): def status(db, cfg): - """{protocol: {enabled, port, service}}: whether it's on, its port, and what the process running it says.""" + """{protocol: {enabled, port, service, add_files}}: whether it's on, its port, what the process running it says, + and whether files people add are imported.""" s, out = settings_of(cfg), {} for proto, (_, port) in PROTOCOLS.items(): row = db.one("SELECT * FROM $r", r=_row(proto)) or {} row.pop("id", None) - out[proto] = {"enabled": bool(s.get("enabled") and s.get(proto)), "port": int(s[port]), "service": row or None} + out[proto] = {"enabled": bool(s.get("enabled") and s.get(proto)), "port": int(s[port]), "service": row or None, + "add_files": bool(s.get("add_files"))} # fmt: skip return out diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 669133df4..1541ec2ed 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -555,8 +555,9 @@ def labels(): "account": None, }, # the graph as files over S3 and WebDAV (shares.py, docs/file-tree.md): off until an admin turns it on. Then the process - # that runs routines (`lens worker`) serves it, read-only, on these ports. bind is a startup setting only. - "shares": {"enabled": False, "bind": "0.0.0.0", "s3": True, "s3_port": 9555, "webdav": True, "webdav_port": 9556}, + # that runs routines (`lens worker`) serves it on these ports; add_files imports the files people add to it + # (share_import.py), off: read-only. bind is a startup setting only. + "shares": {"enabled": False, "bind": "0.0.0.0", "s3": True, "s3_port": 9555, "webdav": True, "webdav_port": 9556, "add_files": True}, # notifications to webhooks and Matterbridge (docs/notifications.md): targets reach public addresses only, and the # private networks listed here (a Matterbridge on the LAN or the Docker network); app_url is where links in messages # point (null: FRONTEND_URL) diff --git a/fastapi_backend/app/domain/uploads.py b/fastapi_backend/app/domain/uploads.py index dbe34b34e..77b2e17f6 100644 --- a/fastapi_backend/app/domain/uploads.py +++ b/fastapi_backend/app/domain/uploads.py @@ -143,15 +143,10 @@ def _home(db, sid, collection): return store.default_collection(db, sid) -def start(db, cfg, ns, filename, size, by, title=None, modified=None, attach=None, pipeline=None, collection=None, hold=False): - """A new upload into namespace `ns`, by `by` ({id, email}). Its name, type and size are checked, and the disk - must have room for it; nothing is in the archive until the last byte arrives. `modified` is the file's own time - (milliseconds since 1970), which dates the recording when its name doesn't. `attach` is a transcript-only - recording in `ns` the file becomes the audio of, instead of a recording of its own; `pipeline` runs instead of the - namespace's once a new recording is made; `collection` (one of the namespace's) is where it goes, else the - namespace's default. `hold` keeps it out of the archive once it's all here (state "held", no namespace yet), for - the assistant to place with `place` (a file dropped into a conversation).""" - sweep(db, cfg) +def check(cfg, filename, size, attach=None): + """The name a file is kept under, once its type and size are ones that can be uploaded here (ValueError, or + TooLarge, saying why not): what start() checks before anything is sent, for servers that ask before they read a + file (the file shares, share_import.py).""" u = cfg["uploads"] name = clean_name(filename) ext = pathlib.Path(name).suffix @@ -169,6 +164,19 @@ def start(db, cfg, ns, filename, size, by, title=None, modified=None, attach=Non raise ValueError("the file is empty") if size > u["max_mb"] * MB: raise TooLarge(f"files up to {u['max_mb']} MB") + return name + + +def start(db, cfg, ns, filename, size, by, title=None, modified=None, attach=None, pipeline=None, collection=None, hold=False): + """A new upload into namespace `ns`, by `by` ({id, email}). Its name, type and size are checked, and the disk + must have room for it; nothing is in the archive until the last byte arrives. `modified` is the file's own time + (milliseconds since 1970), which dates the recording when its name doesn't. `attach` is a transcript-only + recording in `ns` the file becomes the audio of, instead of a recording of its own; `pipeline` runs instead of the + namespace's once a new recording is made; `collection` (one of the namespace's) is where it goes, else the + namespace's default. `hold` keeps it out of the archive once it's all here (state "held", no namespace yet), for + the assistant to place with `place` (a file dropped into a conversation).""" + sweep(db, cfg) + name = check(cfg, filename, size, attach) folder = _dir(cfg) / ".partial" folder.mkdir(parents=True, exist_ok=True) if shutil.disk_usage(folder).free < size + ROOM: @@ -434,12 +442,14 @@ def _attach(db, cfg, rid, sid, dest, st, fp, by, probed): return jobs.add_steps(db, rid, ATTACH_STEPS, by=by) -def take(db, cfg, ns, path, filename, by, title=None, pipeline=None, collection=None, hold=False): - """A whole file that arrived some other way (an import webhook, a chat channel), at `path` on the same disk as the - uploads folder, uploaded into namespace `ns` in one go: it's checked as start() checks a file, moved into place - and finished as if its last chunk had just arrived. The namespace must exist; with `hold` there's none yet (ns ""): - it's held for the assistant to place, as a file dropped into a conversation is. Returns the upload.""" - row = start(db, cfg, ns, filename, os.path.getsize(path), by, title=title, pipeline=pipeline, collection=collection, hold=hold) +def take(db, cfg, ns, path, filename, by, title=None, pipeline=None, collection=None, hold=False, modified=None): + """A whole file that arrived some other way (an import webhook, a chat channel, a file share), at `path` on the same + disk as the uploads folder, uploaded into namespace `ns` in one go: it's checked as start() checks a file, moved + into place and finished as if its last chunk had just arrived. The namespace must exist; with `hold` there's none + yet (ns ""): it's held for the assistant to place, as a file dropped into a conversation is. `modified` is the + file's own time, as for start(). Returns the upload.""" + size = os.path.getsize(path) + row = start(db, cfg, ns, filename, size, by, title=title, modified=modified, pipeline=pipeline, collection=collection, hold=hold) os.replace(path, _part(cfg, row["id"])) try: return received(db, cfg, row["id"]) diff --git a/fastapi_backend/app/domain/webdav.py b/fastapi_backend/app/domain/webdav.py index 4e3ec99d1..72f39eff1 100644 --- a/fastapi_backend/app/domain/webdav.py +++ b/fastapi_backend/app/domain/webdav.py @@ -1,9 +1,13 @@ -"""The asset tree over WebDAV (docs/file-tree.md#webdav): a read-only WebDAV server (RFC 4918, class 1) that Finder, -Windows Explorer, GNOME Files, Dolphin, rclone and davfs2 mount as a drive. +"""The asset tree over WebDAV (docs/file-tree.md#webdav): a WebDAV server (RFC 4918) that Finder, Windows Explorer, +GNOME Files, Dolphin, rclone and davfs2 mount as a drive. Each person signs in with one of their access keys (s3.py): the key id as the user name and its secret as the password (HTTP Basic). They see the namespaces and collections the web app shows them. OPTIONS, PROPFIND (depth 0 -or 1), GET and HEAD (with Range); anything that writes answers 403. Run by shares.py. +or 1), GET and HEAD (with Range). While `shares.add_files` is on, a file put into the tree is imported +(share_import.py): PUT (whole or chunked), MKCOL under Collections makes a collection, DELETE and MOVE act on what +share_import.py keeps aside (an app's own files, or a file uploaded under a temporary name and then renamed), and +LOCK, UNLOCK and PROPPATCH answer as clients expect (class 2: Finder and Windows mount a class 1 server read-only) +without storing anything. Anything else that writes answers 403. Run by shares.py. """ from __future__ import annotations @@ -11,17 +15,27 @@ import base64 import datetime as dt import email.utils +import errno import http.server import re +import secrets import urllib.parse +import xml.etree.ElementTree as ET from typing import Any from xml.sax.saxutils import escape -from . import auth, filetree, keyring, s3 +from . import auth, filetree, http_body, keyring, s3, share_import, uploads ALLOW = "OPTIONS, GET, HEAD, PROPFIND" +ALLOW_ADD = ALLOW + ", PUT, MKCOL, DELETE, MOVE, LOCK, UNLOCK, PROPPATCH" REALM = 'Basic realm="Lens", charset="UTF-8"' MAX_BODY = 1 << 20 +LOCK_SECONDS = 3600 + + +def _attr(s): + """Text for a double-quoted XML attribute.""" + return escape(s, {'"': """}) def _http_date(s): @@ -70,11 +84,14 @@ class Handler(http.server.BaseHTTPRequestHandler): def log_message(self, fmt, *args): pass + def _adds(self): + return share_import.accepting(self.server.cfg_fn(), "webdav") + def _send(self, status, body=b"", ctype="text/plain; charset=utf-8", headers=None): self.send_response(status) self.send_header("Content-Type", ctype) self.send_header("Content-Length", str(len(body))) - self.send_header("DAV", "1") + self.send_header("DAV", "1, 2" if self._adds() else "1") for k, v in (headers or {}).items(): self.send_header(k, v) self.end_headers() @@ -114,13 +131,213 @@ def _parts(self): def do_OPTIONS(self): # noqa: N802 - http.server's names self._drain() - self._send(200, headers={"Allow": ALLOW, "MS-Author-Via": "DAV"}) + self._send(200, headers={"Allow": ALLOW_ADD if self._adds() else ALLOW, "MS-Author-Via": "DAV"}) def _write(self): self._drain() self._send(403, b"the file tree is read-only\n", headers={"Allow": ALLOW}) - do_PUT = do_POST = do_DELETE = do_MKCOL = do_MOVE = do_COPY = do_PROPPATCH = do_LOCK = do_UNLOCK = _write # noqa: N815 + do_POST = do_COPY = _write # noqa: N815 + + def handle_expect_100(self): + """Hold back 100 Continue until the upload is known to be allowed (_go_on).""" + return True + + def _go_on(self): + if self.headers.get("expect", "").lower() == "100-continue": + self.send_response_only(100) + self.end_headers() + + def _adding(self): + """(the account, its Adder) for a request that writes, or None (an answer was sent).""" + if not self._adds(): + self._write() + return None + acct = self._account() + if acct is None: + self._skip() + return None + db = self.server.db + return acct, share_import.Adder(db, self.server.cfg_fn(), acct, "webdav", self._tree(acct)) + + def _skip(self): + """Don't read a body that's still to come: drop a small one, else close the connection after the answer.""" + if not http_body.drain(self.rfile, {k.lower(): v for k, v in self.headers.items()}): + self.close_connection = True + + def _refused(self, e): + """The answer for something share_import.py or the body reader turned down.""" + if isinstance(e, share_import.Refused): # a PermissionError (as keyring.Locked is), so before OSError + return self._send(403, f"{e}\n".encode()) + if isinstance(e, http_body.BodyError): + return self._send(413 if e.code == "EntityTooLarge" else e.status, f"{e.message}\n".encode()) + if isinstance(e, share_import.TooLarge): + return self._send(413, f"{e}\n".encode()) + if isinstance(e, share_import.Rejected): + return self._send(415, f"{e}\n".encode()) + if isinstance(e, filetree.NotFound): + return self._send(409, b"no such folder: files go in a namespace, or in Collections/...\n") + if isinstance(e, keyring.Locked): + return self._send(403, b"this namespace is a locked vault: an owner unlocks it in the web app\n") + if isinstance(e, OSError) and e.errno == errno.ENOSPC: + return self._send(507, b"the server doesn't have room for this file\n") + if isinstance(e, OSError): + self.server.log(f"webdav: {type(e).__name__}: {e}") + return self._send(500, b"something went wrong on the server\n") + return self._send(403, f"{e}\n".encode()) + + def do_PUT(self): # noqa: N802 + got = self._adding() + if got is None: + return + _, adder = got + headers = {k.lower(): v for k, v in self.headers.items()} + path = "/".join(self._parts()) + dest = share_import.spool(adder.cfg) + declared = headers.get("content-length") or headers.get("x-expected-entity-length") or "" + try: + try: + before = adder.tree.stat(path) + except filetree.NotFound: + before = None + adder.check(path, int(declared) if declared.isdigit() else None) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as e: + self._skip() + return self._refused(e) + try: + self._go_on() + if "chunked" in headers.get("transfer-encoding", "").lower(): + self.close_connection = True # a chunked body may have been read past its end + limit = int(adder.cfg["uploads"]["max_mb"]) * uploads.MB + _, md5 = http_body.receive(self.rfile, headers, dest, limit) + adder.put(path, dest, headers.get("x-oc-mtime"), md5) # imported (moved away), or kept aside there + except (share_import.Refused, share_import.Rejected, filetree.NotFound, http_body.BodyError, OSError) as e: + share_import._remove(dest) + if isinstance(e, http_body.BodyError): + self.close_connection = True + try: + return self._refused(e) + except (BrokenPipeError, ConnectionResetError): # the client went away mid-file + return None + self._send(204 if before is not None else 201, headers={"ETag": f'"{md5}"'}) + + def do_MKCOL(self): # noqa: N802 + got = self._adding() + if got is None: + return + _, adder = got + if int(self.headers.get("content-length") or 0) or self.headers.get("transfer-encoding"): + self._skip() + return self._send(415, b"MKCOL takes no body\n") + try: + made = adder.folder("/".join(self._parts())) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as e: + return self._refused(e) + self._send(201 if made else 405, b"" if made else b"there's a folder there already\n") + + def do_DELETE(self): # noqa: N802 + got = self._adding() + if got is None: + return + self._drain() + _, adder = got + path = "/".join(self._parts()) + try: + adder.delete(path) + except share_import.Refused as e: + try: + adder.tree.stat(path) + except filetree.NotFound: + return self._send(404, b"not found\n") + return self._refused(e) + self._send(204) + + def do_MOVE(self): # noqa: N802 + got = self._adding() + if got is None: + return + self._drain() + _, adder = got + dest = urllib.parse.urlsplit(self.headers.get("destination", "")).path + to = "/".join(urllib.parse.unquote(p) for p in dest.split("/") if p) + if not to: + return self._send(400, b"say where to move it (Destination)\n") + try: + existed = adder.tree.stat(to) is not None + except filetree.NotFound: + existed = False + if existed and self.headers.get("overwrite", "T").upper() == "F": + return self._send(412, b"there's a file there already\n") + try: + adder.move("/".join(self._parts()), to) + except (share_import.Refused, share_import.Rejected, filetree.NotFound, OSError) as e: + return self._refused(e) + self._send(204 if existed else 201) + + def do_LOCK(self): # noqa: N802 + """A lock that holds nothing back (one person's own files, nobody else writes them): what Finder and Windows + ask for before they write. On a name that isn't there yet it makes an empty file, as RFC 4918 says.""" + got = self._adding() + if got is None: + return + self._drain() + _, adder = got + path = "/".join(self._parts()) + status = 200 + try: + adder.tree.stat(path) + except filetree.NotFound: + empty = share_import.spool(adder.cfg) + try: + empty.touch() + adder.put(path, empty) + except (share_import.Refused, share_import.Rejected, filetree.NotFound, OSError) as e: + share_import._remove(empty) + return self._refused(e) + status = 201 + token = self.headers.get("if", "").strip("()<> ") # a refresh names the lock it renews + token = token if re.fullmatch(r"opaquelocktoken:[0-9a-f-]{1,64}", token) else f"opaquelocktoken:{secrets.token_hex(16)}" + depth = "0" if self.headers.get("depth", "infinity") == "0" else "infinity" + body = ( + '<?xml version="1.0" encoding="utf-8"?>\n<D:prop xmlns:D="DAV:"><D:lockdiscovery><D:activelock>' + "<D:locktype><D:write/></D:locktype><D:lockscope><D:exclusive/></D:lockscope>" + f"<D:depth>{depth}</D:depth><D:owner>{escape(str(got[0].get('email') or ''))}</D:owner>" + f"<D:timeout>Second-{LOCK_SECONDS}</D:timeout><D:locktoken><D:href>{token}</D:href></D:locktoken>" + f"<D:lockroot><D:href>{escape(href(self._parts(), False))}</D:href></D:lockroot>" + "</D:activelock></D:lockdiscovery></D:prop>" + ).encode() + self._send(status, body, "application/xml; charset=utf-8", headers={"Lock-Token": f"<{token}>"}) + + def do_UNLOCK(self): # noqa: N802 + if self._adding() is None: + return + self._drain() + self._send(204) + + def do_PROPPATCH(self): # noqa: N802 + """Properties aren't kept: say each one was set, so clients that set times after a PUT (Windows) carry on.""" + got = self._adding() + if got is None: + return + n = int(self.headers.get("content-length") or 0) + raw = self.rfile.read(n) if 0 < n <= MAX_BODY else (self._drain() or b"") + try: + if b"<!DOCTYPE" in raw or b"<!ENTITY" in raw: + raise ET.ParseError("no DTDs") + names = [p.tag for x in ET.fromstring(raw).iter() if x.tag == "{DAV:}prop" for p in x] if raw else [] + except ET.ParseError: + return self._send(400, b"PROPPATCH takes an XML body\n") + try: + got[1].tree.stat("/".join(self._parts())) + except filetree.NotFound: + return self._send(404, b"not found\n") + props = "".join(f'<x:{t.split("}")[-1]} xmlns:x="{_attr(t[1:].split("}")[0] if "}" in t else "")}"/>' for t in names) + body = ( + f'<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:"><D:response>' + f"<D:href>{escape(href(self._parts(), False))}</D:href><D:propstat><D:prop>{props}</D:prop>" + "<D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>" + ).encode() + self._send(207, body, "application/xml; charset=utf-8") def _tree(self, acct): db = self.server.db diff --git a/fastapi_backend/app/schemas/filetree.py b/fastapi_backend/app/schemas/filetree.py index b1be7cb74..45185f1e5 100644 --- a/fastapi_backend/app/schemas/filetree.py +++ b/fastapi_backend/app/schemas/filetree.py @@ -54,6 +54,7 @@ class ShareServer(ResponseModel): enabled: bool = Field(description="whether an admin turned it on (shares.enabled and its own switch)") port: int = Field(description="the port it listens on in the worker (Docker publishes LENS_S3_PORT, LENS_WEBDAV_PORT)") service: S3Service | None = Field(None, description="what the process running it says; none while it's off") + add_files: bool = Field(False, description="whether files people add through it are imported (shares.add_files)") class S3Status(ShareServer): diff --git a/fastapi_backend/tests/api/test_s3_writes.py b/fastapi_backend/tests/api/test_s3_writes.py new file mode 100644 index 000000000..b4413f79b --- /dev/null +++ b/fastapi_backend/tests/api/test_s3_writes.py @@ -0,0 +1,276 @@ +"""Adding files over S3 (docs/file-tree.md#adding-files): a real S3 client (botocore) puts files into the tree and +they're imported where the path says, as an upload is; what may not be added is refused before it's sent.""" + +from __future__ import annotations + +import datetime as dt +import hashlib +import http.client +import io + +import botocore.session +import pytest +from botocore.config import Config +from botocore.exceptions import ClientError + +from app.domain import hierarchy, http_body, s3, share_import, shares, store +from tests.helpers import login, make_user, seed, text_pdf + +PDF = text_pdf([["Harbour minutes", "The lamp was lit at dusk."]]) + + +@pytest.fixture +def env(client, db, cfg, folder): + seed(db, cfg, folder) + pods = store.ns_id(db, "pods") + talks = hierarchy.create(db, pods, "Talks") + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + conf = {"cfg": cfg} + server = shares.Server("127.0.0.1", 0, db, lambda: conf["cfg"], log=print).start() + keys = {} + for who in ("ed", "vi"): + h = login(client, f"{who}@x.io", f"{'editor' if who == 'ed' else 'viewer'} password 1") + k = client.post("/api/v1/shares/s3/keys", headers=h, json={}).json() + keys[who] = (k["id"], k["secret"]) + share_import.REGISTRY.items.clear() + yield {"port": server.port, "keys": keys, "talks": talks, "pods": pods, "conf": conf} + server.stop() + + +def _s3(env, who="ed", **cfg): + kid, secret = env["keys"][who] + return botocore.session.get_session().create_client( + "s3", + endpoint_url=f"http://127.0.0.1:{env['port']}", + region_name="us-east-1", + aws_access_key_id=kid, + aws_secret_access_key=secret, + config=Config(signature_version="s3v4", s3={"addressing_style": "path"}, retries={"max_attempts": 1}, **cfg), + ) + + +def _code(fn): + with pytest.raises(ClientError) as e: + fn() + return e.value.response["Error"]["Code"], e.value.response["Error"].get("Message") + + +def _recording(db, title): + return db.one("SELECT record::id(id) AS id, collection, source, size, recorded_at FROM recording WHERE title = $t", t=title) + + +def test_a_file_put_into_a_collection_is_imported_there(client, db, env): + c = _s3(env) + put = c.put_object(Bucket="pods", Key="Collections/Talks/minutes.pdf", Body=PDF, Metadata={"mtime": "1700000000.5"}) + assert put["ETag"] == f'"{hashlib.md5(PDF).hexdigest()}"' + rec = _recording(db, "minutes") + assert rec["collection"] == env["talks"] and rec["source"] == "document" and rec["size"] == len(PDF) + assert rec["recorded_at"].startswith("2023-11-14") # the file's own time, from rclone's mtime + assert db.one("SELECT VALUE id FROM job WHERE recording = $r LIMIT 1", r=rec["id"]) # the namespace's pipeline is queued + # the client finds it under the name it gave, beside the asset's own files + head = c.head_object(Bucket="pods", Key="Collections/Talks/minutes.pdf") + assert head["ContentLength"] == len(PDF) + assert c.get_object(Bucket="pods", Key="Collections/Talks/minutes.pdf")["Body"].read() == PDF + keys = [o["Key"] for o in c.list_objects_v2(Bucket="pods", Prefix="Collections/Talks/")["Contents"]] + assert "Collections/Talks/minutes.pdf" in keys and f"Collections/Talks/minutes [{rec['id']}].pdf" in keys + # only for them: others see the asset as Lens names it + others = [o["Key"] for o in _s3(env, "vi").list_objects_v2(Bucket="pods", Prefix="Collections/Talks/")["Contents"]] + assert "Collections/Talks/minutes.pdf" not in others and f"Collections/Talks/minutes [{rec['id']}].pdf" in others + audit = db.one("SELECT * FROM audit_log WHERE action = 'share.import'") + assert audit["target"] == f"recording:{rec['id']}" and audit["detail"]["via"] == "s3" and audit["email"] == "ed@x.io" + # the same file again isn't added twice + c.put_object(Bucket="pods", Key="Collections/Talks/again.pdf", Body=PDF) + assert len(db.values("SELECT VALUE id FROM recording WHERE source = 'document'")) == 1 + assert c.head_object(Bucket="pods", Key="Collections/Talks/again.pdf")["ContentLength"] == len(PDF) + + +def test_folders_become_collections(db, env): + c = _s3(env) + c.put_object(Bucket="pods", Key="Collections/Trips/Lisbon 2026/minutes.pdf", Body=PDF) + trips = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Trips") + lisbon = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Lisbon 2026") + assert lisbon["parent"] == trips["id"] and _recording(db, "minutes")["collection"] == lisbon["id"] + # a folder marker makes an empty one; a name differing only in case is the same collection + c.put_object(Bucket="pods", Key="Collections/trips/Porto/", Body=b"") + porto = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Porto") + assert porto["parent"] == trips["id"] + assert db.one("SELECT * FROM audit_log WHERE action = 'collection.create' AND detail.name = 'Porto'")["detail"]["via"] == "s3" + # anywhere else in the namespace: its default collection + c.put_object(Bucket="pods", Key="Topics/minutes 2.pdf", Body=PDF + b"\n%2") + assert _recording(db, "minutes 2")["collection"] == store.default_collection(db, env["pods"]) + assert _code(lambda: c.put_object(Bucket="pods", Key="Nowhere/x.pdf", Body=PDF))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="pods", Key="Topics/Made up/", Body=b""))[0] == "AccessDenied" + + +def test_what_may_not_be_added_is_refused(db, cfg, env): + c = _s3(env) + assert _code(lambda: _s3(env, "vi").put_object(Bucket="pods", Key="Collections/Talks/x.pdf", Body=PDF))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="calls", Key="x.pdf", Body=PDF))[0] == "NoSuchBucket" + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/General/ep1 [1].json", Body=b"{}"))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/Talks", Body=PDF))[0] == "AccessDenied" # a folder + code, msg = _code(lambda: c.put_object(Bucket="pods", Key="tool.exe", Body=b"MZ")) + assert code == "InvalidArgument" and "can't be uploaded here" in msg + assert _code(lambda: c.create_bucket(Bucket="new-ns"))[0] == "AccessDenied" + assert ( + _code(lambda: c.copy_object(Bucket="pods", Key="b.pdf", CopySource="pods/Collections/General/ep1 [1].json"))[0] == "NotImplemented" + ) + assert _code(lambda: c.delete_object(Bucket="pods", Key="Collections/General/ep1 [1].json"))[0] == "AccessDenied" + c.delete_object(Bucket="pods", Key="Collections/General/nothing.pdf") # nothing there: done + env["conf"]["cfg"] = {**cfg, "uploads": {**cfg["uploads"], "max_mb": 1}} + assert _code(lambda: c.put_object(Bucket="pods", Key="big.pdf", Body=PDF + b"\0" * (2 << 20)))[0] == "EntityTooLarge" + env["conf"]["cfg"] = {**cfg, "shares": {**cfg["shares"], "add_files": False}} + code, msg = _code(lambda: c.put_object(Bucket="pods", Key="x.pdf", Body=PDF)) + assert code == "AccessDenied" and "read-only" in msg + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + + +def test_files_an_os_writes_for_itself_and_empty_files_are_kept_aside(db, env): + c = _s3(env) + c.put_object(Bucket="pods", Key="Collections/Talks/.DS_Store", Body=b"Bud1") + c.put_object(Bucket="pods", Key="Collections/Talks/report.pdf", Body=b"") + assert c.get_object(Bucket="pods", Key="Collections/Talks/.DS_Store")["Body"].read() == b"Bud1" + assert c.head_object(Bucket="pods", Key="Collections/Talks/report.pdf")["ContentLength"] == 0 + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + c.put_object(Bucket="pods", Key="Collections/Talks/report.pdf", Body=PDF) # then its bytes + assert _recording(db, "report")["collection"] == env["talks"] + out = c.delete_objects( + Bucket="pods", Delete={"Objects": [{"Key": "Collections/Talks/.DS_Store"}, {"Key": "Collections/Talks/ep9.pdf"}]} + ) + assert sorted(d["Key"] for d in out["Deleted"]) == ["Collections/Talks/.DS_Store", "Collections/Talks/ep9.pdf"] + assert _code(lambda: c.head_object(Bucket="pods", Key="Collections/Talks/.DS_Store"))[0] == "404" + # kept aside for the person who wrote them only, for a while + assert "Collections/Talks/report.pdf" not in [o["Key"] for o in _s3(env, "vi").list_objects_v2(Bucket="pods")["Contents"]] + for names in share_import.REGISTRY.items.values(): + for it in names.values(): + it.expires = 0 + assert "Collections/Talks/report.pdf" not in [o["Key"] for o in c.list_objects_v2(Bucket="pods")["Contents"]] + + +def test_multipart_uploads(db, env): + c = _s3(env) + body = PDF + b"%" * (6 << 20) + up = c.create_multipart_upload(Bucket="pods", Key="Collections/Talks/long.pdf") + uid = up["UploadId"] + assert [u["UploadId"] for u in c.list_multipart_uploads(Bucket="pods").get("Uploads", [])] == [uid] + e1 = c.upload_part(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid, PartNumber=1, Body=body[: 5 << 20])["ETag"] + e2 = c.upload_part(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid, PartNumber=2, Body=body[5 << 20 :])["ETag"] + assert [p["PartNumber"] for p in c.list_parts(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid)["Parts"]] == [1, 2] + # not someone else's, and not out of order + assert _code(lambda: _s3(env, "vi").abort_multipart_upload(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid))[0] in ( + "AccessDenied", + "NoSuchUpload", + ) + wrong = {"Parts": [{"PartNumber": 2, "ETag": e2}, {"PartNumber": 1, "ETag": e1}]} + assert ( + _code(lambda: c.complete_multipart_upload(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid, MultipartUpload=wrong))[0] + == "InvalidPartOrder" + ) + done = c.complete_multipart_upload( + Bucket="pods", + Key="Collections/Talks/long.pdf", + UploadId=uid, + MultipartUpload={"Parts": [{"PartNumber": 1, "ETag": e1}, {"PartNumber": 2, "ETag": e2}]}, + ) + assert done["ETag"].endswith('-2"') + rec = _recording(db, "long") + assert rec["collection"] == env["talks"] and rec["size"] == len(body) + assert c.get_object(Bucket="pods", Key="Collections/Talks/long.pdf", Range="bytes=0-4")["Body"].read() == b"%PDF-" + assert _code(lambda: c.list_parts(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid))[0] == "NoSuchUpload" + # an upload given up goes + up = c.create_multipart_upload(Bucket="pods", Key="Collections/Talks/gone.pdf")["UploadId"] + c.upload_part(Bucket="pods", Key="Collections/Talks/gone.pdf", UploadId=up, PartNumber=1, Body=PDF) + c.abort_multipart_upload(Bucket="pods", Key="Collections/Talks/gone.pdf", UploadId=up) + assert not c.list_multipart_uploads(Bucket="pods").get("Uploads") + + +def _signed_put(env, key, chunks, payload="STREAMING-AWS4-HMAC-SHA256-PAYLOAD", spoil=False, trailer=None): + """PUT `chunks` aws-chunked by hand, signed as SigV4 streaming uploads are (minio-go and the AWS SDKs do this).""" + kid, secret = env["keys"]["ed"] + now = dt.datetime.now(dt.UTC) + amz, date = now.strftime("%Y%m%dT%H%M%SZ"), now.strftime("%Y%m%d") + scope = f"{date}/us-east-1/s3/aws4_request" + host = f"127.0.0.1:{env['port']}" + path = s3.canonical_path(f"/pods/{key}") + size = sum(len(c) for c in chunks) + plain = not payload.startswith("STREAMING-") + headers = {"host": host, "x-amz-content-sha256": payload, "x-amz-date": amz} + if not plain: + headers.update({"x-amz-decoded-content-length": str(size), "content-encoding": "aws-chunked"}) + if trailer: + headers["x-amz-trailer"] = trailer[0] + names = sorted(headers) + canonical = "\n".join(["PUT", path, "", "".join(f"{n}:{headers[n]}\n" for n in names), ";".join(names), payload]) + to_sign = "\n".join(["AWS4-HMAC-SHA256", amz, scope, hashlib.sha256(canonical.encode()).hexdigest()]) + seed = s3.signature(secret, date, "us-east-1", "s3", to_sign) + signer = http_body.Signed(s3.signing_key(secret, date, "us-east-1", "s3"), amz, scope, seed) + body = io.BytesIO() + for c in [] if plain else [*chunks, b""]: + if payload.startswith("STREAMING-AWS4"): + sig = hmac_chunk(signer, c) + if spoil and c: + sig = "0" * 64 + body.write(f"{len(c):x};chunk-signature={sig}\r\n".encode()) + else: + body.write(f"{len(c):x}\r\n".encode()) + if c: + body.write(c + b"\r\n") + if trailer: + body.write(f"{trailer[0]}:{trailer[1]}\r\n".encode()) + body.write(b"".join(chunks) if plain else b"\r\n") + headers["authorization"] = f"AWS4-HMAC-SHA256 Credential={kid}/{scope},SignedHeaders={';'.join(names)},Signature={seed}" + conn = http.client.HTTPConnection("127.0.0.1", env["port"]) + conn.request("PUT", path, body=body.getvalue(), headers=headers) + r = conn.getresponse() + return r.status, r.read() + + +def hmac_chunk(signer, chunk): + import hmac + + to_sign = "\n".join(["AWS4-HMAC-SHA256-PAYLOAD", signer.amz, signer.scope, signer.prev, http_body.EMPTY_SHA256, + hashlib.sha256(chunk).hexdigest()]) # fmt: skip + signer.prev = hmac.new(signer.key, to_sign.encode(), hashlib.sha256).hexdigest() + return signer.prev + + +def test_streaming_uploads_have_their_chunks_checked(db, env): + status, _ = _signed_put(env, "Collections/Talks/streamed.pdf", [PDF[:100], PDF[100:]]) + assert status == 200 and _recording(db, "streamed")["size"] == len(PDF) + status, body = _signed_put(env, "Collections/Talks/spoilt.pdf", [PDF[:100], PDF[100:]], spoil=True) + assert status == 403 and b"SignatureDoesNotMatch" in body and not _recording(db, "spoilt") + # unsigned chunks with a CRC32 trailer (the AWS SDKs' default for uploads) + import base64 + import zlib + + other = PDF + b"\n%3" + crc = base64.b64encode(zlib.crc32(other).to_bytes(4, "big")).decode() + status, _ = _signed_put( + env, "Collections/Talks/trailed.pdf", [other], "STREAMING-UNSIGNED-PAYLOAD-TRAILER", trailer=("x-amz-checksum-crc32", crc) + ) + assert status == 200 and _recording(db, "trailed") + status, body = _signed_put( + env, "Collections/Talks/bad crc.pdf", [other + b"x"], "STREAMING-UNSIGNED-PAYLOAD-TRAILER", trailer=("x-amz-checksum-crc32", crc) + ) + assert status == 400 and b"BadDigest" in body + + +def test_a_body_that_isnt_what_was_signed_is_refused(db, env): + status, body = _signed_put(env, "Collections/Talks/x.pdf", [PDF], hashlib.sha256(b"something else").hexdigest()) + assert status == 400 and b"XAmzContentSHA256Mismatch" in body and not _recording(db, "x") + status, _ = _signed_put(env, "Collections/Talks/y.pdf", [PDF], hashlib.sha256(PDF).hexdigest()) + assert status == 200 and _recording(db, "y") + + +def test_people_with_a_role_on_a_collection_add_there_only(client, db, env): + uid = make_user(db, "co@x.io", "collection password 1") + hierarchy.give(db, env["talks"], uid, "admin") + h = login(client, "co@x.io", "collection password 1") + k = client.post("/api/v1/shares/s3/keys", headers=h, json={}).json() + env["keys"]["co"] = (k["id"], k["secret"]) + c = _s3(env, "co") + c.put_object(Bucket="pods", Key="Collections/Talks/Day 1/minutes.pdf", Body=PDF) + day = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Day 1") + assert day["parent"] == env["talks"] and _recording(db, "minutes")["collection"] == day["id"] + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/Other/x.pdf", Body=PDF))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="pods", Key="x.pdf", Body=PDF))[0] == "AccessDenied" # the default collection isn't theirs diff --git a/fastapi_backend/tests/api/test_webdav.py b/fastapi_backend/tests/api/test_webdav.py index 45678effb..ea82f6958 100644 --- a/fastapi_backend/tests/api/test_webdav.py +++ b/fastapi_backend/tests/api/test_webdav.py @@ -47,7 +47,7 @@ def _listing(body): def test_mount_and_read(env): status, h, _ = _call(env, "OPTIONS", "/", auth_header=False) - assert status == 200 and h["DAV"] == "1" and "PROPFIND" in h["Allow"] + assert status == 200 and h["DAV"] == "1, 2" and "PROPFIND" in h["Allow"] and "PUT" in h["Allow"] status, _, body = _call(env, "PROPFIND", "/", {"Depth": "1"}) assert status == 207 and [(x[0], x[1]) for x in _listing(body)] == [("/", True), ("/pods/", True)] status, _, body = _call(env, "PROPFIND", "/pods/Collections/General/", {"Depth": "1"}) @@ -65,6 +65,24 @@ def test_mount_and_read(env): assert status == 200 and h["Content-Length"] == str(size) and body == b"" +def test_read_only_when_files_are_not_taken(db, cfg, folder): + seed(db, cfg, folder) + uid = make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + from app.domain import s3 + + kid, secret = s3.create_key(db, cfg, uid) + off = {**cfg, "shares": {**cfg["shares"], "add_files": False}} + server = shares.Server("127.0.0.1", 0, db, lambda: off, handler=webdav.Handler).start() + env = {"port": server.port, "auth": "Basic " + base64.b64encode(f"{kid}:{secret}".encode()).decode()} + try: + status, h, _ = _call(env, "OPTIONS", "/") + assert h["DAV"] == "1" and "PUT" not in h["Allow"] + assert _call(env, "PUT", "/pods/Collections/General/x.pdf", body=b"%PDF-1.4")[0] == 403 + assert _call(env, "LOCK", "/pods/Collections/General/x.pdf")[0] == 403 + finally: + server.stop() + + def test_what_webdav_refuses(env): assert _call(env, "PROPFIND", "/", {"Depth": "1"}, auth_header=False)[0] == 401 assert _call(env, "PROPFIND", "/calls/", {"Depth": "1"})[0] == 404 # not theirs diff --git a/fastapi_backend/tests/api/test_webdav_writes.py b/fastapi_backend/tests/api/test_webdav_writes.py new file mode 100644 index 000000000..5abf3887b --- /dev/null +++ b/fastapi_backend/tests/api/test_webdav_writes.py @@ -0,0 +1,157 @@ +"""Adding files over WebDAV (docs/file-tree.md#adding-files): what Finder, Windows Explorer and rclone send when +they copy files and folders onto the drive, and the files are imported where they were put.""" + +from __future__ import annotations + +import base64 +import http.client +import xml.etree.ElementTree as ET + +import pytest + +from app.domain import auth, hierarchy, share_import, shares, store, webdav +from tests.helpers import login, make_user, seed, text_pdf + +D = "{DAV:}" +PDF = text_pdf([["Harbour minutes", "The lamp was lit at dusk."]]) + + +@pytest.fixture +def env(client, db, cfg, folder): + seed(db, cfg, folder) + pods = store.ns_id(db, "pods") + talks = hierarchy.create(db, pods, "Talks") + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + keys = {} + for who, pw in (("ed", "editor password 1"), ("vi", "viewer password 1")): + k = client.post("/api/v1/shares/s3/keys", headers=login(client, f"{who}@x.io", pw), json={}).json() + keys[who] = "Basic " + base64.b64encode(f"{k['id']}:{k['secret']}".encode()).decode() + conf = {"cfg": cfg} + server = shares.Server("127.0.0.1", 0, db, lambda: conf["cfg"], handler=webdav.Handler).start() + auth._FAILS.clear() + share_import.REGISTRY.items.clear() + yield {"port": server.port, "keys": keys, "talks": talks, "pods": pods, "conf": conf} + server.stop() + + +def _call(env, method, path, headers=None, body=None, who="ed", chunked=False): + c = http.client.HTTPConnection("127.0.0.1", env["port"], timeout=10) + h = {"Authorization": env["keys"][who], **(headers or {})} + if chunked: + c.putrequest(method, path) + for k, v in {**h, "Transfer-Encoding": "chunked"}.items(): + c.putheader(k, v) + c.endheaders() + for i in range(0, len(body), 100): + piece = body[i : i + 100] + c.send(f"{len(piece):x}\r\n".encode() + piece + b"\r\n") + c.send(b"0\r\n\r\n") + else: + c.request(method, path, body=body, headers=h) + r = c.getresponse() + out = (r.status, dict(r.getheaders()), r.read()) + c.close() + return out + + +def _names(env, path, who="ed"): + status, _, body = _call(env, "PROPFIND", path, {"Depth": "1"}, who=who) + assert status == 207, body + return [r.find(f".//{D}displayname").text for r in ET.fromstring(body).findall(f"{D}response")][1:] + + +def _recording(db, title): + return db.one("SELECT record::id(id) AS id, collection, size FROM recording WHERE title = $t", t=title) + + +def test_finder_copies_a_file(db, env): + """Finder: an empty file, a lock, the bytes chunked, its ._ file, PROPPATCH, unlock.""" + path = "/pods/Collections/Talks/minutes.pdf" + assert _call(env, "PUT", path, body=b"")[0] == 201 + status, h, body = _call(env, "LOCK", path, {"Timeout": "Second-600"}, body=b'<?xml version="1.0"?><D:lockinfo xmlns:D="DAV:"/>') + assert status == 200 and h["Lock-Token"].startswith("<opaquelocktoken:") and b"activelock" in body + status, h, _ = _call(env, "PUT", path, {"X-Expected-Entity-Length": str(len(PDF)), "If": f"({h['Lock-Token']})"}, PDF, chunked=True) + assert status == 204 and h["ETag"].strip('"') + assert _call(env, "PUT", "/pods/Collections/Talks/._minutes.pdf", body=b"\0\5\26\7AppleDouble")[0] == 201 + status, _, body = _call( + env, + "PROPPATCH", + path, + body=b'<?xml version="1.0"?><D:propertyupdate xmlns:D="DAV:" xmlns:Z="urn:schemas-microsoft-com:"><D:set><D:prop><Z:Win32LastModifiedTime>Thu, 15 Oct 2026 10:00:00 GMT</Z:Win32LastModifiedTime></D:prop></D:set></D:propertyupdate>', + ) + assert status == 207 and b"Win32LastModifiedTime" in body and b"200 OK" in body + assert _call(env, "UNLOCK", path, {"Lock-Token": "<opaquelocktoken:1>"})[0] == 204 + rec = _recording(db, "minutes") + assert rec["collection"] == env["talks"] and rec["size"] == len(PDF) + names = _names(env, "/pods/Collections/Talks/") + assert {"minutes.pdf", "._minutes.pdf", f"minutes [{rec['id']}].pdf"} <= set(names) + assert _call(env, "GET", path)[2] == PDF + assert "minutes.pdf" not in _names(env, "/pods/Collections/Talks/", "vi") # their own view only + assert _call(env, "DELETE", "/pods/Collections/Talks/._minutes.pdf")[0] == 204 # Finder tidies up + assert _call(env, "DELETE", path)[0] == 403 # an import isn't undone from here + + +def test_windows_copies_a_file(db, env): + """Windows Explorer: a lock on a new name makes it (empty), then the bytes, its times, unlock.""" + path = "/pods/notes.pdf" + status, h, _ = _call(env, "LOCK", path, {"Depth": "0"}, body=b'<?xml version="1.0"?><D:lockinfo xmlns:D="DAV:"/>') + assert status == 201 + status, h2, _ = _call(env, "LOCK", path, {"If": f"({h['Lock-Token']})"}) # renewed + assert status == 200 and h2["Lock-Token"] == h["Lock-Token"] + assert _call(env, "PUT", path, body=PDF)[0] == 204 + assert _recording(db, "notes")["collection"] == store.default_collection(db, env["pods"]) + + +def test_a_folder_dragged_in_becomes_collections(db, env): + assert _call(env, "MKCOL", "/pods/Collections/Trips/")[0] == 201 + assert _call(env, "MKCOL", "/pods/Collections/Trips/Lisbon/")[0] == 201 + assert _call(env, "MKCOL", "/pods/Collections/Trips/")[0] == 405 # there already + assert _call(env, "PUT", "/pods/Collections/Trips/Lisbon/day1.pdf", body=PDF)[0] == 201 + lisbon = next(c for c in hierarchy.of_space(db, env["pods"]) if c["name"] == "Lisbon") + assert _recording(db, "day1")["collection"] == lisbon["id"] + assert _call(env, "MKCOL", "/pods/Topics/New/")[0] == 403 + assert _call(env, "MKCOL", "/pods/Collections/Trips/x/", body=b"<x/>")[0] == 415 + + +def test_an_upload_under_a_temporary_name_is_imported_when_renamed(db, env): + assert _call(env, "PUT", "/pods/Collections/Talks/talk.pdf.filepart", body=PDF)[0] == 201 + assert not _recording(db, "talk") + status, _, _ = _call(env, "MOVE", "/pods/Collections/Talks/talk.pdf.filepart", + {"Destination": f"http://127.0.0.1:{env['port']}/pods/Collections/Talks/talk.pdf"}) # fmt: skip + assert status == 201 and _recording(db, "talk")["collection"] == env["talks"] + assert "talk.pdf.filepart" not in _names(env, "/pods/Collections/Talks/") + status, _, _ = _call(env, "MOVE", "/pods/Collections/General/ep1%20%5B1%5D.txt", {"Destination": "/pods/Collections/Talks/ep1.txt"}) + assert status == 403 # Lens's own files stay where they are + + +def test_what_may_not_be_added_is_refused(db, cfg, env): + assert _call(env, "PUT", "/pods/Collections/Talks/x.pdf", body=PDF, who="vi")[0] == 403 + assert _call(env, "LOCK", "/pods/Collections/Talks/x.pdf", who="vi")[0] == 403 + assert _call(env, "PUT", "/pods/Collections/General/ep1%20%5B1%5D.json", body=b"{}")[0] == 403 + status, _, body = _call(env, "PUT", "/pods/tool.exe", body=b"MZ") + assert status == 415 and b"can't be uploaded here" in body + assert _call(env, "PUT", "/pods/Somewhere/x.pdf", body=PDF)[0] == 409 + assert _call(env, "PUT", "/calls/x.pdf", body=PDF)[0] == 409 # not theirs: no such folder, as for one that isn't there + assert _call(env, "PUT", "/x.pdf", body=PDF)[0] == 403 + assert _call(env, "COPY", "/pods/Collections/General/ep1%20%5B1%5D.txt", {"Destination": "/pods/x.txt"})[0] == 403 + env["conf"]["cfg"] = {**cfg, "uploads": {**cfg["uploads"], "max_mb": 1}} + # refused from what the client says it will send, before a byte of it is read + assert _ask(env, "PUT /pods/big.pdf", {"Content-Length": str(2 << 20), "Expect": "100-continue"}).startswith(b"HTTP/1.1 413") + assert _ask(env, "PUT /pods/big.pdf", {"X-Expected-Entity-Length": str(2 << 20), "Transfer-Encoding": "chunked"}).startswith( + b"HTTP/1.1 413" + ) + # and a chunked body that turns out larger than it said, once it has arrived + assert _call(env, "PUT", "/pods/big.pdf", {"X-Expected-Entity-Length": "10"}, PDF + b"\0" * (1 << 20), chunked=True)[0] == 413 + assert _ask(env, "PUT /pods/x.pdf", {"Content-Length": "5", "Expect": "100-continue"}).startswith(b"HTTP/1.1 100") + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + + +def _ask(env, line, headers): + """Send a request's headers only, and the first answer's status line and headers.""" + import socket + + with socket.create_connection(("127.0.0.1", env["port"]), timeout=10) as s: + h = {"Host": "x", "Authorization": env["keys"]["ed"], **headers} + s.sendall((f"{line} HTTP/1.1\r\n" + "".join(f"{k}: {v}\r\n" for k, v in h.items()) + "\r\n").encode()) + return s.recv(4096) diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index b0d2a2f70..1803f2517 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -14937,6 +14937,12 @@ export type S3Status = { * what the process running it says; none while it's off */ service?: S3Service | null; + /** + * Add Files + * + * whether files people add through it are imported (shares.add_files) + */ + add_files?: boolean; webdav: ShareServer; /** * Keys @@ -16454,6 +16460,12 @@ export type ShareServer = { * what the process running it says; none while it's off */ service?: S3Service | null; + /** + * Add Files + * + * whether files people add through it are imported (shares.add_files) + */ + add_files?: boolean; [key: string]: unknown; }; diff --git a/nextjs-frontend/components/account/s3-keys.tsx b/nextjs-frontend/components/account/s3-keys.tsx index de1c83963..74b6ed4fd 100644 --- a/nextjs-frontend/components/account/s3-keys.tsx +++ b/nextjs-frontend/components/account/s3-keys.tsx @@ -54,8 +54,11 @@ export function S3Keys() { <div className="flex min-w-0 flex-1 flex-col gap-1"> <h2 className="text-[18px] font-bold leading-[1.2] text-fg">File share access keys</h2> <p className="text-[13.5px] leading-[1.4] text-fg-secondary"> - Read the archive as folders and files from any S3 client (rclone, Cyberduck, the AWS CLI). A key reads as - you; nothing can be changed through it. + Read the archive as folders and files from any S3 client (rclone, Cyberduck, the AWS CLI), or mount it as a + drive over WebDAV. A key acts as you + {status.data?.add_files + ? ": files you copy into a collection’s folder are imported there, where you may add files." + : "; nothing can be changed through it."} {status.data && !status.data.enabled && " An admin turns sharing on in Settings, File shares."} </p> </div> diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 35e0a8134..ffbb1f9e4 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -346,7 +346,7 @@ export const SECTIONS: SectionSpec[] = [ label: "File shares", backend: ["shares"], description: - "Off unless you turn it on. The archive’s assets as folders and files (namespaces and collections, a metadata file beside each asset), served read-only over S3 and WebDAV by Lens’s workers; people make their own access keys.", + "Off unless you turn it on. The archive’s assets as folders and files (namespaces and collections, a metadata file beside each asset), served over S3 and WebDAV by Lens’s workers; people make their own access keys, and files they add are imported where they put them.", }, { id: "uploads", @@ -2432,12 +2432,19 @@ export const FIELDS: FieldSpec[] = [ kind: "switch", hint: "Off: nothing listens. The API and the MCP tools read the tree either way", }, + { + section: "shares", + key: "add_files", + label: "Import files people add", + kind: "switch", + hint: "A file copied into a collection’s folder is imported there, as an upload; a new folder under Collections is a new collection. Off: read-only", + }, { section: "shares", key: "s3", label: "S3", kind: "switch", - hint: "Read-only; each person signs in with their own access key", + hint: "Each person signs in with their own access key", }, { section: "shares", key: "s3_port", label: "S3 port", kind: "int", min: 1, max: 65535 }, { @@ -2445,7 +2452,7 @@ export const FIELDS: FieldSpec[] = [ key: "webdav", label: "WebDAV", kind: "switch", - hint: "Read-only; mount it as a drive in Finder, Windows Explorer or a file manager", + hint: "Mount it as a drive in Finder, Windows Explorer or a file manager", }, { section: "shares", key: "webdav_port", label: "WebDAV port", kind: "int", min: 1, max: 65535 }, // Sensors (opt-in) diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 49f9e7ffe..37bf4ddc8 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -53121,6 +53121,12 @@ ], "description": "what the process running it says; none while it's off" }, + "add_files": { + "type": "boolean", + "title": "Add Files", + "description": "whether files people add through it are imported (shares.add_files)", + "default": false + }, "webdav": { "$ref": "#/components/schemas/ShareServer" }, @@ -55835,6 +55841,12 @@ } ], "description": "what the process running it says; none while it's off" + }, + "add_files": { + "type": "boolean", + "title": "Add Files", + "description": "whether files people add through it are imported (shares.add_files)", + "default": false } }, "additionalProperties": true, diff --git a/threat-model.json b/threat-model.json index fd496b8d8..2a73a090d 100644 --- a/threat-model.json +++ b/threat-model.json @@ -1119,7 +1119,7 @@ "property": "P-SHARES", "violation_symptom": "info-leak, integrity-bypass", "severity_tier": "security", - "conditions": "the file shares (S3, WebDAV) are off until an admin enables them; every request carries a person's own access key (SigV4 or Basic), reads as them and never writes", + "conditions": "the file shares (S3, WebDAV) are off until an admin enables them; every request carries a person's own access key (SigV4 or Basic) and reads as them; files written are imported only where that person may import, checked and size-capped as uploads, never over an asset or a file Lens made; shares.add_files off: read-only", "provenance": "documented", "source": "docs/file-tree.md \"Off until an admin turns it on\"" }, diff --git a/threat-model.md b/threat-model.md index a5485eeeb..9706b9ea0 100644 --- a/threat-model.md +++ b/threat-model.md @@ -105,7 +105,7 @@ optional listeners (MQTT, syslog, tunnel, local model server) run inside workers | Ingest & processing (Ingest) | `routes/{uploads,files,imports}.py`, `domain/{ingest,documents,convert,video,uploads}.py`, `templates.py`, `render.py` | child processes (ffmpeg, LibreOffice, Chromium, poppler, Tesseract), files | In | | Query engines (Query) | `domain/cypher.py`, `domain/rdf.py` (SPARQL), `domain/rdf_import.py`, `domain/search.py` | DB | In | | Assistant & extensions (Assistant) | `domain/{chat,ai_tools,ops_tools,extensions,tool_nodes,flow,custom_nodes,code_tools,mcp_client,marketplace}.py`, MCP write tools | network (model servers, admins' MCP servers, the MCP runner), child process (code tools) | In | -| Optional services, off by default (Optional) | `domain/{sensors,mqtt,syslog,shares,s3,webdav}.py`, `tunnel.py`, `https.py`, `local_llm.py`, `components.py`, `telemetry.py` | listening sockets, downloads, child processes | In | +| Optional services, off by default (Optional) | `domain/{sensors,mqtt,syslog,shares,s3,webdav,share_import,http_body}.py`, `tunnel.py`, `https.py`, `local_llm.py`, `components.py`, `telemetry.py` | listening sockets, downloads, child processes | In | | Packaging variants | `docker-compose.prod.yml`, `install.sh`, `cloudron/`, `packaging/{synology,qnap}`, `proxmox/` | host | §1.6 | | Dev and CI tooling, samples | `docker-compose.yml`, `make dev`, `.github/`, `ci/`, `examples/`, `overrides/`, `prod-frontend-deploy.yml`, `watcher.py`, `commands/`, `fastapi_backend/tests/` | host | Out (§1.3) | @@ -241,7 +241,7 @@ machine requires HTTPS, and a changed DB credential when SurrealDB runs as its o | `documents.web_networks`, `LENS_WEB_NETWORKS`; `notifications.networks` | empty | widen the outbound guard to listed private networks. `documents.web_networks` is in Settings → Documents for admins (and agents with an admin token), who can't list loopback, `0.0.0.0/8` or link-local (cloud metadata) networks there; only archive.yaml and `LENS_WEB_NETWORKS`, on the server, can | Yes (operator and admin input) | *(documented, docs/configuration.md "The web app can't list this machine (loopback) or link-local addresses")* | | Chromium sandbox | off as root or without user namespaces | the `netguard` proxy is the only barrier | Yes | *(documented, docs/configuration.md "runs with its sandbox where it can")* | | `sensors.enabled`, `sensors.mqtt_anonymous` | off (`bind` `0.0.0.0`); off | opens MQTT and syslog listeners; anonymous MQTT publish | Yes; anonymous is the admin's choice | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | -| `shares.enabled` | off (`bind` `0.0.0.0`) | opens the read-only S3 and WebDAV servers for the asset tree | Yes | *(documented, docs/file-tree.md "Off until an admin turns it on")* | +| `shares.enabled` | off (`bind` `0.0.0.0`) | opens the S3 and WebDAV servers for the asset tree; with `shares.add_files` (on) files people add through them are imported | Yes | *(documented, docs/file-tree.md "Off until an admin turns it on")* | | `tunnel.mode`, `local_llm.enabled`, `telemetry`, `analytics.enabled` | off | outbound tunnel (Cloudflare terminates TLS, Q30); local model server; OTLP export; GA4 Measurement Protocol export of daily counts | Yes | *(documented, docs/remote-access.md "It's off until an admin turns it on")* | | `https.enabled` (and install.sh's `LENS_HTTPS=1`, which Settings › HTTPS can also ask the upgrade agent for) | off | Caddy, fetched at run time, terminates TLS on ports 443/80 in the worker and proxies to the web app; the DNS provider's credentials are written to `data_dir/https/caddy.json` (mode 600), not a command line | Yes | *(documented, docs/https.md "Off by default")* | | Reachable from the network: install.sh's `LENS_PUBLIC` (kept as `LENS_BIND`), also switched from Settings › Access & embedding, `PUT /api/v1/admin/install` or MCP `set_install_settings` | `1` (`0.0.0.0`) on a server without a desktop, else `0` (`127.0.0.1`) | publishes the web app's port to the network over plain HTTP, and the sensor, file-share and Grafana ports with it unless `LENS_SENSOR_BIND` or `LENS_SHARE_BIND` say otherwise. Admins (an agent with an admin's write-scope token too) can open it without a shell: the host upgrade agent runs install.sh again; audited as `lens.install` | Supported on a trusted LAN only until HTTPS is set up, as the headless-host row | *(documented, docs/updates.md "Setting the machine up again")* | @@ -292,7 +292,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | MQTT 1883, syslog 5514, `POST /sensors/push/{token}/{stream}` | payloads, hub logins, token in the path | yes, once enabled | data, size/rate, x-credential | admin enables; operator firewalls | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | | NFS share (port 12049 on the host, 2049 in the worker) | NFS and MOUNT RPCs, client address | yes, once enabled, from `nfs.allowed_networks` | data, x-network-source | admin enables and picks the account; operator firewalls | *(documented, docs/file-shares.md "NFS has no login")* | | SMB share (port 1445 on the host, 4445 in the worker) | SMB2 requests, NTLM sign-in with a file access key, client address | yes, once enabled, from `smb.allowed_networks` | data, x-credential, x-network-source | admin enables; operator firewalls | *(documented, docs/smb.md "What keeps it safe")* | -| S3 9555 and WebDAV 9556 (file shares) | signed S3 requests (SigV4 header or presigned link); WebDAV with an access key as Basic sign-in | yes, once enabled | x-credential, resource-name | admin enables; each request carries a person's own access key; read-only; failed WebDAV sign-ins throttled per address | *(documented, docs/file-tree.md "Off until an admin turns it on")* | +| S3 9555 and WebDAV 9556 (file shares) | signed S3 requests (SigV4 header or presigned link); WebDAV with an access key as Basic sign-in | yes, once enabled | x-credential, resource-name | admin enables; each request carries a person's own access key; files added are imported only where that person may import (`shares.add_files` off: read-only), checked as uploads, size-capped while streamed, never over Lens's own files; failed WebDAV sign-ins throttled per address | *(documented, docs/file-tree.md "Off until an admin turns it on")* | | `.env`, `archive.yaml`, `LENS_*`, `ARCHIVE_SECRET_KEY`, `RCLONE_BINARY` | process configuration | no (operator) | x-env-config | keep unreadable to others | *(maintainer, 2026-10)* | | workers and DB peers | DB credential, shared database rows | no (trusted peer) | x-peer | run peers on trusted hosts | *(inferred, Q9)* | | `.lens-control/inbox/upgrade` (host upgrade agent), `/metrics`, `.lens-control/prometheus-token` | `ref` (release tag or this install's branch; empty: the version running), with `https=`, `public=` (`0`/`1`) and `target=` (`full`/`lean`); `action=services` with `monitoring=on`/`off`; the scrape token | the inbox is writable by the containers (and local users); `/metrics` by anyone reaching the web app | x-env-config, credential | the agent installs only a release tag or the install's branch, changes only the HTTPS ports, the web app's network reach and the edition (any other value is dropped) and only toggles the `monitoring` profile, nothing else from the file, so a writer of the inbox can at most restart Lens with those choices or open it to the network; `/metrics` answers only the scrape token (constant-time compare); the token file is world-readable on the host and grants metrics only, which carry no personal content | *(documented, docs/updates.md, docs/prometheus.md "Privacy")* | @@ -525,7 +525,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-SENSORS | Sensors are off until an admin enables them. MQTT needs a hub login unless `mqtt_anonymous` is on. Syslog hears only listed networks. | integrity-bypass | security-critical | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | Voided by: `sensors.mqtt_anonymous` — `domain/sensors.py:984`; `sensors.syslog_networks`, `sensors.bind`. | | P-NFS | The NFS share is off until an admin turns it on. It is read-only, shows only what one chosen account sees in the web app, and turns away every client outside `nfs.allowed_networks` before rclone reads a byte. | info-leak | security-critical | *(documented, docs/file-shares.md "NFS has no login")* | Voided by: `nfs.allowed_networks` set wide, or a client address spoofed on the local network (NFS has no login) — `domain/share_relay.py`; Docker publishing that rewrites client addresses. | | P-SMB | The SMB share is off until an admin turns it on. It is read-only, signs people in with their own file access keys and shows each only what they see in the web app, refuses guests and SMB1 sessions, gives every connection its own NTLM challenge, and closes connections from outside `smb.allowed_networks` before reading a byte. | info-leak | security-critical | *(documented, docs/smb.md "What keeps it safe")* | Voided by: `smb.allowed_networks` set wide; SMB 2.0 has no encryption on the wire, so a key's NTLM exchange and the files read can be watched on the network — `domain/smb_share.py`. | -| P-SHARES | The file shares (S3 and WebDAV) are off until an admin enables them. Every request is signed (S3: AWS Signature Version 4) or signed in (WebDAV: Basic) with an access key someone made while signed in to the web app; it reads as them, sees only what the web app shows them, and never writes. Keys expire as API keys do, stop with a disabled account, and are revoked at once. | info-leak, integrity-bypass | security-critical | *(documented, docs/file-tree.md "Off until an admin turns it on")* | Voided by: `shares.enabled` with `LENS_BIND=0.0.0.0` sends files, S3 signatures and WebDAV's Basic secret in clear over plain HTTP on the network; key secrets are sealed with the server key (`secret.key`), not hashed, since SigV4 signs with the secret itself — `domain/s3.py`, `domain/webdav.py`. | +| P-SHARES | The file shares (S3 and WebDAV) are off until an admin enables them. Every request is signed (S3: AWS Signature Version 4) or signed in (WebDAV: Basic) with an access key someone made while signed in to the web app; it reads as them and sees only what the web app shows them. A file it writes is imported as an upload by that person would be (the import permission or an editor's role on the collection; making a collection as the web app allows; the upload type and size checks, with the body streamed to disk and cut off at the size limit), never replaces or deletes an asset or a file Lens made, and with `shares.add_files` off nothing is written. Keys expire as API keys do, stop with a disabled account, and are revoked at once. | info-leak, integrity-bypass | security-critical | *(documented, docs/file-tree.md "Off until an admin turns it on")* | Voided by: `shares.enabled` with `LENS_BIND=0.0.0.0` sends files, S3 signatures and WebDAV's Basic secret in clear over plain HTTP on the network; key secrets are sealed with the server key (`secret.key`), not hashed, since SigV4 signs with the secret itself; a leaked key can add files as its owner can — `domain/s3.py`, `domain/webdav.py`, `domain/share_import.py`. | | P-CHILD-SECRETS | The tunnel token reaches cloudflared in its environment, not its command line; llama-server gets its own random API key. | info-leak | correctness-only | *(documented, docs/remote-access.md "in its environment, not on its command line")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-REMOTE-NAMES | An IMAP message is named by mailbox validity and uid, so a rebuilt mailbox never serves another message under an old name. | wrong-output | correctness-only | *(documented, docs/processing.md "an old one never reads another message")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-NOTIFY | Each notification event is claimed once across processes and retried with backoff. | wrong-output | correctness-only | *(documented, docs/notifications.md "claimed once")* | Voided by: none identified (not required for this symptom; no exhaustive search) | diff --git a/threat-model.yaml b/threat-model.yaml index d0c9a3ac8..fdabd5b15 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:f430a4e9bedcf37ca6a5f5e2e782a3e72a9ac674b6dc78ebc91fc152af9e92a4" +prose_version: "threat-model.md@sha256:00a211d9cad408ad3e7b856b7902e404c0ab9ab1fc2057d4f12020911d6796f3" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -1113,7 +1113,7 @@ build_flags: default: "off" security_relevant: true maintainer_stance: supported - x-scope: "opens the read-only S3 and WebDAV servers for the asset tree on shares.s3_port and shares.webdav_port" + x-scope: "opens the S3 and WebDAV servers for the asset tree on shares.s3_port and shares.webdav_port; with shares.add_files (on) files people add are imported where they may import" affects_properties: - {property_id: P-SHARES, effect: narrows} provenance: {kind: documented, source: 'docs/file-tree.md "Off until an admin turns it on"'} @@ -1552,9 +1552,9 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "the file shares (S3, WebDAV) are off until an admin enables them; every request carries a person's own access key (SigV4 or Basic), reads as them and never writes" + conditions: "the file shares (S3, WebDAV) are off until an admin enables them; every request carries a person's own access key (SigV4 or Basic) and reads as them; files written are imported only where that person may import, checked and size-capped as uploads, never over an asset or a file Lens made; shares.add_files off: read-only" violation_symptoms: [info-leak, integrity-bypass] - x-voided-by: "plain HTTP on the network with LENS_BIND=0.0.0.0 (WebDAV's Basic secret in clear); secrets sealed with the server key, not hashed (domain/s3.py)" + x-voided-by: "plain HTTP on the network with LENS_BIND=0.0.0.0 (WebDAV's Basic secret in clear); secrets sealed with the server key, not hashed (domain/s3.py); a leaked key adds files as its owner can (domain/share_import.py)" provenance: {kind: documented, source: 'docs/file-tree.md "Off until an admin turns it on"'} - id: P-CHILD-SECRETS kind: confidentiality From 2c4e0efcd70b66ef9ca6582257477e8b54ef2d02 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad <noreply@anthropic.com> Date: Thu, 15 Oct 2026 13:00:47 +0000 Subject: [PATCH 2/4] fix(shares): let a refused upload's client read the answer Closing a connection with part of a refused body unread reset it, so a client still sending (a chunked body over the size limit) got a broken pipe instead of 413. The S3 and WebDAV servers now read and drop what follows for up to two seconds before closing (RFC 9112, 9.6). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g --- fastapi_backend/app/domain/http_body.py | 30 ++++++++++++++++++- fastapi_backend/app/domain/s3.py | 3 +- fastapi_backend/app/domain/webdav.py | 3 +- .../tests/api/test_webdav_writes.py | 7 +++-- 4 files changed, 36 insertions(+), 7 deletions(-) diff --git a/fastapi_backend/app/domain/http_body.py b/fastapi_backend/app/domain/http_body.py index c581374c3..30c181ab6 100644 --- a/fastapi_backend/app/domain/http_body.py +++ b/fastapi_backend/app/domain/http_body.py @@ -1,7 +1,8 @@ """Request bodies as the file shares receive them (s3.py, webdav.py): streamed to a file, never held in memory, as Content-Length bytes, HTTP/1.1 chunked transfer coding, or S3's aws-chunked coding (STREAMING-… payloads, whose chunk signatures are checked when they're signed). Each body is checked against what the client said about it: -its length, the SHA-256 it signed, its Content-MD5, and an x-amz-checksum-crc32 or -sha256. +its length, the SHA-256 it signed, its Content-MD5, and an x-amz-checksum-crc32 or -sha256. A body refused part way +is read and dropped for a moment before the connection closes (Lingering), so the client reads the answer. """ from __future__ import annotations @@ -9,12 +10,16 @@ import base64 import hashlib import hmac +import http.server +import socket +import time import zlib from collections.abc import Iterator PIECE = 1 << 16 LINE_MAX = 8192 EMPTY_SHA256 = hashlib.sha256(b"").hexdigest() +LINGER_SECONDS = 2.0 class BodyError(ValueError): @@ -186,3 +191,26 @@ def drain(rfile, headers: dict[str, str], most: int = 1 << 20) -> bool: return False rfile.read(n) return True + + +class Lingering(http.server.BaseHTTPRequestHandler): + """A request handler that, as it closes a connection, first reads and drops what the client is still sending (a + body it refused before reading it all) for up to LINGER_SECONDS. Closing with that unread would reset the + connection, and the client, still sending, would never read the answer (RFC 9112, 9.6).""" + + def finish(self): + super().finish() + if self.close_connection: + linger(self.connection) + + +def linger(sock, seconds: float = LINGER_SECONDS) -> None: + try: + sock.shutdown(socket.SHUT_WR) + end = time.monotonic() + seconds + while (left := end - time.monotonic()) > 0: + sock.settimeout(left) + if not sock.recv(PIECE): + return + except OSError: # the client went away, or kept sending past the moment + pass diff --git a/fastapi_backend/app/domain/s3.py b/fastapi_backend/app/domain/s3.py index 9773a6451..ed55e6688 100644 --- a/fastapi_backend/app/domain/s3.py +++ b/fastapi_backend/app/domain/s3.py @@ -23,7 +23,6 @@ import errno import hashlib import hmac -import http.server import json import os import re @@ -408,7 +407,7 @@ def k(s): # ---------- the server ---------- -class Handler(http.server.BaseHTTPRequestHandler): +class Handler(http_body.Lingering): server_version = "Lens" protocol_version = "HTTP/1.1" server: Any # shares.Server diff --git a/fastapi_backend/app/domain/webdav.py b/fastapi_backend/app/domain/webdav.py index 72f39eff1..e8c277dc4 100644 --- a/fastapi_backend/app/domain/webdav.py +++ b/fastapi_backend/app/domain/webdav.py @@ -16,7 +16,6 @@ import datetime as dt import email.utils import errno -import http.server import re import secrets import urllib.parse @@ -76,7 +75,7 @@ def multistatus(tree, parts, depth): return ('<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:">' + "".join(out) + "</D:multistatus>").encode() -class Handler(http.server.BaseHTTPRequestHandler): +class Handler(http_body.Lingering): server_version = "Lens" protocol_version = "HTTP/1.1" server: Any # shares.Server diff --git a/fastapi_backend/tests/api/test_webdav_writes.py b/fastapi_backend/tests/api/test_webdav_writes.py index 5abf3887b..972c004cf 100644 --- a/fastapi_backend/tests/api/test_webdav_writes.py +++ b/fastapi_backend/tests/api/test_webdav_writes.py @@ -141,8 +141,11 @@ def test_what_may_not_be_added_is_refused(db, cfg, env): assert _ask(env, "PUT /pods/big.pdf", {"X-Expected-Entity-Length": str(2 << 20), "Transfer-Encoding": "chunked"}).startswith( b"HTTP/1.1 413" ) - # and a chunked body that turns out larger than it said, once it has arrived - assert _call(env, "PUT", "/pods/big.pdf", {"X-Expected-Entity-Length": "10"}, PDF + b"\0" * (1 << 20), chunked=True)[0] == 413 + # and a chunked body that turns out larger than it said, once it has arrived; the client, still sending the rest, + # reads the answer (the server drops what follows rather than reset the connection) + big = PDF + b"\0" * (4 << 20) + assert _call(env, "PUT", "/pods/big.pdf", {"X-Expected-Entity-Length": "10"}, big, chunked=True)[0] == 413 + assert _call(env, "PUT", "/pods/big.pdf", {"Content-Length": str(len(big))}, big)[0] == 413 assert _ask(env, "PUT /pods/x.pdf", {"Content-Length": "5", "Expect": "100-continue"}).startswith(b"HTTP/1.1 100") assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") From c0790b374441bf4d0dcc221e5392cabbd0faa064 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad <noreply@anthropic.com> Date: Sat, 17 Oct 2026 04:44:27 +0000 Subject: [PATCH 3/4] docs(readme): files added over S3 and WebDAV are built, with their checks Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g --- README.md | 9 ++++----- docs/requirements.md | 2 ++ 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 99d840496..7079e0edd 100644 --- a/README.md +++ b/README.md @@ -68,9 +68,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p podcasts. They sign in with OAuth and see only what their person may read; the account menu copies the MCP address in one click. A namespace can also have an MCP server of its own (`/mcp/<namespace>`) and answer OpenAI's chat completions API for any app, cited from that namespace alone, each off until its owner turns it on, with API keys that - open that one namespace and nothing else. The same folders and files can be mounted read-only over S3, WebDAV, NFS or - SMB, so file managers and scripts read Lens too. The web app runs on the same REST API, so an agent can do what a - person can. + open that one namespace and nothing else. The same folders and files can be mounted over S3, WebDAV, NFS or SMB (S3 + and WebDAV also at the web app's address, through remote access), so file managers and scripts read Lens too, and a + file copied in over S3 or WebDAV is imported where it was put. The web app runs on the same REST API, so an agent can + do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app), [API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md); @@ -211,8 +212,6 @@ These are open pull requests, not yet part of Lens: - **Web clipper:** signing in, or using your API token, not only an import webhook's ([#406](https://github.com/adeelahmad/lense/pull/406)). - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)). -- **Shares:** S3 and WebDAV reached through remote access ([#402](https://github.com/adeelahmad/lense/pull/402)), - and files added over them imported ([#408](https://github.com/adeelahmad/lense/pull/408)). ## Planned diff --git a/docs/requirements.md b/docs/requirements.md index 6fd4395cb..5e12ee708 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -64,6 +64,8 @@ request, to another check that proves the same thing; loosening a requirement or | OpenAI's chat completions API per namespace, cited from that namespace alone, off until turned on | [test_completions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_completions.py) `test_off_until_an_owner_turns_it_on`, `test_answers_from_the_namespace_alone_with_citations`, `test_streams_as_openai_does` | | API keys that open one namespace and nothing else | [test_namespace_keys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_namespace_keys.py) `test_a_key_for_one_namespace_opens_its_servers_alone`, `test_it_has_the_persons_role_there_and_no_more` | | The graph's folders and files mounted read-only over S3, WebDAV, NFS and SMB | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_people_read_their_namespaces_over_s3`, `test_what_s3_refuses`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_mount_and_read`, `test_what_webdav_refuses`; [test_nfs_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_nfs_share.py) `test_the_runner_follows_the_setting`; [test_smb_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_share.py) `test_reads_the_tree_as_its_key_sees_it`, `test_nothing_can_be_written` | +| S3 and WebDAV at the web app's address, through remote access, once an admin allows it | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`, `test_through_the_web_app_each_visitor_has_its_own_throttle` | +| A file copied in over S3 or WebDAV is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused` | | The web app runs on the same REST API, through a client generated from its OpenAPI schema | [ci.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/ci.yml) `OpenAPI schema is up to date` | ## Ask, and get cited answers From 9c466407e6c6364d9bc72117e1e9273f62f7e611 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad <noreply@anthropic.com> Date: Sat, 17 Oct 2026 05:00:07 +0000 Subject: [PATCH 4/4] fix(ci): the upgrade check counts entities of every type Since #395 the entity-types upgrade step types Terms such as Monday as Dates, which the entities list leaves out unless asked for, so the check read them as gone. It now lists the quiet types too: an entity may change type in an upgrade, but none may go. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g --- ci/upgrade/lens_upgrade.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/ci/upgrade/lens_upgrade.py b/ci/upgrade/lens_upgrade.py index a0b3c75b2..8593e2fd6 100755 --- a/ci/upgrade/lens_upgrade.py +++ b/ci/upgrade/lens_upgrade.py @@ -192,7 +192,10 @@ def facts(api, token): c["text"] for c in api.get(f"/resources/{rid}/comments") ) entities = api.get("/entities", namespaces=NS, limit=200)["items"] - out["entities"] = sorted(e["name"] for e in entities) + # every entity by name, whatever its type: an upgrade step may type a Term (Monday becomes a Date, which the list + # leaves out unless asked for), but none may go + quiet = api.get("/entities", namespaces=NS, limit=200, types="DATE,NUMBER")["items"] + out["entities"] = sorted(e["name"] for e in entities + quiet) for e in entities: if e["name"] == ENTITY["name"]: out[f"entity {e['name']}"] = {