diff --git a/CHANGELOG.md b/CHANGELOG.md index 74f6ab9a5..5c4343910 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -72,6 +72,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version shares; then a new access key's dialog gives those addresses. Signing in is the same: a person's own access key, read-only. MQTT, syslog, SMB and NFS aren't HTTP and stay on their own ports ([The graph as files](docs/file-tree.md#at-the-web-apps-address), [Remote access](docs/remote-access.md#what-it-reaches)). +- **Files added over S3 and WebDAV are imported.** Copy a file into a collection's folder (`rclone copy`, Finder, + Windows Explorer, any S3 client) and it's imported there as an upload is, with the same checks and the namespace's + pipeline; a new folder under `Collections` is a new collection, and a file anywhere else goes to the namespace's + default collection. It takes the import permission (or an editor's role on the collection), files the system writes + for itself (`.DS_Store`, `Thumbs.db`, half-written files) are never imported, and a re-sync copies nothing twice. + The S3 server now takes streamed and multipart uploads; WebDAV takes PUT, MKCOL and Finder's and Explorer's ways of + copying. This is on whenever the shares are; **Import files people add** in Settings, File shares + (`shares.add_files`) turns it off and makes them read-only again ([Adding files](docs/file-tree.md#adding-files)). - **The web clipper takes your API token, or just signs in.** It used to take only an import webhook's token, so an API token was refused with "Lens didn't recognise this token: make a new import webhook and paste its token". Now its options have **Sign in**: Lens asks you to allow the clipper, nothing to copy, and clips go into the namespace you diff --git a/README.md b/README.md index bd05eb16d..f2853345c 100644 --- a/README.md +++ b/README.md @@ -68,9 +68,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p podcasts. They sign in with OAuth and see only what their person may read; the account menu copies the MCP address in one click. A namespace can also have an MCP server of its own (`/mcp/`) and answer OpenAI's chat completions API for any app, cited from that namespace alone, each off until its owner turns it on, with API keys that - open that one namespace and nothing else. The same folders and files can be mounted read-only over S3, WebDAV, NFS or - SMB, so file managers and scripts read Lens too. The web app runs on the same REST API, so an agent can do what a - person can. + open that one namespace and nothing else. The same folders and files can be mounted over S3, WebDAV, NFS or SMB (S3 + and WebDAV also at the web app's address, through remote access), so file managers and scripts read Lens too, and a + file copied in over S3 or WebDAV is imported where it was put. The web app runs on the same REST API, so an agent can + do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app), [API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md); @@ -212,8 +213,6 @@ These are open pull requests, not yet part of Lens: - **Web clipper:** signing in, or using your API token, not only an import webhook's ([#406](https://github.com/adeelahmad/lense/pull/406)). - **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)). -- **Shares:** S3 and WebDAV reached through remote access ([#402](https://github.com/adeelahmad/lense/pull/402)), - and files added over them imported ([#408](https://github.com/adeelahmad/lense/pull/408)). ## Planned diff --git a/docs/file-tree.md b/docs/file-tree.md index b4b520000..892b93472 100644 --- a/docs/file-tree.md +++ b/docs/file-tree.md @@ -3,12 +3,13 @@ Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read -Lens without learning its API. +Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3 or WebDAV +is imported there ([Adding files](#adding-files)). | Where | Status | | --- | --- | | The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built | -| An [S3 server](#s3) and a [WebDAV server](#webdav) (read-only, off until turned on) | built | +| An [S3 server](#s3) and a [WebDAV server](#webdav) (off until turned on); files added through them are imported | built | | An [NFS share](file-shares.md) (read-only, off until turned on, allowed networks only) | built | | An [SMB share](smb.md) (read-only, off until turned on) | built | @@ -45,7 +46,8 @@ Names are made safe for every file system: `/ \ : * ? " < > |` and control chara Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces they have a role in, and in a namespace where they only have roles on some collections, those collections and the -topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The tree only reads. +topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The API only +reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers. ## The API @@ -59,7 +61,7 @@ The MCP server has `list_files` and `read_file` ([MCP server](mcp.md)): `read_fi ## S3 Off until an admin turns it on, in Settings, File shares (`shares.enabled`). Then Lens's worker serves the tree -read-only over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the +over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the host), where the web app is: this machine only unless Lens was opened to the network (`LENS_BIND`; `LENS_SHARE_BIND` sets it apart). Each namespace you can read is a bucket; keys are the paths above. @@ -84,10 +86,14 @@ endpoint = http://lens.local:9555 force_path_style = true ``` -then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, or `rclone mount lens: ~/lens`. +then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, `rclone copy ./talks +"lens:pods/Collections/Talks"` ([Adding files](#adding-files)), or `rclone mount lens: ~/lens`. Supported: listing buckets, listing objects (v1 and v2, by folder or recursively, a page at a time), reading objects whole or -in ranges, and presigned links. Anything that writes answers AccessDenied. Requests must be signed with Signature -Version 4. +in ranges, presigned links, and adding files: PutObject (whole, streamed with `aws-chunked` and signed chunks, or with +a checksum), multipart uploads, a key ending in `/` for a folder, a copy of a file just added onto itself (to set its +time, as rclone does), and DeleteObject(s) for files kept aside. A file's `x-amz-meta-mtime` (rclone's) dates it. +Making or deleting buckets and deleting assets answer AccessDenied; copying other objects, NotImplemented. Requests must be signed with +Signature Version 4. The server speaks plain HTTP. On a network you don't trust, keep it on this machine or behind TLS (a reverse proxy), since files travel in clear. @@ -95,17 +101,51 @@ since files travel in clear. ## WebDAV Off until an admin turns it on, with S3 (Settings, File shares; `shares.webdav`, on by default once shares are). -Lens's worker serves the same tree read-only over WebDAV on port 9556 (`shares.webdav_port`; with Docker, +Lens's worker serves the same tree over WebDAV on port 9556 (`shares.webdav_port`; with Docker, `LENS_WEBDAV_PORT`), so it mounts as a drive: in Finder, Go, Connect to Server; in Windows Explorer, Map network drive; in GNOME Files or Dolphin, `dav://lens.local:9556/`; or `rclone` with `type = webdav`. Sign in with an access key: its id as the user name and its secret as the password. The folders at the top are your -namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work; anything that writes answers -403. Eight failed sign-ins from one address within 15 minutes are refused for a while. +namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work, and so does adding files +([Adding files](#adding-files)): PUT (with `Expect: 100-continue`, refused before the file is sent when it can't go +there), MKCOL for a folder, DELETE and MOVE for files kept aside, and PROPPATCH (answered, not kept), so dragging +files and folders in from Finder or Windows Explorer works. LOCK and UNLOCK answer as a class 2 server so Finder mounts +it writable, but they lock nothing. Deleting or moving assets answers 403. Eight failed sign-ins from one address within 15 +minutes are refused for a while. Like S3, it speaks plain HTTP, and Basic sign-in sends the secret with every request: keep it on this machine or behind TLS on a network you don't trust. +## Adding files + +A file copied into the tree over S3 or WebDAV is imported, as an upload in the web app is: the same checks of its type +and size (Settings, Uploads), the namespace's pipeline, and the same file twice is added once. Where it's put decides +where it goes: + +| Put it in | It goes to | +| --- | --- | +| `/Collections//…` | that collection | +| `/Collections//…` | a new collection, made with any missing ones above it (up to 8 deep), as **New collection** makes one | +| `/`, `/Collections/`, or a folder under `Topics/` or `Entities/` | the namespace's default collection; topics and entities come from the analysis | + +Folder names match collections whatever their case. It must be yours to add to: the import permission in the +namespace, or an editor's or admin's role on the collection (making a collection takes what making one in the web app +takes). Otherwise the file is refused before it's sent where the client allows (S3's and WebDAV's `Expect: +100-continue`). Files Lens made (an asset's own file and its `.json`, `.txt` and `.pdf`) can't be written over. Each +file added is audited (`share.import`, with the protocol and path), and each collection made too. + +After a file is imported, the asset shows as ` [<id>]` like every other; the name you gave it stays in its folder, +for you, for 10 minutes, so the copying app finds the file it just wrote and a re-sync sees it's there (with its own +time and MD5, so `rclone copy` and `rclone check` agree). Files an operating system or an app writes for itself +(`.DS_Store`, `._` files, `Thumbs.db`, `desktop.ini`, Office's lock files, `.part`, `.tmp` and other hidden or +half-written files) and empty files (Finder and Explorer make one before they write it) are kept aside for 10 minutes +after they last change, seen only by you, and never imported; renaming one to an ordinary name imports it then. They're +kept on the uploads disk, in memory per worker, so a restart forgets them. + +Adding files is on whenever the shares are; turn it off in Settings, File shares (**Import files people add**, +`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The +[NFS](file-shares.md) and [SMB](smb.md) shares stay read-only for now. + ## At the web app's address The web app can relay both shares at its own address: S3 at `/s3` and WebDAV at `/dav`. They then work wherever the web @@ -135,12 +175,16 @@ How it works: the web app hands requests under `/s3` and `/dav` to the worker's files, else this machine's ports 9555 and 9556). The path and `Host` stay as the client sent them, since S3 clients sign both, and `X-Forwarded-Prefix` tells the share server the request came through the web app. Failed WebDAV sign-ins are counted per visitor, from `X-Forwarded-For` as the web app and the proxies Lens trusts report it -(Settings, Access & embedding, Trusted proxies). Signing in is the same as at the ports: a person's own access key, -read-only. +(Settings, Access & embedding, Trusted proxies). Signing in is the same as at the ports: a person's own access key, and +files added that way are imported as at the ports ([Adding files](#adding-files)). ## Refine later -- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks for clients that insist on them. +- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks that lock. +- Delete, rename and move assets through the shares (only files just added can be today), and copy objects over S3. +- A read-only access key, for a person who may add files but wants a key that can't. +- Keep the names files were added under past 10 minutes and across restarts; a folder copy that hits a file type Lens + doesn't take stops in Finder. - Virtual-host S3 addressing at the web app's address, for clients that can't take an endpoint with a path (MinIO's `mc`). - Show supplementary files (captions, translations, attachments) beside each asset. diff --git a/docs/requirements.md b/docs/requirements.md index 37741ba08..63c6ed9c4 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -64,6 +64,8 @@ request, to another check that proves the same thing; loosening a requirement or | OpenAI's chat completions API per namespace, cited from that namespace alone, off until turned on | [test_completions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_completions.py) `test_off_until_an_owner_turns_it_on`, `test_answers_from_the_namespace_alone_with_citations`, `test_streams_as_openai_does` | | API keys that open one namespace and nothing else | [test_namespace_keys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_namespace_keys.py) `test_a_key_for_one_namespace_opens_its_servers_alone`, `test_it_has_the_persons_role_there_and_no_more` | | The graph's folders and files mounted read-only over S3, WebDAV, NFS and SMB | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_people_read_their_namespaces_over_s3`, `test_what_s3_refuses`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_mount_and_read`, `test_what_webdav_refuses`; [test_nfs_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_nfs_share.py) `test_the_runner_follows_the_setting`; [test_smb_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_share.py) `test_reads_the_tree_as_its_key_sees_it`, `test_nothing_can_be_written` | +| S3 and WebDAV at the web app's address, through remote access, once an admin allows it | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`, `test_through_the_web_app_each_visitor_has_its_own_throttle` | +| A file copied in over S3 or WebDAV is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused` | | The web app runs on the same REST API, through a client generated from its OpenAPI schema | [ci.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/ci.yml) `OpenAPI schema is up to date` | ## Ask, and get cited answers diff --git a/fastapi_backend/app/domain/filetree.py b/fastapi_backend/app/domain/filetree.py index 79692c413..cc8aece6c 100644 --- a/fastapi_backend/app/domain/filetree.py +++ b/fastapi_backend/app/domain/filetree.py @@ -16,8 +16,9 @@ What someone sees is what the web app shows them: the namespaces they have a role in (admins: all of them), and in a namespace where they only have roles on some collections, those collections and the topics and entities of the -assets in them. The tree only reads; names are made safe for every file system (no `/ \\ : * ? " < > |`) and unique -in their folder, ignoring case. +assets in them. Names are made safe for every file system (no `/ \\ : * ? " < > |`) and unique in their folder, +ignoring case. The tree itself only reads: a file someone adds through a share is imported by share_import.py, which +shows it to them under the name they gave it for a while (overlay). """ from __future__ import annotations @@ -58,17 +59,24 @@ class NotAFolder(ValueError): @dataclass(frozen=True) class Viewer: """Who is looking: an admin sees every namespace; others their namespaces (`roles`) and, in namespaces they have - no role in, the collections they were given a role on, with the ones inside them (`collections`).""" + no role in, the collections they were given a role on, with the ones inside them (`collections`). `account` and + `perms` (their permissions in each namespace, roles.py) are for adding files (share_import.py).""" admin: bool = False roles: dict[int, str] = field(default_factory=dict) collections: dict[int, dict[int, str]] = field(default_factory=dict) + account: int | None = None + perms: dict[int, frozenset[str]] = field(default_factory=dict) @classmethod def of(cls, db, account: dict[str, Any]) -> Viewer: """For an account row ({id, admin}).""" + from . import roles as rl + admin = bool(account.get("admin")) - return cls(admin, auth.roles(db, account), {} if admin else hierarchy.roles_of(db, account["id"])) + perms = auth.permissions(db, account) + roles = {sid: rl.rank(p) for sid, p in perms.items()} # as auth.roles + return cls(admin, roles, {} if admin else hierarchy.roles_of(db, account["id"]), account.get("id"), perms) def spaces(self, db) -> dict[int, set[int] | None]: """{namespace: None (all of it) or the collections seen}.""" @@ -84,7 +92,9 @@ def spaces(self, db) -> dict[int, set[int] | None]: @dataclass class Entry: """A file or a folder. A file's bytes come from `data` (made here) or `path` (stored, maybe encrypted) or a storage - source (`remote`: {source, path, space}).""" + source (`remote`: {source, path, space}). `added` marks a file someone just added through a share, shown to them + under the name they gave it (share_import.py): "arrived" (imported: the asset's own file) or "pending" (kept + aside, not imported).""" name: str folder: bool @@ -95,11 +105,14 @@ class Entry: path: str | None = None encrypted: bool = False remote: dict[str, Any] | None = None + added: str | None = None + md5: str | None = None # of a file just added, as the client sent it (its ETag over S3) + mtime: str | None = None # the time the client gave a file just added, as it wrote it (rclone's x-amz-meta-mtime) @property def version(self) -> str: - """Changes when the file does (for ETags): its name, size and time.""" - return hashlib.sha256(f"{self.name}\0{self.size}\0{self.modified}".encode()).hexdigest()[:24] + """Changes when the file does (for ETags): its name, size and time; a file just added, its MD5.""" + return self.md5 or hashlib.sha256(f"{self.name}\0{self.size}\0{self.modified}".encode()).hexdigest()[:24] def out(self) -> dict[str, Any]: d: dict[str, Any] = {"name": self.name, "folder": self.folder, "modified": self.modified} @@ -203,7 +216,23 @@ def list(self, path: str) -> list[Entry]: raise NotAFolder(path) from None raise + def forget(self) -> None: + """Read the database afresh from now on (after a file was added or a collection made).""" + self._cols.clear() + self._seen.clear() + self._topic_rows.clear() + def _list(self, parts: list[str]) -> list[Entry]: + out = self._listing(parts) + if parts and self.viewer.account is not None: + from . import share_import + + extra = share_import.overlay(self, parts, {e.name for e in out}) + if extra: + out = sorted(out + extra, key=lambda e: e.name.casefold()) + return out + + def _listing(self, parts: list[str]) -> list[Entry]: if not parts: return [_folder(n) for n in sorted(self._names.values(), key=str.casefold)] sid = self._space(parts[0]) diff --git a/fastapi_backend/app/domain/http_body.py b/fastapi_backend/app/domain/http_body.py new file mode 100644 index 000000000..30c181ab6 --- /dev/null +++ b/fastapi_backend/app/domain/http_body.py @@ -0,0 +1,216 @@ +"""Request bodies as the file shares receive them (s3.py, webdav.py): streamed to a file, never held in memory, as +Content-Length bytes, HTTP/1.1 chunked transfer coding, or S3's aws-chunked coding (STREAMING-… payloads, whose +chunk signatures are checked when they're signed). Each body is checked against what the client said about it: +its length, the SHA-256 it signed, its Content-MD5, and an x-amz-checksum-crc32 or -sha256. A body refused part way +is read and dropped for a moment before the connection closes (Lingering), so the client reads the answer. +""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import http.server +import socket +import time +import zlib +from collections.abc import Iterator + +PIECE = 1 << 16 +LINE_MAX = 8192 +EMPTY_SHA256 = hashlib.sha256(b"").hexdigest() +LINGER_SECONDS = 2.0 + + +class BodyError(ValueError): + """A body that's malformed, cut short, too large, or isn't what was signed. `code` names it as S3 does.""" + + def __init__(self, code: str, message: str, status: int = 400): + super().__init__(message) + self.code, self.message, self.status = code, message, status + + +class _Stream: + """Bytes from an iterator of pieces, read by length or by line.""" + + def __init__(self, pieces: Iterator[bytes]): + self.pieces, self.buf = pieces, b"" + + def _fill(self) -> bool: + piece = next(self.pieces, b"") + self.buf += piece + return bool(piece) + + def readline(self) -> bytes: + while b"\n" not in self.buf: + if len(self.buf) > LINE_MAX: + raise BodyError("IncompleteBody", "a chunk's header line is too long") + if not self._fill(): + raise BodyError("IncompleteBody", "the body ended early") + line, _, self.buf = self.buf.partition(b"\n") + return line.rstrip(b"\r") + + def read(self, n: int) -> Iterator[bytes]: + """Exactly `n` bytes, a piece at a time.""" + while n > 0: + if not self.buf and not self._fill(): + raise BodyError("IncompleteBody", "the body ended early") + out, self.buf = self.buf[:n], self.buf[n:] + n -= len(out) + yield out + + +def _fixed(rfile, length: int) -> Iterator[bytes]: + while length > 0: + piece = rfile.read(min(PIECE, length)) + if not piece: + raise BodyError("IncompleteBody", "the body ended before Content-Length bytes") + length -= len(piece) + yield piece + + +def _chunk_size(line: bytes) -> tuple[int, dict[str, str]]: + head, *params = line.decode("latin-1").split(";") + try: + size = int(head.strip(), 16) + except ValueError: + raise BodyError("IncompleteBody", "a chunk's size isn't a hex number") from None + return size, dict(p.strip().split("=", 1) for p in params if "=" in p) + + +def _http_chunked(rfile) -> Iterator[bytes]: + s = _Stream(iter(lambda: rfile.read1(PIECE) if hasattr(rfile, "read1") else rfile.read(1), b"")) + while True: + size, _ = _chunk_size(s.readline()) + if size == 0: + while s.readline(): # trailers, then the blank line + pass + return + yield from s.read(size) + if s.readline(): + raise BodyError("IncompleteBody", "a chunk isn't followed by CRLF") + + +class Signed: + """What a STREAMING-AWS4-HMAC-SHA256-PAYLOAD body's chunks are signed with: the signing key, the request's time and + scope, and the seed signature (the request's own).""" + + def __init__(self, key: bytes, amz: str, scope: str, seed: str): + self.key, self.amz, self.scope, self.prev = key, amz, scope, seed + + def check(self, sig: str, chunk_sha256: str) -> None: + to_sign = "\n".join(["AWS4-HMAC-SHA256-PAYLOAD", self.amz, self.scope, self.prev, EMPTY_SHA256, chunk_sha256]) + want = hmac.new(self.key, to_sign.encode(), hashlib.sha256).hexdigest() + if not hmac.compare_digest(want, sig or ""): + raise BodyError("SignatureDoesNotMatch", "a chunk's signature doesn't match", 403) + self.prev = want + + +def _aws_chunked(raw: Iterator[bytes], signed: Signed | None, trailer: dict[str, str]) -> Iterator[bytes]: + s = _Stream(raw) + while True: + size, params = _chunk_size(s.readline()) + h = hashlib.sha256() if signed else None + for piece in s.read(size): + if h: + h.update(piece) + yield piece + if signed: + signed.check(params.get("chunk-signature", ""), h.hexdigest()) # type: ignore[union-attr] + if size == 0: + while line := s.readline(): # trailers (a checksum, its signature), then the blank line + k, _, v = line.decode("latin-1").partition(":") + trailer[k.strip().lower()] = v.strip() + return + if s.readline(): + raise BodyError("IncompleteBody", "a chunk isn't followed by CRLF") + + +def receive(rfile, headers: dict[str, str], dest, limit: int, signed: Signed | None = None) -> tuple[int, str]: + """Stream a request's body into the file `dest` (lowercased `headers`): (its size, its MD5 in hex). BodyError when + it's malformed, over `limit` bytes, or not what the client said it is.""" + te = headers.get("transfer-encoding", "").lower() + if "chunked" in te: + raw = _http_chunked(rfile) + elif headers.get("content-length", "").strip().isdigit(): + length = int(headers["content-length"]) + if length > limit and not headers.get("x-amz-content-sha256", "").startswith("STREAMING-"): + raise BodyError("EntityTooLarge", "the file is larger than this server takes") + raw = _fixed(rfile, length) + else: + raise BodyError("MissingContentLength", "send Content-Length, or the body chunked", 411) + payload = headers.get("x-amz-content-sha256", "") + trailer: dict[str, str] = {} + if payload.startswith("STREAMING-"): + if payload.startswith("STREAMING-AWS4-HMAC-SHA256-PAYLOAD") and signed is None: + raise BodyError("InvalidRequest", "a signed streaming body needs a signed request", 403) + body = _aws_chunked(raw, signed if payload.startswith("STREAMING-AWS4-HMAC-SHA256-PAYLOAD") else None, trailer) + else: + body = raw + want = {k: headers[k] for k in ("x-amz-checksum-crc32", "x-amz-checksum-sha256") if headers.get(k)} + named = {t.strip().lower() for t in headers.get("x-amz-trailer", "").split(",") if t.strip()} + sha = hashlib.sha256() if len(payload) == 64 or "x-amz-checksum-sha256" in want or "x-amz-checksum-sha256" in named else None + crc = 0 if "x-amz-checksum-crc32" in want or "x-amz-checksum-crc32" in named else None + md5, size = hashlib.md5(usedforsecurity=False), 0 + with open(dest, "wb") as f: + for piece in body: + size += len(piece) + if size > limit: + raise BodyError("EntityTooLarge", "the file is larger than this server takes") + md5.update(piece) + if sha: + sha.update(piece) + if crc is not None: + crc = zlib.crc32(piece, crc) + f.write(piece) + want.update({k: v for k, v in trailer.items() if k in ("x-amz-checksum-crc32", "x-amz-checksum-sha256")}) + if headers.get("x-amz-decoded-content-length", "").isdigit() and int(headers["x-amz-decoded-content-length"]) != size: + raise BodyError("IncompleteBody", "the body isn't as long as x-amz-decoded-content-length says") + if len(payload) == 64 and sha and not hmac.compare_digest(sha.hexdigest(), payload.lower()): + raise BodyError("XAmzContentSHA256Mismatch", "the body isn't what was signed") + if headers.get("content-md5") and not hmac.compare_digest(base64.b64encode(md5.digest()).decode(), headers["content-md5"].strip()): + raise BodyError("BadDigest", "the body doesn't match its Content-MD5") + if "x-amz-checksum-sha256" in want and sha and base64.b64encode(sha.digest()).decode() != want["x-amz-checksum-sha256"]: + raise BodyError("BadDigest", "the body doesn't match its x-amz-checksum-sha256") + if ( + "x-amz-checksum-crc32" in want + and crc is not None + and base64.b64encode(crc.to_bytes(4, "big")).decode() != want["x-amz-checksum-crc32"] + ): + raise BodyError("BadDigest", "the body doesn't match its x-amz-checksum-crc32") + return size, md5.hexdigest() + + +def drain(rfile, headers: dict[str, str], most: int = 1 << 20) -> bool: + """Read and drop a small body (up to `most` bytes) so the connection can carry on; False when it can't be (too + large, chunked, or the client waits for 100 Continue): close the connection instead.""" + if headers.get("expect", "").lower() == "100-continue" or "chunked" in headers.get("transfer-encoding", "").lower(): + return False + n = int(headers.get("content-length") or 0) if (headers.get("content-length") or "0").isdigit() else most + 1 + if n > most: + return False + rfile.read(n) + return True + + +class Lingering(http.server.BaseHTTPRequestHandler): + """A request handler that, as it closes a connection, first reads and drops what the client is still sending (a + body it refused before reading it all) for up to LINGER_SECONDS. Closing with that unread would reset the + connection, and the client, still sending, would never read the answer (RFC 9112, 9.6).""" + + def finish(self): + super().finish() + if self.close_connection: + linger(self.connection) + + +def linger(sock, seconds: float = LINGER_SECONDS) -> None: + try: + sock.shutdown(socket.SHUT_WR) + end = time.monotonic() + seconds + while (left := end - time.monotonic()) > 0: + sock.settimeout(left) + if not sock.recv(PIECE): + return + except OSError: # the client went away, or kept sending past the moment + pass diff --git a/fastapi_backend/app/domain/s3.py b/fastapi_backend/app/domain/s3.py index 642b7125c..dc5a45e1c 100644 --- a/fastapi_backend/app/domain/s3.py +++ b/fastapi_backend/app/domain/s3.py @@ -1,14 +1,18 @@ -"""The file tree over S3 (docs/file-tree.md#s3): a read-only S3 server any S3 client mounts or syncs (rclone, -Cyberduck, the AWS CLI, s3fs, boto3), each person's namespaces as buckets. +"""The file tree over S3 (docs/file-tree.md#s3): an S3 server any S3 client mounts or syncs (rclone, Cyberduck, the +AWS CLI, s3fs, boto3), each person's namespaces as buckets. Off until an admin turns it on (`shares.enabled`); then the process that runs routines (`lens worker`) listens on `shares.s3_port`, as the sensor hub does (shares.py). Every request is signed (AWS Signature Version 4, in the Authorization header or a presigned link) with an access key someone made for themselves in the web app; the key acts as them and sees what they see in the web app. Keys' secrets are sealed with the server's key (they must be readable: SigV4 signs -with the secret itself), shown once, and can be revoked. Writes answer AccessDenied. +with the secret itself), shown once, and can be revoked. Path-style addressing (http://host:port/<bucket>/<key>), any region name. Supported: ListBuckets, HeadBucket, -GetBucketLocation, ListObjects (v1 and v2, with a delimiter or recursive), HeadObject and GetObject (with Range). +GetBucketLocation, ListObjects (v1 and v2, with a delimiter or recursive), HeadObject and GetObject (with Range). While +`shares.add_files` is on, a file put into the tree is imported (share_import.py): PutObject (whole, chunked, or +aws-chunked with signed chunks), multipart uploads, a folder marker (a key ending in /) under Collections makes a +collection, and DeleteObject(s) removes only what share_import.py keeps aside. Anything else that writes answers +AccessDenied or NotImplemented. With `shares.remote` on, the web app relays it at its own address too (https://<web app>/s3/<bucket>/<key>), path and all: clients sign that path, and the bucket and key come after /s3 (shares.web_path()). @@ -19,17 +23,22 @@ import base64 import datetime as dt import email.utils +import errno import hashlib import hmac -import http.server +import json +import os import re import secrets +import shutil import threading +import time import urllib.parse +import xml.etree.ElementTree as ET from typing import Any from xml.sax.saxutils import escape -from . import auth, filetree, keyring, settings, store +from . import auth, filetree, http_body, keyring, settings, share_import, store, uploads R = store.R NS = "http://s3.amazonaws.com/doc/2006-03-01/" @@ -39,6 +48,9 @@ MAX_KEYS = 1000 MAX_PRESIGN_SECONDS = 7 * 24 * 3600 NAME_MAX = 80 +UPLOAD_ID = re.compile(r"^[A-Za-z0-9_-]{32}$") +MAX_PARTS = 10000 +XML_MAX = 1 << 20 # a CompleteMultipartUpload or DeleteObjects body class S3Error(Exception): @@ -196,9 +208,12 @@ def _hmac(key, msg): return hmac.new(key, msg.encode(), hashlib.sha256).digest() +def signing_key(secret, date, region, service): + return _hmac(_hmac(_hmac(_hmac(("AWS4" + secret).encode(), date), region), service), "aws4_request") + + def signature(secret, date, region, service, to_sign): - k = _hmac(_hmac(_hmac(_hmac(("AWS4" + secret).encode(), date), region), service), "aws4_request") - return hmac.new(k, to_sign.encode(), hashlib.sha256).hexdigest() + return hmac.new(signing_key(secret, date, region, service), to_sign.encode(), hashlib.sha256).hexdigest() def _amz_time(s): @@ -208,9 +223,10 @@ def _amz_time(s): raise S3Error(403, "AccessDenied", "the request's date is missing or malformed") from None -def verify(db, cfg, method, raw_path, pairs, headers, now=None): +def verify(db, cfg, method, raw_path, pairs, headers, now=None, chunks=None): """Check a request's signature; the account it acts as. `pairs`: the query string's (name, value) pairs, `headers` - lowercased names to values.""" + lowercased names to values. `chunks`, a dict, gets what a streaming body's chunks are signed with + (http_body.Signed) under "signed".""" now = now or dt.datetime.now(dt.timezone.utc) q = dict(pairs) presigned = "X-Amz-Signature" in q @@ -258,6 +274,9 @@ def verify(db, cfg, method, raw_path, pairs, headers, now=None): if not hmac.compare_digest(signature(secret, date, region, service, to_sign), sig): raise S3Error(403, "SignatureDoesNotMatch", "the signature doesn't match: check the secret key") db.q("UPDATE $r SET last_used_at = $t", r=R("file_key", kid), t=store.now()) + if chunks is not None: + stamp = amz or when.strftime("%Y%m%dT%H%M%SZ") + chunks["signed"] = http_body.Signed(signing_key(secret, date, region, service), stamp, "/".join(scope[1:]), sig) return acct @@ -391,7 +410,7 @@ def k(s): # ---------- the server ---------- -class Handler(http.server.BaseHTTPRequestHandler): +class Handler(http_body.Lingering): server_version = "Lens" protocol_version = "HTTP/1.1" server: Any # shares.Server @@ -419,9 +438,74 @@ def do_GET(self): def do_HEAD(self): self._handle(head=True) + def handle_expect_100(self): + """Hold back 100 Continue until the request is known to be allowed (_go_on): a refused upload is never sent.""" + return True + + def _go_on(self): + if self.headers.get("expect", "").lower() == "100-continue": + self.send_response_only(100) + self.end_headers() + def _write(self): - self.rfile.read(min(int(self.headers.get("content-length") or 0), 1 << 20)) # what was sent, to keep the connection - self._fail(S3Error(403, "AccessDenied", "the file tree is read-only")) + from . import shares + + raw, _, query = self.path.partition("?") + headers = {k.lower(): v for k, v in self.headers.items()} + self._read = False + try: + pairs = urllib.parse.parse_qsl(query, keep_blank_values=True) + db, cfg = self.server.db, self.server.cfg_fn() + prefix = shares.web_path(headers, "s3") + if prefix and not shares.remote(cfg): + raise S3Error( + 403, "AccessDenied", "S3 isn't open at the web app's address: an admin turns that on in Settings, File shares" + ) + chunks: dict[str, Any] = {} + acct = verify(db, cfg, self.command, raw, pairs, headers, chunks=chunks) + raw = shares.share_path(raw, prefix) + if not share_import.accepting(cfg, "s3"): + raise S3Error(403, "AccessDenied", "the file tree is read-only") + adder = share_import.Adder(db, cfg, acct, "s3") + bucket, _, key = urllib.parse.unquote(raw).lstrip("/").partition("/") + q = dict(pairs) + if not bucket or (not key and self.command != "POST"): + raise S3Error(403, "AccessDenied", "namespaces are made in the web app" if bucket else "name a bucket") + if bucket not in [e.name for e in adder.tree.list("")]: + raise S3Error(404, "NoSuchBucket", "no such bucket, or not yours to read") + Writes(self, db, cfg, acct, adder, bucket, key, q, headers, chunks.get("signed")).run() + except S3Error as e: + self._refuse(e, raw, headers) + except http_body.BodyError as e: + self._refuse(S3Error(e.status, e.code, e.message), raw, headers) + except share_import.TooLarge as e: + self._refuse(S3Error(400, "EntityTooLarge", str(e)), raw, headers) + except share_import.Rejected as e: + self._refuse(S3Error(400, "InvalidArgument", str(e)), raw, headers) + except share_import.Refused as e: + self._refuse(S3Error(403, "AccessDenied", str(e)), raw, headers) + except filetree.NotFound: + self._refuse(S3Error(403, "AccessDenied", "no such folder: files go in a namespace, or in Collections/…"), raw, headers) + except keyring.Locked: + self._refuse(S3Error(403, "AccessDenied", "this namespace is a locked vault: an owner unlocks it in the web app"), raw, headers) + except (BrokenPipeError, ConnectionResetError): + self.close_connection = True + except OSError as e: + if e.errno != errno.ENOSPC: + self.server.log(f"s3: {type(e).__name__}: {e}") + msg = "the server doesn't have room for this file" if e.errno == errno.ENOSPC else "something went wrong on the server" + self._refuse(S3Error(507 if e.errno == errno.ENOSPC else 500, "InternalError", msg), raw, headers) + except Exception as e: # noqa: BLE001 - say so, and keep serving + self.server.log(f"s3: {type(e).__name__}: {e}") + self._refuse(S3Error(500, "InternalError", "something went wrong on the server"), raw, headers) + + def _refuse(self, e, raw, headers): + try: + if not self._read and not http_body.drain(self.rfile, headers): + self.close_connection = True # the body is still coming (or never will): don't read it + self._fail(e, raw) + except (BrokenPipeError, ConnectionResetError): # the client went away mid-file + self.close_connection = True do_PUT = do_POST = do_DELETE = _write # noqa: N815 - http.server's names @@ -453,9 +537,13 @@ def _handle(self, head): return self._send(200, _xml(f'<LocationConstraint xmlns="{NS}"></LocationConstraint>'), head=head) if head: return self._send(200, head=True) - if set(q) & {"acl", "policy", "versioning", "lifecycle", "cors", "tagging", "uploads", "website", "object-lock"}: - raise S3Error(501, "NotImplemented", "the file tree only lists and reads") + if "uploads" in q: + return self._send(200, list_uploads(cfg, acct, bucket)) + if set(q) & {"acl", "policy", "versioning", "lifecycle", "cors", "tagging", "website", "object-lock"}: + raise S3Error(501, "NotImplemented", "the file tree only lists, reads and takes files") return self._send(200, list_objects(tree, bucket, q)) + if "uploadId" in q and not head: + return self._send(200, list_parts(cfg, acct, bucket, key, q["uploadId"])) return self._object(tree, db, cfg, bucket, key, head) except S3Error as e: self._fail(e, raw) @@ -496,6 +584,8 @@ def _object(self, tree, db, cfg, bucket, key, head): self.send_header("Accept-Ranges", "bytes") if status == 206: self.send_header("Content-Range", f"bytes {start}-{end}/{e.size}") + if e.mtime: + self.send_header("x-amz-meta-mtime", e.mtime) # as rclone sent it self.end_headers() if head or end < start: return @@ -503,6 +593,293 @@ def _object(self, tree, db, cfg, bucket, key, head): self.wfile.write(chunk) +# ---------- adding files ---------- +def _mtime(headers): + """The file's own time a client sent (rclone: x-amz-meta-mtime, seconds since 1970, as text), or None.""" + return headers.get("x-amz-meta-mtime") or None + + +def _limit(cfg): + return int(cfg["uploads"]["max_mb"]) * uploads.MB + + +class Writes: + """One request that writes (PUT, POST, DELETE), for an account allowed to add files.""" + + def __init__(self, h, db, cfg, acct, adder, bucket, key, q, headers, signed): + self.h, self.db, self.cfg, self.acct, self.adder = h, db, cfg, acct, adder + self.bucket, self.key, self.q, self.headers, self.signed = bucket, key, q, headers, signed + self.path = f"{bucket}/{key}" + + def run(self): + m, q = self.h.command, self.q + if not self.key: # POST to the bucket + if "delete" in q: + return self.delete_many() + raise S3Error(501, "NotImplemented", "the file tree only lists, reads and takes files") + if "uploadId" in q: + if m == "PUT" and "partNumber" in q: + return self.part() + if m == "POST": + return self.complete() + if m == "DELETE": + return self.abort() + if m == "POST" and "uploads" in q: + return self.start() + if m == "DELETE": + return self.delete() + if m == "PUT" and not set(q) - {"x-id"}: # x-id: the AWS SDKs' operation name + if self.headers.get("x-amz-copy-source"): + return self.retime() + return self.folder() if self.key.endswith("/") else self.put() + raise S3Error(501, "NotImplemented", "copying, tags and the like aren't served: put a file, or upload it in parts") + + def _declared(self): + n = self.headers.get("x-amz-decoded-content-length") or self.headers.get("content-length") or "" + return int(n) if n.isdigit() else None + + def _receive(self, dest): + """The body into `dest`: (size, md5 hex).""" + self.h._go_on() + self.h._read = True + if "chunked" in self.headers.get("transfer-encoding", "").lower(): + self.h.close_connection = True # a chunked body may have been read past its end + return http_body.receive(self.h.rfile, self.headers, dest, _limit(self.cfg), self.signed) + + def put(self): + self.adder.check(self.path, self._declared()) + dest = share_import.spool(self.cfg) + try: + _, md5 = self._receive(dest) + self.adder.put(self.path, dest, _mtime(self.headers), md5) # imported (moved away), or kept aside there + except BaseException: + share_import._remove(dest) + raise + self.h._send(200, headers={"ETag": f'"{md5}"'}) + + def retime(self): + """CopyObject of a file just added onto itself with new metadata: rclone setting its time. Nothing else.""" + source = urllib.parse.unquote(self.headers["x-amz-copy-source"].split("?")[0]).lstrip("/") + replace = self.headers.get("x-amz-metadata-directive", "").upper() == "REPLACE" + if source != self.path or not replace or not self.adder.touch(self.path, _mtime(self.headers)): + raise S3Error(501, "NotImplemented", "copying isn't served: put the file instead") + self.h._read = True + e = self.adder.tree.stat(self.path) + self.h._send(200, _xml(f'<CopyObjectResult xmlns="{NS}"><LastModified>{_iso(e.modified)}</LastModified>' + f"<ETag>"{e.version}"</ETag></CopyObjectResult>")) # fmt: skip + + def folder(self): + if self._declared(): + raise S3Error(400, "InvalidArgument", "a key ending in / is a folder: it holds no bytes") + self.h._read = True + self.adder.folder(self.path.rstrip("/")) + self.h._send(200, headers={"ETag": f'"{hashlib.md5(b"", usedforsecurity=False).hexdigest()}"'}) + + def delete(self): + self.h._read = True + try: + self.adder.delete(self.path) + except share_import.Refused: + try: + self.adder.tree.stat(self.path) + except filetree.NotFound: + return self.h._send(204) # nothing there: S3 says done + raise + self.h._send(204) + + def delete_many(self): + body = self._xml_body() + quiet = (body.findtext("{*}Quiet") or body.findtext("Quiet") or "").lower() == "true" + done, errs = [], [] + for o in body.iter(): + if o.tag.split("}")[-1] != "Object": + continue + k = next((c.text or "" for c in o if c.tag.split("}")[-1] == "Key"), "") + try: + self.adder.delete(f"{self.bucket}/{k}") + done.append(k) + except share_import.Refused as e: + try: + self.adder.tree.stat(f"{self.bucket}/{k}") + errs.append((k, str(e))) + except filetree.NotFound: + done.append(k) + out = [] if quiet else [f"<Deleted><Key>{escape(k)}</Key></Deleted>" for k in done] + out += [f"<Error><Key>{escape(k)}</Key><Code>AccessDenied</Code><Message>{escape(m)}</Message></Error>" for k, m in errs] + self.h._send(200, _xml(f'<DeleteResult xmlns="{NS}">' + "".join(out) + "</DeleteResult>")) + + def _xml_body(self): + n = self._declared() or 0 + if n > XML_MAX: + raise S3Error(400, "MalformedXML", "the request is too large") + dest = share_import.spool(self.cfg) + try: + self._receive(dest) + raw = dest.read_bytes() + finally: + share_import._remove(dest) + if len(raw) > XML_MAX: + raise S3Error(400, "MalformedXML", "the request is too large") + if b"<!DOCTYPE" in raw or b"<!ENTITY" in raw: + raise S3Error(400, "MalformedXML", "the request isn't valid") + try: + return ET.fromstring(raw) + except ET.ParseError: + raise S3Error(400, "MalformedXML", "the request isn't valid XML") from None + + # ---------- in parts ---------- + def start(self): + self.adder.check(self.path) + self.h._read = True + uid = start_upload(self.cfg, self.acct["id"], self.bucket, self.key, _mtime(self.headers)) + self.h._send(200, _xml(f'<InitiateMultipartUploadResult xmlns="{NS}"><Bucket>{escape(self.bucket)}</Bucket>' + f"<Key>{escape(self.key)}</Key><UploadId>{uid}</UploadId></InitiateMultipartUploadResult>")) # fmt: skip + + def part(self): + d, _ = upload_of(self.cfg, self.acct["id"], self.bucket, self.key, self.q["uploadId"]) + try: + n = int(self.q["partNumber"]) + except ValueError: + n = 0 + if not 1 <= n <= MAX_PARTS: + raise S3Error(400, "InvalidArgument", f"part numbers go from 1 to {MAX_PARTS}") + tmp = d / f"{n:05d}.part" + try: + size, md5 = self._receive(tmp) + os.replace(tmp, d / f"{n:05d}") + (d / f"{n:05d}.md5").write_text(md5) + finally: + share_import._remove(tmp) + if sum(p["size"] for p in parts_of(d)) > _limit(self.cfg): + raise S3Error(400, "EntityTooLarge", f"files up to {self.cfg['uploads']['max_mb']} MB") + self.h._send(200, headers={"ETag": f'"{md5}"'}) + + def complete(self): + d, meta = upload_of(self.cfg, self.acct["id"], self.bucket, self.key, self.q["uploadId"]) + body = self._xml_body() + listed = [] + for p in body.iter(): + if p.tag.split("}")[-1] != "Part": + continue + kids = {c.tag.split("}")[-1]: (c.text or "").strip() for c in p} + try: + listed.append((int(kids.get("PartNumber", "")), kids.get("ETag", "").strip('"'))) + except ValueError: + raise S3Error(400, "MalformedXML", "a part's number isn't a number") from None + if not listed: + raise S3Error(400, "MalformedXML", "list the parts to put together") + if [n for n, _ in listed] != sorted({n for n, _ in listed}): + raise S3Error(400, "InvalidPartOrder", "list the parts in order, each once") + have = {p["number"]: p for p in parts_of(d)} + for n, etag in listed: + if n not in have or (etag and etag != have[n]["etag"]): + raise S3Error(400, "InvalidPart", f"part {n} wasn't uploaded, or its ETag differs") + whole = share_import.spool(self.cfg) + md5s = hashlib.md5(usedforsecurity=False) + self.h._read = True + try: + with open(whole, "wb") as out: + for n, _ in listed: + with open(d / f"{n:05d}", "rb") as f: + shutil.copyfileobj(f, out, 1 << 20) + md5s.update(bytes.fromhex(have[n]["etag"])) + self.adder.put(self.path, whole, meta.get("mtime")) + except BaseException: + share_import._remove(whole) + raise + shutil.rmtree(d, ignore_errors=True) + etag = f"{md5s.hexdigest()}-{len(listed)}" + self.h._send(200, _xml(f'<CompleteMultipartUploadResult xmlns="{NS}"><Location>/{escape(self.path)}</Location>' + f"<Bucket>{escape(self.bucket)}</Bucket><Key>{escape(self.key)}</Key>" + f"<ETag>"{etag}"</ETag></CompleteMultipartUploadResult>")) # fmt: skip + + def abort(self): + d, _ = upload_of(self.cfg, self.acct["id"], self.bucket, self.key, self.q["uploadId"]) + self.h._read = True + shutil.rmtree(d, ignore_errors=True) + self.h._send(204) + + +def _uploads_dir(cfg, uid): + if not UPLOAD_ID.match(uid or ""): + raise S3Error(404, "NoSuchUpload", "no such upload") + return uploads.incoming(cfg) / f"s3mp-{uid}" + + +def start_upload(cfg, account, bucket, key, mtime=None): + """A new multipart upload of `key` by an account: its id. Uploads nobody finished within uploads.expire_hours go.""" + sweep_uploads(cfg) + uid = secrets.token_urlsafe(24) + d = _uploads_dir(cfg, uid) + d.mkdir() + meta = {"account": account, "bucket": bucket, "key": key, "created": store.now(), "mtime": mtime} + (d / "meta.json").write_text(json.dumps(meta)) + return uid + + +def upload_of(cfg, account, bucket, key, uid): + """(its folder, its meta) for one of the account's uploads of this key; NoSuchUpload otherwise.""" + d = _uploads_dir(cfg, uid) + try: + meta = json.loads((d / "meta.json").read_text()) + except (OSError, ValueError): + raise S3Error(404, "NoSuchUpload", "no such upload: it was finished, cancelled or expired") from None + if (meta.get("account"), meta.get("bucket"), meta.get("key")) != (account, bucket, key): + raise S3Error(404, "NoSuchUpload", "no such upload") + return d, meta + + +def parts_of(d): + out = [] + for p in sorted(d.glob("[0-9][0-9][0-9][0-9][0-9]")): + try: + out.append({"number": int(p.name), "size": p.stat().st_size, "etag": (d / f"{p.name}.md5").read_text().strip(), + "modified": p.stat().st_mtime}) # fmt: skip + except (OSError, ValueError): + continue + return out + + +def sweep_uploads(cfg): + cut = time.time() - int(cfg["uploads"]["expire_hours"]) * 3600 + for d in uploads.incoming(cfg).glob("s3mp-*"): + try: + if d.stat().st_mtime < cut and all(p.stat().st_mtime < cut for p in d.iterdir()): + shutil.rmtree(d, ignore_errors=True) + except OSError: + pass + + +def _stamp(t): + return dt.datetime.fromtimestamp(t, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.000Z") + + +def list_parts(cfg, acct, bucket, key, uid): + d, _ = upload_of(cfg, acct["id"], bucket, key, uid) + parts = "".join( + f"<Part><PartNumber>{p['number']}</PartNumber><LastModified>{_stamp(p['modified'])}</LastModified>" + f"<ETag>"{p['etag']}"</ETag><Size>{p['size']}</Size></Part>" + for p in parts_of(d) + ) + return _xml(f'<ListPartsResult xmlns="{NS}"><Bucket>{escape(bucket)}</Bucket><Key>{escape(key)}</Key><UploadId>{uid}</UploadId>' + f"<IsTruncated>false</IsTruncated>{parts}</ListPartsResult>") # fmt: skip + + +def list_uploads(cfg, acct, bucket): + """The account's unfinished multipart uploads in a bucket.""" + out = [] + for d in sorted(uploads.incoming(cfg).glob("s3mp-*")): + try: + meta = json.loads((d / "meta.json").read_text()) + except (OSError, ValueError): + continue + if meta.get("account") == acct["id"] and meta.get("bucket") == bucket: + out.append(f"<Upload><Key>{escape(meta['key'])}</Key><UploadId>{d.name[5:]}</UploadId>" + f"<Initiated>{_iso(meta.get('created'))}</Initiated></Upload>") # fmt: skip + return _xml(f'<ListMultipartUploadsResult xmlns="{NS}"><Bucket>{escape(bucket)}</Bucket><IsTruncated>false</IsTruncated>' + + "".join(out) + "</ListMultipartUploadsResult>") # fmt: skip + + def password_account(db, cfg, kid, secret): """The account an access key and its secret sign in as (WebDAV's Basic sign-in), or None.""" found = key_account(db, cfg, kid) diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index ac23542dd..bba6483eb 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -86,7 +86,7 @@ "triage", ), # bind is a startup setting only - "shares": ("enabled", "s3", "s3_port", "webdav", "webdav_port", "remote"), + "shares": ("enabled", "s3", "s3_port", "webdav", "webdav_port", "add_files", "remote"), "video": ( "sample_seconds", "scene_threshold", @@ -1425,7 +1425,7 @@ def _web_setting(key, value): def _share_setting(key, value): - if key in ("enabled", "s3", "webdav", "remote"): + if key in ("enabled", "s3", "webdav", "add_files", "remote"): if not isinstance(value, bool): raise ValueError(f"shares.{key} is true or false") return value diff --git a/fastapi_backend/app/domain/share_import.py b/fastapi_backend/app/domain/share_import.py new file mode 100644 index 000000000..56bdc84e7 --- /dev/null +++ b/fastapi_backend/app/domain/share_import.py @@ -0,0 +1,477 @@ +"""Files people add through the file shares (docs/file-tree.md#adding-files): a file written into the tree over S3, +WebDAV, SMB or NFS is imported as if it had been uploaded in the web app (uploads.take): the same checks of its type +and size, the namespace's pipeline, and the same file twice isn't added twice. + +Where it's written decides where it goes: + + <namespace>/Collections/<collection>/<inside it>/<file> that collection (made, with any above it, when missing) + <namespace>/<file>, <namespace>/Collections/<file> the namespace's default collection + <namespace>/Topics/…/<file>, <namespace>/Entities/…/<file> the default collection too (topics and entities come + from the analysis) + +It must be the person's to add to: the import permission in the namespace, or an editor's or admin's role on the +collection, as an upload in the web app. Making a collection takes what making one in the web app takes. Files Lens +made (an asset's own file, its .json, .txt and .pdf) can't be written over. + +A client that has just written a file looks for it under the name it gave, which the tree doesn't have (the asset +shows as `<title> [<id>]`). So the name stays in its folder, for the person who wrote it, for ARRIVED_SECONDS, +holding the asset's own file. Files an operating system or an app writes for itself (.DS_Store, ._ files, Thumbs.db, +desktop.ini, Office lock files, `.part` and `.tmp` files, anything hidden), and empty files (Finder and Explorer make +one before they write it) are kept aside instead, on the uploads disk for PENDING_SECONDS after they last changed, +and never imported; renaming one to an ordinary name imports it then (an upload under a temporary name). + +Each server process keeps these in memory (they last minutes); it's the person's own view, nobody else sees them. +""" + +from __future__ import annotations + +import datetime as dt +import mimetypes +import os +import pathlib +import re +import secrets +import threading +import time +from dataclasses import dataclass, field +from . import auth, filetree, hierarchy, store, uploads + +ARRIVED_SECONDS = 600 # how long a file's own name stays, after it was imported +PENDING_SECONDS = 600 # how long a file kept aside stays after it last changed +SWEEP_SECONDS = 60 +PROTOCOLS = ("s3", "webdav", "smb", "nfs") +# what an operating system or an app writes for itself, never imported: hidden files (.DS_Store, ._ AppleDouble, +# .~lock files), Office's ~$ owner files, Windows' Thumbs.db and desktop.ini, the Mac's Icon\r, NTFS streams (a:b), +# and files still arriving under a temporary name +JUNK = re.compile( + r"^(?:\..*|~\$.*|thumbs\.db|ehthumbs(?:_vista)?\.db|desktop\.ini|icon\r|.*:.*" + r"|.*\.(?:tmp|temp|part|partial|filepart|crdownload|download|swp|swx|lock))$", + re.IGNORECASE | re.DOTALL, +) + + +class Refused(PermissionError): + """Not theirs to add to, or no file can go there; says why.""" + + +class Rejected(ValueError): + """A file that can't be imported (its type, its size...); says why.""" + + +class TooLarge(Rejected): + """Over uploads.max_mb.""" + + +def accepting(cfg, protocol): + """Whether a share takes files people add: shares.add_files for S3 and WebDAV, smb.add_files, nfs.add_files.""" + section = "shares" if protocol in ("s3", "webdav") else protocol + return bool({**store.DEFAULTS[section], **(cfg.get(section) or {})}.get("add_files")) + + +def junk(name): + """Whether a file name is one an operating system or an app writes for itself (JUNK).""" + return bool(JUNK.match(name or "")) + + +def spool(cfg): + """A new file name on the uploads disk for a file to arrive in (not made yet); take() moves it from there.""" + return uploads.incoming(cfg) / f"share-{secrets.token_hex(12)}" + + +# ---------- what each person has just added ---------- +@dataclass +class Item: + kind: str # "arrived" (imported: `recording`) or "pending" (kept aside: `path`) + expires: float + recording: int | None = None + path: str | None = None + modified: str | None = None + md5: str | None = None # as the client sent it + mtime: str | None = None # the time the client gave it, as it wrote it (seconds since 1970) + holds: int = 0 # writers that have it open (SMB): it doesn't expire meanwhile + + +@dataclass +class Registry: + """{(account, folder's parts): {name: Item}}, in memory, guarded by a lock.""" + + items: dict[tuple[int, tuple[str, ...]], dict[str, Item]] = field(default_factory=dict) + lock: threading.Lock = field(default_factory=threading.Lock) + swept: float = 0.0 + + def folder(self, account, folder): + self.prune() + with self.lock: + return dict(self.items.get((account, tuple(folder)), {})) + + def get(self, account, folder, name): + return self.folder(account, folder).get(name) + + def put(self, account, folder, name, item): + with self.lock: + old = self.items.setdefault((account, tuple(folder)), {}).get(name) + self.items[(account, tuple(folder))][name] = item + if old and old.kind == "pending" and old.path and old.path != item.path: + _remove(old.path) + + def pop(self, account, folder, name): + with self.lock: + got = self.items.get((account, tuple(folder)), {}).pop(name, None) + if not self.items.get((account, tuple(folder)), True): + self.items.pop((account, tuple(folder)), None) + return got + + def hold(self, account, folder, name, by): + with self.lock: + it = self.items.get((account, tuple(folder)), {}).get(name) + if it is not None: + it.holds = max(0, it.holds + by) + it.expires = time.monotonic() + PENDING_SECONDS + + def prune(self): + now = time.monotonic() + gone = [] + with self.lock: + for key, names in list(self.items.items()): + for name, it in list(names.items()): + if it.expires < now and not it.holds: + gone.append(names.pop(name)) + if not names: + self.items.pop(key) + for it in gone: + if it.kind == "pending" and it.path: + _remove(it.path) + + def paths(self): + with self.lock: + return {it.path for names in self.items.values() for it in names.values() if it.path} + + +REGISTRY = Registry() + + +def _remove(path): + try: + os.unlink(path) + except OSError: + pass + + +def sweep(cfg): + """Files kept aside that this process no longer knows (it restarted) go from the uploads disk.""" + REGISTRY.prune() + if time.monotonic() - REGISTRY.swept < SWEEP_SECONDS: + return + REGISTRY.swept = time.monotonic() + keep, cut = REGISTRY.paths(), time.time() - 2 * max(PENDING_SECONDS, ARRIVED_SECONDS) + for p in uploads.incoming(cfg).glob("share-*"): + try: + if str(p) not in keep and p.stat().st_mtime < cut: + p.unlink() + except OSError: + pass + + +def _seconds(t): + """A time in seconds since 1970 (a number or its text) that's believable, or None.""" + try: + t = float(t) if t is not None else None + except ValueError: + return None + return t if t is not None and 0 < t < time.time() + 86400 else None + + +def _iso(t=None): + return dt.datetime.fromtimestamp(t if t is not None else time.time(), dt.timezone.utc).isoformat(timespec="seconds") + + +def overlay(tree, parts, taken): + """The files someone just added to a folder, as the tree shows them to them: named as they gave them, beside the + folder's own files (`taken`: their names, which win).""" + items = REGISTRY.folder(tree.viewer.account, parts) + if not items: + return [] + out = [] + arrived = {it.recording for it in items.values() if it.kind == "arrived"} + recs = ( + { + r["id"]: r + for r in tree.db.rows( + f"SELECT {filetree.REC_FIELDS} FROM recording WHERE id IN $ids", ids=[filetree.R("recording", int(x)) for x in arrived] + ) + } + if arrived + else {} + ) + for name, it in items.items(): + if name in taken: + continue + if it.kind == "arrived": + rec = recs.get(it.recording) + own = tree._primary(rec) if rec else None + if own is None: + continue + own.name, own.added, own.md5, own.mtime = name, "arrived", it.md5, it.mtime + out.append(own) + elif it.path and os.path.isfile(it.path): + ctype = mimetypes.guess_type(name)[0] or "application/octet-stream" + out.append(filetree.Entry(name, False, os.path.getsize(it.path), it.modified, ctype, path=it.path, added="pending", + md5=it.md5, mtime=it.mtime)) # fmt: skip + return out + + +# ---------- adding ---------- +@dataclass +class Place: + """Where a file is being written: its namespace, the collection it goes into (`collection`; None: the default), + or the collections still to make for it (`missing`, inside `parent`), its folder's parts and its name.""" + + space: int + namespace: str + folder: tuple[str, ...] + name: str + collection: int | None = None + parent: int | None = None + missing: list[str] = field(default_factory=list) + + +class Adder: + """One person adding files to the tree over one protocol (PROTOCOLS). `account`: their account row ({id, email, + admin}); `tree`: their view of the tree, when the server has one already.""" + + def __init__(self, db, cfg, account, protocol, tree=None): + self.db, self.cfg, self.account, self.protocol = db, cfg, account, protocol + self.tree = tree or filetree.Tree(db, cfg, filetree.Viewer.of(db, account)) + self.viewer: filetree.Viewer = self.tree.viewer + sweep(cfg) + + # ---------- where ---------- + def place(self, path, folder=False): + """Where a file (or, `folder`, a folder) at `path` goes. NotFound for a namespace or folder this person doesn't + see; Refused for a place nothing can be added.""" + parts = filetree.Tree.split(path) + if len(parts) < 2: + raise Refused("files go inside a namespace" if not folder else "namespaces are made in the web app") + tree = self.tree + sid = tree._space(parts[0]) + *up, name = parts + if name in (".", "..") or len(name.encode()) > 255: + raise Refused("that name can't be used") + top = up[1] if len(up) > 1 else None + where = Place(sid, parts[0], tuple(up), name) + if folder: + if top != filetree.COLLECTIONS: + raise Refused("folders can be made inside Collections only: each one is a collection") + self._walk(where, up[2:] + [name]) + return where + if top == filetree.COLLECTIONS: + self._walk(where, up[2:]) + elif top is not None and not self._view_folder(sid, top, up[2:]): + raise filetree.NotFound(path) + return where + + def _view_folder(self, sid, top, names): + """Whether Topics/… or Entities/… names a folder this person sees.""" + tree = self.tree + if top == filetree.TOPICS: + tid = None + for n in names[: filetree.TOPIC_DEPTH + 1]: + kids = tree._narrower(sid, tid) + if n not in kids: + return False + tid = kids[n] + return len(names) <= filetree.TOPIC_DEPTH + if top == filetree.ENTITIES: + if not names: + return True + types = tree._types(sid) + if names[0] not in types or len(names) > 2: + return False + return len(names) == 1 or names[1] in tree._entities(sid, types[names[0]]) + return False + + def _walk(self, where, names): + """Follow collection folders down from the top: the collection they reach, or the deepest one there is and the + names still to make inside it.""" + cid = None + for i, n in enumerate(names): + kids = self.tree._children(where.space, cid) + hit = kids.get(n) or next((c for k, c in kids.items() if k.casefold() == n.casefold()), None) + if hit is None: + where.parent, where.missing = cid, list(names[i:]) + return + cid = hit["id"] + where.collection = cid + + # ---------- may they ---------- + def may_import(self, sid, cid): + """The import permission in the namespace, or an editor's or admin's role on the collection.""" + if "import" in self.viewer.perms.get(sid, ()): + return True + role = self.viewer.collections.get(sid, {}).get(cid) if cid is not None else None + return hierarchy.ROLES.get(role or "", 0) >= auth.ROLES["editor"] + + def may_make(self, sid, parent): + """Editors of the namespace make collections anywhere in it; an admin of a collection, inside it.""" + return auth.allows(self.viewer.roles, sid, "editor") or ( + parent is not None and self.viewer.collections.get(sid, {}).get(parent) == "admin" + ) + + def _target(self, where): + return where.collection if where.collection is not None or where.missing else store.default_collection(self.db, where.space) + + def check(self, path, size=None): + """Whether this person may write a file at `path` (of `size` bytes, when known), before anything is sent: + the Place, or NotFound, Refused or Rejected.""" + where = self.place(path) + self._not_lens(where) + if where.missing: + if not self.may_make(where.space, where.parent): + raise Refused("making a collection needs editor access to the namespace, or admin of the collection it goes in") + if len(where.missing) + len(hierarchy.path(self.db, where.parent) if where.parent else []) > hierarchy.MAX_DEPTH: + raise Rejected(f"collections go at most {hierarchy.MAX_DEPTH} deep") + elif not self.may_import(where.space, self._target(where)): + raise Refused("adding files here needs the import permission (an editor's role)") + if not junk(where.name) and size: + try: + uploads.check(self.cfg, where.name, size) + except uploads.TooLarge as e: + raise TooLarge(str(e)) from None + except ValueError as e: + raise Rejected(str(e)) from None + return where + + def _not_lens(self, where): + """Files Lens made, and folders, can't be written over. Lens's files are all named `<title> [<id>].<ext>`.""" + if filetree.ASSET_ID.search(where.name): + try: + e = self.tree.stat("/".join([*where.folder, where.name])) + except filetree.NotFound: + return + if not e.added: + raise Refused("Lens made this file: it can't be written over") + return + top = where.folder[1] if len(where.folder) > 1 else None + if top is None: + folders = {filetree.COLLECTIONS, filetree.ENTITIES, filetree.TOPICS} + elif top == filetree.COLLECTIONS and not where.missing: + folders = set(self.tree._children(where.space, where.collection if len(where.folder) > 2 else None)) + else: + folders = set() + if where.name in folders: + raise Refused("that's a folder") + + def _make(self, where): + """The collections still missing on the way to `where`; the one it reaches.""" + cid = where.parent + for n in where.missing: + try: + cid = hierarchy.create(self.db, where.space, n, cid, by=self.account.get("email")) + except ValueError as e: + kids = self.tree._children(where.space, cid) + hit = next((c for k, c in kids.items() if k.casefold() == n.casefold()), None) # made meanwhile + if hit is None: + raise Rejected(str(e)) from None + cid = hit["id"] + else: + auth.audit(self.db, self._by(), "collection.create", f"collection:{cid}", {"name": n, "namespace": where.namespace, + "parent": where.parent, "via": self.protocol}) # fmt: skip + self.tree.forget() + where.collection, where.missing = cid, [] + return cid + + def _by(self): + return {"id": self.account["id"], "email": self.account.get("email")} + + # ---------- doing it ---------- + def folder(self, path): + """Make the collection a folder at `path` stands for (and any above it); whether one was made.""" + where = self.place(path, folder=True) + if not where.missing: + return False + if not self.may_make(where.space, where.parent): + raise Refused("making a collection needs editor access to the namespace, or admin of the collection it goes in") + self._make(where) + return True + + def touch(self, path, modified): + """Set the time of a file just added (an S3 client copying it onto itself with new metadata); False when it + isn't one.""" + parts = filetree.Tree.split(path) + if not parts or _seconds(modified) is None: + return False + with REGISTRY.lock: + it = REGISTRY.items.get((self.account["id"], tuple(parts[:-1])), {}).get(parts[-1]) + if it is None: + return False + it.mtime = str(modified) + return True + + def put(self, path, src, modified=None, md5=None): + """The file at `src` (on the uploads disk: spool(); it's moved away) was written at `path`: import it, or keep it + aside (junk or empty). `modified`: the file's own time in seconds since 1970 (a number, or the text the client + sent, which is shown back to it as it was), when the client said; `md5`: its MD5 (hex), when the server worked + it out. Returns {kind: "arrived", recording, duplicate} or {kind: "pending"}.""" + src = pathlib.Path(src) + stamp = str(modified) if modified is not None else None + modified = _seconds(modified) + where = self.check(path, src.stat().st_size) + key = (self.account["id"], where.folder, where.name) + if junk(where.name) or src.stat().st_size == 0: + if modified: + os.utime(src, (modified, modified)) + item = Item("pending", time.monotonic() + PENDING_SECONDS, path=str(src), modified=_iso(modified), md5=md5, mtime=stamp) + REGISTRY.put(*key, item) + return {"kind": "pending"} + # check() saw they may import there; whoever may make a collection may import into it + cid = self._make(where) if where.missing else self._target(where) + try: + up = uploads.take(self.db, self.cfg, where.namespace, str(src), where.name, self._by(), collection=cid, + modified=int(modified * 1000) if modified else None) # fmt: skip + except uploads.TooLarge as e: + raise TooLarge(str(e)) from None + except ValueError as e: + raise Rejected(str(e)) from None + finally: + _remove(src) # take() moved it unless it refused it + self.tree.forget() + rid = up.get("recording") + REGISTRY.put(*key, Item("arrived", time.monotonic() + ARRIVED_SECONDS, recording=rid, md5=md5, mtime=stamp)) + auth.audit(self.db, self._by(), "share.import", f"recording:{rid}", + {"via": self.protocol, "path": "/".join([*where.folder, where.name]), "duplicate": bool(up.get("duplicate"))}) # fmt: skip + return {"kind": "arrived", "recording": rid, "duplicate": bool(up.get("duplicate"))} + + def pending(self, path): + """A file kept aside at `path` for this person: its Item, or None.""" + parts = filetree.Tree.split(path) + if not parts: + return None + it = REGISTRY.get(self.account["id"], tuple(parts[:-1]), parts[-1]) + return it if it and it.kind == "pending" else None + + def delete(self, path): + """Delete a file kept aside (an app tidying up after itself); everything else is Refused.""" + parts = filetree.Tree.split(path) + it = self.pending(path) + if it is None: + raise Refused("files can be added here, not deleted") + REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) + if it.path: + _remove(it.path) + + def move(self, src, dst): + """Rename a file kept aside: to an ordinary name, it's imported then. Everything else is Refused.""" + it = self.pending(src) + if it is None or not it.path: + raise Refused("files can be added here, not moved or renamed") + parts = filetree.Tree.split(src) + moving = spool(self.cfg) + os.replace(it.path, moving) + REGISTRY.pop(self.account["id"], tuple(parts[:-1]), parts[-1]) + try: + return self.put(dst, moving, os.stat(moving).st_mtime) + except BaseException: + if moving.exists(): # back where it was, so the client can try again + os.replace(moving, it.path) + REGISTRY.put(self.account["id"], tuple(parts[:-1]), parts[-1], it) + raise diff --git a/fastapi_backend/app/domain/shares.py b/fastapi_backend/app/domain/shares.py index 6320bf3fc..13bfbb86a 100644 --- a/fastapi_backend/app/domain/shares.py +++ b/fastapi_backend/app/domain/shares.py @@ -1,5 +1,6 @@ -"""File shares (docs/file-tree.md): the servers that share the graph as files (filetree.py) on the network, read-only, with -people's own access keys (s3.py): S3 (s3.py) and WebDAV (webdav.py). +"""File shares (docs/file-tree.md): the servers that share the graph as files (filetree.py) on the network, with +people's own access keys (s3.py): S3 (s3.py) and WebDAV (webdav.py). Files people add through them are imported +(share_import.py) while `shares.add_files` is on. Off until an admin turns them on (`shares.enabled`, then `s3` and `webdav`); the process that runs routines (`lens worker`) follows the settings as they change, as the sensor hub does, and says what each server is doing in @@ -165,10 +166,12 @@ def start(db, cfg_fn, stop, log=None, name=None): def status(db, cfg): - """{protocol: {enabled, port, service}}: whether it's on, its port, and what the process running it says.""" + """{protocol: {enabled, port, service, add_files}}: whether it's on, its port, what the process running it says, + and whether files people add are imported.""" s, out = settings_of(cfg), {} for proto, (_, port) in PROTOCOLS.items(): row = db.one("SELECT * FROM $r", r=_row(proto)) or {} row.pop("id", None) - out[proto] = {"enabled": bool(s.get("enabled") and s.get(proto)), "port": int(s[port]), "service": row or None} + out[proto] = {"enabled": bool(s.get("enabled") and s.get(proto)), "port": int(s[port]), "service": row or None, + "add_files": bool(s.get("add_files"))} # fmt: skip return out diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 250f8f3f0..8304f3ccc 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -593,8 +593,9 @@ def labels(): "account": None, }, # the graph as files over S3 and WebDAV (shares.py, docs/file-tree.md): off until an admin turns it on. Then the process - # that runs routines (`lens worker`) serves it, read-only, on these ports. bind is a startup setting only. remote: also at - # the web app's address (/s3, /dav), so through Settings › Remote access and HTTPS too; off until an admin turns it on. + # that runs routines (`lens worker`) serves it on these ports; add_files imports the files people add to it + # (share_import.py), off: read-only. bind is a startup setting only. remote: also at the web app's address (/s3, /dav), + # so through Settings › Remote access and HTTPS too; off until an admin turns it on. "shares": { "enabled": False, "bind": "0.0.0.0", @@ -602,6 +603,7 @@ def labels(): "s3_port": 9555, "webdav": True, "webdav_port": 9556, + "add_files": True, "remote": False, }, # notifications to webhooks and Matterbridge (docs/notifications.md): targets reach public addresses only, and the diff --git a/fastapi_backend/app/domain/uploads.py b/fastapi_backend/app/domain/uploads.py index 69b445d53..67231bdcc 100644 --- a/fastapi_backend/app/domain/uploads.py +++ b/fastapi_backend/app/domain/uploads.py @@ -143,15 +143,10 @@ def _home(db, sid, collection): return store.default_collection(db, sid) -def start(db, cfg, ns, filename, size, by, title=None, modified=None, attach=None, pipeline=None, collection=None, hold=False): - """A new upload into namespace `ns`, by `by` ({id, email}). Its name, type and size are checked, and the disk - must have room for it; nothing is in the archive until the last byte arrives. `modified` is the file's own time - (milliseconds since 1970), which dates the recording when its name doesn't. `attach` is a transcript-only - recording in `ns` the file becomes the audio of, instead of a recording of its own; `pipeline` runs instead of the - namespace's once a new recording is made; `collection` (one of the namespace's) is where it goes, else the - namespace's default. `hold` keeps it out of the archive once it's all here (state "held", no namespace yet), for - the assistant to place with `place` (a file dropped into a conversation).""" - sweep(db, cfg) +def check(cfg, filename, size, attach=None): + """The name a file is kept under, once its type and size are ones that can be uploaded here (ValueError, or + TooLarge, saying why not): what start() checks before anything is sent, for servers that ask before they read a + file (the file shares, share_import.py).""" u = cfg["uploads"] name = clean_name(filename) ext = pathlib.Path(name).suffix @@ -169,6 +164,19 @@ def start(db, cfg, ns, filename, size, by, title=None, modified=None, attach=Non raise ValueError("the file is empty") if size > u["max_mb"] * MB: raise TooLarge(f"files up to {u['max_mb']} MB") + return name + + +def start(db, cfg, ns, filename, size, by, title=None, modified=None, attach=None, pipeline=None, collection=None, hold=False): + """A new upload into namespace `ns`, by `by` ({id, email}). Its name, type and size are checked, and the disk + must have room for it; nothing is in the archive until the last byte arrives. `modified` is the file's own time + (milliseconds since 1970), which dates the recording when its name doesn't. `attach` is a transcript-only + recording in `ns` the file becomes the audio of, instead of a recording of its own; `pipeline` runs instead of the + namespace's once a new recording is made; `collection` (one of the namespace's) is where it goes, else the + namespace's default. `hold` keeps it out of the archive once it's all here (state "held", no namespace yet), for + the assistant to place with `place` (a file dropped into a conversation).""" + sweep(db, cfg) + name = check(cfg, filename, size, attach) folder = _dir(cfg) / ".partial" folder.mkdir(parents=True, exist_ok=True) if shutil.disk_usage(folder).free < size + ROOM: @@ -434,12 +442,14 @@ def _attach(db, cfg, rid, sid, dest, st, fp, by, probed): return jobs.add_steps(db, rid, ATTACH_STEPS, by=by) -def take(db, cfg, ns, path, filename, by, title=None, pipeline=None, collection=None, hold=False): - """A whole file that arrived some other way (an import webhook, a chat channel), at `path` on the same disk as the - uploads folder, uploaded into namespace `ns` in one go: it's checked as start() checks a file, moved into place - and finished as if its last chunk had just arrived. The namespace must exist; with `hold` there's none yet (ns ""): - it's held for the assistant to place, as a file dropped into a conversation is. Returns the upload.""" - row = start(db, cfg, ns, filename, os.path.getsize(path), by, title=title, pipeline=pipeline, collection=collection, hold=hold) +def take(db, cfg, ns, path, filename, by, title=None, pipeline=None, collection=None, hold=False, modified=None): + """A whole file that arrived some other way (an import webhook, a chat channel, a file share), at `path` on the same + disk as the uploads folder, uploaded into namespace `ns` in one go: it's checked as start() checks a file, moved + into place and finished as if its last chunk had just arrived. The namespace must exist; with `hold` there's none + yet (ns ""): it's held for the assistant to place, as a file dropped into a conversation is. `modified` is the + file's own time, as for start(). Returns the upload.""" + size = os.path.getsize(path) + row = start(db, cfg, ns, filename, size, by, title=title, modified=modified, pipeline=pipeline, collection=collection, hold=hold) os.replace(path, _part(cfg, row["id"])) try: return received(db, cfg, row["id"]) diff --git a/fastapi_backend/app/domain/webdav.py b/fastapi_backend/app/domain/webdav.py index accae3b6d..d1bb0689d 100644 --- a/fastapi_backend/app/domain/webdav.py +++ b/fastapi_backend/app/domain/webdav.py @@ -1,9 +1,13 @@ -"""The asset tree over WebDAV (docs/file-tree.md#webdav): a read-only WebDAV server (RFC 4918, class 1) that Finder, -Windows Explorer, GNOME Files, Dolphin, rclone and davfs2 mount as a drive. +"""The asset tree over WebDAV (docs/file-tree.md#webdav): a WebDAV server (RFC 4918) that Finder, Windows Explorer, +GNOME Files, Dolphin, rclone and davfs2 mount as a drive. Each person signs in with one of their access keys (s3.py): the key id as the user name and its secret as the password (HTTP Basic). They see the namespaces and collections the web app shows them. OPTIONS, PROPFIND (depth 0 -or 1), GET and HEAD (with Range); anything that writes answers 403. Run by shares.py. +or 1), GET and HEAD (with Range). While `shares.add_files` is on, a file put into the tree is imported +(share_import.py): PUT (whole or chunked), MKCOL under Collections makes a collection, DELETE and MOVE act on what +share_import.py keeps aside (an app's own files, or a file uploaded under a temporary name and then renamed), and +LOCK, UNLOCK and PROPPATCH answer as clients expect (class 2: Finder and Windows mount a class 1 server read-only) +without storing anything. Anything else that writes answers 403. Run by shares.py. With `shares.remote` on, the web app relays it at its own address too (https://<web app>/dav/), and links in listings start with /dav (shares.web_path()). @@ -14,17 +18,26 @@ import base64 import datetime as dt import email.utils -import http.server +import errno import re +import secrets import urllib.parse +import xml.etree.ElementTree as ET from typing import Any from xml.sax.saxutils import escape -from . import auth, filetree, keyring, s3 +from . import auth, filetree, http_body, keyring, s3, share_import, uploads ALLOW = "OPTIONS, GET, HEAD, PROPFIND" +ALLOW_ADD = ALLOW + ", PUT, MKCOL, DELETE, MOVE, LOCK, UNLOCK, PROPPATCH" REALM = 'Basic realm="Lens", charset="UTF-8"' MAX_BODY = 1 << 20 +LOCK_SECONDS = 3600 + + +def _attr(s): + """Text for a double-quoted XML attribute.""" + return escape(s, {'"': """}) def _http_date(s): @@ -65,7 +78,7 @@ def multistatus(tree, parts, depth, prefix=""): return ('<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:">' + "".join(out) + "</D:multistatus>").encode() -class Handler(http.server.BaseHTTPRequestHandler): +class Handler(http_body.Lingering): server_version = "Lens" protocol_version = "HTTP/1.1" server: Any # shares.Server @@ -73,11 +86,14 @@ class Handler(http.server.BaseHTTPRequestHandler): def log_message(self, fmt, *args): pass + def _adds(self): + return share_import.accepting(self.server.cfg_fn(), "webdav") + def _send(self, status, body=b"", ctype="text/plain; charset=utf-8", headers=None): self.send_response(status) self.send_header("Content-Type", ctype) self.send_header("Content-Length", str(len(body))) - self.send_header("DAV", "1") + self.send_header("DAV", "1, 2" if self._adds() else "1") for k, v in (headers or {}).items(): self.send_header(k, v) self.end_headers() @@ -138,14 +154,216 @@ def _parts(self): def do_OPTIONS(self): # noqa: N802 - http.server's names self._drain() if not self._closed(): - self._send(200, headers={"Allow": ALLOW, "MS-Author-Via": "DAV"}) + self._send(200, headers={"Allow": ALLOW_ADD if self._adds() else ALLOW, "MS-Author-Via": "DAV"}) def _write(self): self._drain() if not self._closed(): self._send(403, b"the file tree is read-only\n", headers={"Allow": ALLOW}) - do_PUT = do_POST = do_DELETE = do_MKCOL = do_MOVE = do_COPY = do_PROPPATCH = do_LOCK = do_UNLOCK = _write # noqa: N815 + do_POST = do_COPY = _write # noqa: N815 + + def handle_expect_100(self): + """Hold back 100 Continue until the upload is known to be allowed (_go_on).""" + return True + + def _go_on(self): + if self.headers.get("expect", "").lower() == "100-continue": + self.send_response_only(100) + self.end_headers() + + def _adding(self): + """(the account, its Adder) for a request that writes, or None (an answer was sent).""" + if not self._adds(): + self._write() + return None + acct = self._account() + if acct is None: + self._skip() + return None + db = self.server.db + return acct, share_import.Adder(db, self.server.cfg_fn(), acct, "webdav", self._tree(acct)) + + def _skip(self): + """Don't read a body that's still to come: drop a small one, else close the connection after the answer.""" + if not http_body.drain(self.rfile, {k.lower(): v for k, v in self.headers.items()}): + self.close_connection = True + + def _refused(self, e): + """The answer for something share_import.py or the body reader turned down.""" + if isinstance(e, share_import.Refused): # a PermissionError (as keyring.Locked is), so before OSError + return self._send(403, f"{e}\n".encode()) + if isinstance(e, http_body.BodyError): + return self._send(413 if e.code == "EntityTooLarge" else e.status, f"{e.message}\n".encode()) + if isinstance(e, share_import.TooLarge): + return self._send(413, f"{e}\n".encode()) + if isinstance(e, share_import.Rejected): + return self._send(415, f"{e}\n".encode()) + if isinstance(e, filetree.NotFound): + return self._send(409, b"no such folder: files go in a namespace, or in Collections/...\n") + if isinstance(e, keyring.Locked): + return self._send(403, b"this namespace is a locked vault: an owner unlocks it in the web app\n") + if isinstance(e, OSError) and e.errno == errno.ENOSPC: + return self._send(507, b"the server doesn't have room for this file\n") + if isinstance(e, OSError): + self.server.log(f"webdav: {type(e).__name__}: {e}") + return self._send(500, b"something went wrong on the server\n") + return self._send(403, f"{e}\n".encode()) + + def do_PUT(self): # noqa: N802 + got = self._adding() + if got is None: + return + _, adder = got + headers = {k.lower(): v for k, v in self.headers.items()} + path = "/".join(self._parts()) + dest = share_import.spool(adder.cfg) + declared = headers.get("content-length") or headers.get("x-expected-entity-length") or "" + try: + try: + before = adder.tree.stat(path) + except filetree.NotFound: + before = None + adder.check(path, int(declared) if declared.isdigit() else None) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as e: + self._skip() + return self._refused(e) + try: + self._go_on() + if "chunked" in headers.get("transfer-encoding", "").lower(): + self.close_connection = True # a chunked body may have been read past its end + limit = int(adder.cfg["uploads"]["max_mb"]) * uploads.MB + _, md5 = http_body.receive(self.rfile, headers, dest, limit) + adder.put(path, dest, headers.get("x-oc-mtime"), md5) # imported (moved away), or kept aside there + except (share_import.Refused, share_import.Rejected, filetree.NotFound, http_body.BodyError, OSError) as e: + share_import._remove(dest) + if isinstance(e, http_body.BodyError): + self.close_connection = True + try: + return self._refused(e) + except (BrokenPipeError, ConnectionResetError): # the client went away mid-file + return None + self._send(204 if before is not None else 201, headers={"ETag": f'"{md5}"'}) + + def do_MKCOL(self): # noqa: N802 + got = self._adding() + if got is None: + return + _, adder = got + if int(self.headers.get("content-length") or 0) or self.headers.get("transfer-encoding"): + self._skip() + return self._send(415, b"MKCOL takes no body\n") + try: + made = adder.folder("/".join(self._parts())) + except (share_import.Refused, share_import.Rejected, filetree.NotFound) as e: + return self._refused(e) + self._send(201 if made else 405, b"" if made else b"there's a folder there already\n") + + def do_DELETE(self): # noqa: N802 + got = self._adding() + if got is None: + return + self._drain() + _, adder = got + path = "/".join(self._parts()) + try: + adder.delete(path) + except share_import.Refused as e: + try: + adder.tree.stat(path) + except filetree.NotFound: + return self._send(404, b"not found\n") + return self._refused(e) + self._send(204) + + def do_MOVE(self): # noqa: N802 + from . import shares + + got = self._adding() + if got is None: + return + self._drain() + _, adder = got + dest = shares.share_path(urllib.parse.urlsplit(self.headers.get("destination", "")).path, self._prefix()) + to = "/".join(urllib.parse.unquote(p) for p in dest.split("/") if p) + if not to: + return self._send(400, b"say where to move it (Destination)\n") + try: + existed = adder.tree.stat(to) is not None + except filetree.NotFound: + existed = False + if existed and self.headers.get("overwrite", "T").upper() == "F": + return self._send(412, b"there's a file there already\n") + try: + adder.move("/".join(self._parts()), to) + except (share_import.Refused, share_import.Rejected, filetree.NotFound, OSError) as e: + return self._refused(e) + self._send(204 if existed else 201) + + def do_LOCK(self): # noqa: N802 + """A lock that holds nothing back (one person's own files, nobody else writes them): what Finder and Windows + ask for before they write. On a name that isn't there yet it makes an empty file, as RFC 4918 says.""" + got = self._adding() + if got is None: + return + self._drain() + _, adder = got + path = "/".join(self._parts()) + status = 200 + try: + adder.tree.stat(path) + except filetree.NotFound: + empty = share_import.spool(adder.cfg) + try: + empty.touch() + adder.put(path, empty) + except (share_import.Refused, share_import.Rejected, filetree.NotFound, OSError) as e: + share_import._remove(empty) + return self._refused(e) + status = 201 + token = self.headers.get("if", "").strip("()<> ") # a refresh names the lock it renews + token = token if re.fullmatch(r"opaquelocktoken:[0-9a-f-]{1,64}", token) else f"opaquelocktoken:{secrets.token_hex(16)}" + depth = "0" if self.headers.get("depth", "infinity") == "0" else "infinity" + body = ( + '<?xml version="1.0" encoding="utf-8"?>\n<D:prop xmlns:D="DAV:"><D:lockdiscovery><D:activelock>' + "<D:locktype><D:write/></D:locktype><D:lockscope><D:exclusive/></D:lockscope>" + f"<D:depth>{depth}</D:depth><D:owner>{escape(str(got[0].get('email') or ''))}</D:owner>" + f"<D:timeout>Second-{LOCK_SECONDS}</D:timeout><D:locktoken><D:href>{token}</D:href></D:locktoken>" + f"<D:lockroot><D:href>{escape(href(self._parts(), False, self._prefix()))}</D:href></D:lockroot>" + "</D:activelock></D:lockdiscovery></D:prop>" + ).encode() + self._send(status, body, "application/xml; charset=utf-8", headers={"Lock-Token": f"<{token}>"}) + + def do_UNLOCK(self): # noqa: N802 + if self._adding() is None: + return + self._drain() + self._send(204) + + def do_PROPPATCH(self): # noqa: N802 + """Properties aren't kept: say each one was set, so clients that set times after a PUT (Windows) carry on.""" + got = self._adding() + if got is None: + return + n = int(self.headers.get("content-length") or 0) + raw = self.rfile.read(n) if 0 < n <= MAX_BODY else (self._drain() or b"") + try: + if b"<!DOCTYPE" in raw or b"<!ENTITY" in raw: + raise ET.ParseError("no DTDs") + names = [p.tag for x in ET.fromstring(raw).iter() if x.tag == "{DAV:}prop" for p in x] if raw else [] + except ET.ParseError: + return self._send(400, b"PROPPATCH takes an XML body\n") + try: + got[1].tree.stat("/".join(self._parts())) + except filetree.NotFound: + return self._send(404, b"not found\n") + props = "".join(f'<x:{t.split("}")[-1]} xmlns:x="{_attr(t[1:].split("}")[0] if "}" in t else "")}"/>' for t in names) + body = ( + f'<?xml version="1.0" encoding="utf-8"?>\n<D:multistatus xmlns:D="DAV:"><D:response>' + f"<D:href>{escape(href(self._parts(), False, self._prefix()))}</D:href><D:propstat><D:prop>{props}</D:prop>" + "<D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>" + ).encode() + self._send(207, body, "application/xml; charset=utf-8") def _tree(self, acct): db = self.server.db diff --git a/fastapi_backend/app/schemas/filetree.py b/fastapi_backend/app/schemas/filetree.py index ff6e88ef7..4e3b09b46 100644 --- a/fastapi_backend/app/schemas/filetree.py +++ b/fastapi_backend/app/schemas/filetree.py @@ -54,6 +54,7 @@ class ShareServer(ResponseModel): enabled: bool = Field(description="whether an admin turned it on (shares.enabled and its own switch)") port: int = Field(description="the port it listens on in the worker (Docker publishes LENS_S3_PORT, LENS_WEBDAV_PORT)") service: S3Service | None = Field(None, description="what the process running it says; none while it's off") + add_files: bool = Field(False, description="whether files people add through it are imported (shares.add_files)") class S3Status(ShareServer): diff --git a/fastapi_backend/tests/api/test_s3_writes.py b/fastapi_backend/tests/api/test_s3_writes.py new file mode 100644 index 000000000..ea88cb20b --- /dev/null +++ b/fastapi_backend/tests/api/test_s3_writes.py @@ -0,0 +1,288 @@ +"""Adding files over S3 (docs/file-tree.md#adding-files): a real S3 client (botocore) puts files into the tree and +they're imported where the path says, as an upload is; what may not be added is refused before it's sent.""" + +from __future__ import annotations + +import datetime as dt +import hashlib +import http.client +import io + +import botocore.session +import pytest +from botocore.config import Config +from botocore.exceptions import ClientError + +from app.domain import hierarchy, http_body, s3, share_import, shares, store +from tests.helpers import login, make_user, seed, text_pdf, web_relay + +PDF = text_pdf([["Harbour minutes", "The lamp was lit at dusk."]]) + + +@pytest.fixture +def env(client, db, cfg, folder): + seed(db, cfg, folder) + pods = store.ns_id(db, "pods") + talks = hierarchy.create(db, pods, "Talks") + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + conf = {"cfg": cfg} + server = shares.Server("127.0.0.1", 0, db, lambda: conf["cfg"], log=print).start() + keys = {} + for who in ("ed", "vi"): + h = login(client, f"{who}@x.io", f"{'editor' if who == 'ed' else 'viewer'} password 1") + k = client.post("/api/v1/shares/s3/keys", headers=h, json={}).json() + keys[who] = (k["id"], k["secret"]) + share_import.REGISTRY.items.clear() + yield {"port": server.port, "keys": keys, "talks": talks, "pods": pods, "conf": conf} + server.stop() + + +def _s3(env, who="ed", endpoint=None, **cfg): + kid, secret = env["keys"][who] + return botocore.session.get_session().create_client( + "s3", + endpoint_url=endpoint or f"http://127.0.0.1:{env['port']}", + region_name="us-east-1", + aws_access_key_id=kid, + aws_secret_access_key=secret, + config=Config(signature_version="s3v4", s3={"addressing_style": "path"}, retries={"max_attempts": 1}, **cfg), + ) + + +def _code(fn): + with pytest.raises(ClientError) as e: + fn() + return e.value.response["Error"]["Code"], e.value.response["Error"].get("Message") + + +def _recording(db, title): + return db.one("SELECT record::id(id) AS id, collection, source, size, recorded_at FROM recording WHERE title = $t", t=title) + + +def test_a_file_put_into_a_collection_is_imported_there(client, db, env): + c = _s3(env) + put = c.put_object(Bucket="pods", Key="Collections/Talks/minutes.pdf", Body=PDF, Metadata={"mtime": "1700000000.5"}) + assert put["ETag"] == f'"{hashlib.md5(PDF).hexdigest()}"' + rec = _recording(db, "minutes") + assert rec["collection"] == env["talks"] and rec["source"] == "document" and rec["size"] == len(PDF) + assert rec["recorded_at"].startswith("2023-11-14") # the file's own time, from rclone's mtime + assert db.one("SELECT VALUE id FROM job WHERE recording = $r LIMIT 1", r=rec["id"]) # the namespace's pipeline is queued + # the client finds it under the name it gave, beside the asset's own files + head = c.head_object(Bucket="pods", Key="Collections/Talks/minutes.pdf") + assert head["ContentLength"] == len(PDF) + assert c.get_object(Bucket="pods", Key="Collections/Talks/minutes.pdf")["Body"].read() == PDF + keys = [o["Key"] for o in c.list_objects_v2(Bucket="pods", Prefix="Collections/Talks/")["Contents"]] + assert "Collections/Talks/minutes.pdf" in keys and f"Collections/Talks/minutes [{rec['id']}].pdf" in keys + # only for them: others see the asset as Lens names it + others = [o["Key"] for o in _s3(env, "vi").list_objects_v2(Bucket="pods", Prefix="Collections/Talks/")["Contents"]] + assert "Collections/Talks/minutes.pdf" not in others and f"Collections/Talks/minutes [{rec['id']}].pdf" in others + audit = db.one("SELECT * FROM audit_log WHERE action = 'share.import'") + assert audit["target"] == f"recording:{rec['id']}" and audit["detail"]["via"] == "s3" and audit["email"] == "ed@x.io" + # the same file again isn't added twice + c.put_object(Bucket="pods", Key="Collections/Talks/again.pdf", Body=PDF) + assert len(db.values("SELECT VALUE id FROM recording WHERE source = 'document'")) == 1 + assert c.head_object(Bucket="pods", Key="Collections/Talks/again.pdf")["ContentLength"] == len(PDF) + + +def test_folders_become_collections(db, env): + c = _s3(env) + c.put_object(Bucket="pods", Key="Collections/Trips/Lisbon 2026/minutes.pdf", Body=PDF) + trips = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Trips") + lisbon = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Lisbon 2026") + assert lisbon["parent"] == trips["id"] and _recording(db, "minutes")["collection"] == lisbon["id"] + # a folder marker makes an empty one; a name differing only in case is the same collection + c.put_object(Bucket="pods", Key="Collections/trips/Porto/", Body=b"") + porto = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Porto") + assert porto["parent"] == trips["id"] + assert db.one("SELECT * FROM audit_log WHERE action = 'collection.create' AND detail.name = 'Porto'")["detail"]["via"] == "s3" + # anywhere else in the namespace: its default collection + c.put_object(Bucket="pods", Key="Topics/minutes 2.pdf", Body=PDF + b"\n%2") + assert _recording(db, "minutes 2")["collection"] == store.default_collection(db, env["pods"]) + assert _code(lambda: c.put_object(Bucket="pods", Key="Nowhere/x.pdf", Body=PDF))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="pods", Key="Topics/Made up/", Body=b""))[0] == "AccessDenied" + + +def test_files_added_at_the_web_apps_address(db, cfg, env): + """Through the web app's relay (/s3, docs/file-tree.md#at-the-web-apps-address) files are added as at the port, + once an admin opens it there (shares.remote): the client signs the path with /s3 in it.""" + with web_relay(env["port"], "/s3") as port: + c = _s3(env, endpoint=f"http://127.0.0.1:{port}/s3") + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/Talks/far.pdf", Body=PDF))[0] == "AccessDenied" + env["conf"]["cfg"] = {**cfg, "shares": {**(cfg.get("shares") or {}), "remote": True}} + c.put_object(Bucket="pods", Key="Collections/Talks/far.pdf", Body=PDF) + assert c.head_object(Bucket="pods", Key="Collections/Talks/far.pdf")["ContentLength"] == len(PDF) + assert _recording(db, "far")["collection"] == env["talks"] + + +def test_what_may_not_be_added_is_refused(db, cfg, env): + c = _s3(env) + assert _code(lambda: _s3(env, "vi").put_object(Bucket="pods", Key="Collections/Talks/x.pdf", Body=PDF))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="calls", Key="x.pdf", Body=PDF))[0] == "NoSuchBucket" + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/General/ep1 [1].json", Body=b"{}"))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/Talks", Body=PDF))[0] == "AccessDenied" # a folder + code, msg = _code(lambda: c.put_object(Bucket="pods", Key="tool.exe", Body=b"MZ")) + assert code == "InvalidArgument" and "can't be uploaded here" in msg + assert _code(lambda: c.create_bucket(Bucket="new-ns"))[0] == "AccessDenied" + assert ( + _code(lambda: c.copy_object(Bucket="pods", Key="b.pdf", CopySource="pods/Collections/General/ep1 [1].json"))[0] == "NotImplemented" + ) + assert _code(lambda: c.delete_object(Bucket="pods", Key="Collections/General/ep1 [1].json"))[0] == "AccessDenied" + c.delete_object(Bucket="pods", Key="Collections/General/nothing.pdf") # nothing there: done + env["conf"]["cfg"] = {**cfg, "uploads": {**cfg["uploads"], "max_mb": 1}} + assert _code(lambda: c.put_object(Bucket="pods", Key="big.pdf", Body=PDF + b"\0" * (2 << 20)))[0] == "EntityTooLarge" + env["conf"]["cfg"] = {**cfg, "shares": {**cfg["shares"], "add_files": False}} + code, msg = _code(lambda: c.put_object(Bucket="pods", Key="x.pdf", Body=PDF)) + assert code == "AccessDenied" and "read-only" in msg + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + + +def test_files_an_os_writes_for_itself_and_empty_files_are_kept_aside(db, env): + c = _s3(env) + c.put_object(Bucket="pods", Key="Collections/Talks/.DS_Store", Body=b"Bud1") + c.put_object(Bucket="pods", Key="Collections/Talks/report.pdf", Body=b"") + assert c.get_object(Bucket="pods", Key="Collections/Talks/.DS_Store")["Body"].read() == b"Bud1" + assert c.head_object(Bucket="pods", Key="Collections/Talks/report.pdf")["ContentLength"] == 0 + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + c.put_object(Bucket="pods", Key="Collections/Talks/report.pdf", Body=PDF) # then its bytes + assert _recording(db, "report")["collection"] == env["talks"] + out = c.delete_objects( + Bucket="pods", Delete={"Objects": [{"Key": "Collections/Talks/.DS_Store"}, {"Key": "Collections/Talks/ep9.pdf"}]} + ) + assert sorted(d["Key"] for d in out["Deleted"]) == ["Collections/Talks/.DS_Store", "Collections/Talks/ep9.pdf"] + assert _code(lambda: c.head_object(Bucket="pods", Key="Collections/Talks/.DS_Store"))[0] == "404" + # kept aside for the person who wrote them only, for a while + assert "Collections/Talks/report.pdf" not in [o["Key"] for o in _s3(env, "vi").list_objects_v2(Bucket="pods")["Contents"]] + for names in share_import.REGISTRY.items.values(): + for it in names.values(): + it.expires = 0 + assert "Collections/Talks/report.pdf" not in [o["Key"] for o in c.list_objects_v2(Bucket="pods")["Contents"]] + + +def test_multipart_uploads(db, env): + c = _s3(env) + body = PDF + b"%" * (6 << 20) + up = c.create_multipart_upload(Bucket="pods", Key="Collections/Talks/long.pdf") + uid = up["UploadId"] + assert [u["UploadId"] for u in c.list_multipart_uploads(Bucket="pods").get("Uploads", [])] == [uid] + e1 = c.upload_part(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid, PartNumber=1, Body=body[: 5 << 20])["ETag"] + e2 = c.upload_part(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid, PartNumber=2, Body=body[5 << 20 :])["ETag"] + assert [p["PartNumber"] for p in c.list_parts(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid)["Parts"]] == [1, 2] + # not someone else's, and not out of order + assert _code(lambda: _s3(env, "vi").abort_multipart_upload(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid))[0] in ( + "AccessDenied", + "NoSuchUpload", + ) + wrong = {"Parts": [{"PartNumber": 2, "ETag": e2}, {"PartNumber": 1, "ETag": e1}]} + assert ( + _code(lambda: c.complete_multipart_upload(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid, MultipartUpload=wrong))[0] + == "InvalidPartOrder" + ) + done = c.complete_multipart_upload( + Bucket="pods", + Key="Collections/Talks/long.pdf", + UploadId=uid, + MultipartUpload={"Parts": [{"PartNumber": 1, "ETag": e1}, {"PartNumber": 2, "ETag": e2}]}, + ) + assert done["ETag"].endswith('-2"') + rec = _recording(db, "long") + assert rec["collection"] == env["talks"] and rec["size"] == len(body) + assert c.get_object(Bucket="pods", Key="Collections/Talks/long.pdf", Range="bytes=0-4")["Body"].read() == b"%PDF-" + assert _code(lambda: c.list_parts(Bucket="pods", Key="Collections/Talks/long.pdf", UploadId=uid))[0] == "NoSuchUpload" + # an upload given up goes + up = c.create_multipart_upload(Bucket="pods", Key="Collections/Talks/gone.pdf")["UploadId"] + c.upload_part(Bucket="pods", Key="Collections/Talks/gone.pdf", UploadId=up, PartNumber=1, Body=PDF) + c.abort_multipart_upload(Bucket="pods", Key="Collections/Talks/gone.pdf", UploadId=up) + assert not c.list_multipart_uploads(Bucket="pods").get("Uploads") + + +def _signed_put(env, key, chunks, payload="STREAMING-AWS4-HMAC-SHA256-PAYLOAD", spoil=False, trailer=None): + """PUT `chunks` aws-chunked by hand, signed as SigV4 streaming uploads are (minio-go and the AWS SDKs do this).""" + kid, secret = env["keys"]["ed"] + now = dt.datetime.now(dt.UTC) + amz, date = now.strftime("%Y%m%dT%H%M%SZ"), now.strftime("%Y%m%d") + scope = f"{date}/us-east-1/s3/aws4_request" + host = f"127.0.0.1:{env['port']}" + path = s3.canonical_path(f"/pods/{key}") + size = sum(len(c) for c in chunks) + plain = not payload.startswith("STREAMING-") + headers = {"host": host, "x-amz-content-sha256": payload, "x-amz-date": amz} + if not plain: + headers.update({"x-amz-decoded-content-length": str(size), "content-encoding": "aws-chunked"}) + if trailer: + headers["x-amz-trailer"] = trailer[0] + names = sorted(headers) + canonical = "\n".join(["PUT", path, "", "".join(f"{n}:{headers[n]}\n" for n in names), ";".join(names), payload]) + to_sign = "\n".join(["AWS4-HMAC-SHA256", amz, scope, hashlib.sha256(canonical.encode()).hexdigest()]) + seed = s3.signature(secret, date, "us-east-1", "s3", to_sign) + signer = http_body.Signed(s3.signing_key(secret, date, "us-east-1", "s3"), amz, scope, seed) + body = io.BytesIO() + for c in [] if plain else [*chunks, b""]: + if payload.startswith("STREAMING-AWS4"): + sig = hmac_chunk(signer, c) + if spoil and c: + sig = "0" * 64 + body.write(f"{len(c):x};chunk-signature={sig}\r\n".encode()) + else: + body.write(f"{len(c):x}\r\n".encode()) + if c: + body.write(c + b"\r\n") + if trailer: + body.write(f"{trailer[0]}:{trailer[1]}\r\n".encode()) + body.write(b"".join(chunks) if plain else b"\r\n") + headers["authorization"] = f"AWS4-HMAC-SHA256 Credential={kid}/{scope},SignedHeaders={';'.join(names)},Signature={seed}" + conn = http.client.HTTPConnection("127.0.0.1", env["port"]) + conn.request("PUT", path, body=body.getvalue(), headers=headers) + r = conn.getresponse() + return r.status, r.read() + + +def hmac_chunk(signer, chunk): + import hmac + + to_sign = "\n".join(["AWS4-HMAC-SHA256-PAYLOAD", signer.amz, signer.scope, signer.prev, http_body.EMPTY_SHA256, + hashlib.sha256(chunk).hexdigest()]) # fmt: skip + signer.prev = hmac.new(signer.key, to_sign.encode(), hashlib.sha256).hexdigest() + return signer.prev + + +def test_streaming_uploads_have_their_chunks_checked(db, env): + status, _ = _signed_put(env, "Collections/Talks/streamed.pdf", [PDF[:100], PDF[100:]]) + assert status == 200 and _recording(db, "streamed")["size"] == len(PDF) + status, body = _signed_put(env, "Collections/Talks/spoilt.pdf", [PDF[:100], PDF[100:]], spoil=True) + assert status == 403 and b"SignatureDoesNotMatch" in body and not _recording(db, "spoilt") + # unsigned chunks with a CRC32 trailer (the AWS SDKs' default for uploads) + import base64 + import zlib + + other = PDF + b"\n%3" + crc = base64.b64encode(zlib.crc32(other).to_bytes(4, "big")).decode() + status, _ = _signed_put( + env, "Collections/Talks/trailed.pdf", [other], "STREAMING-UNSIGNED-PAYLOAD-TRAILER", trailer=("x-amz-checksum-crc32", crc) + ) + assert status == 200 and _recording(db, "trailed") + status, body = _signed_put( + env, "Collections/Talks/bad crc.pdf", [other + b"x"], "STREAMING-UNSIGNED-PAYLOAD-TRAILER", trailer=("x-amz-checksum-crc32", crc) + ) + assert status == 400 and b"BadDigest" in body + + +def test_a_body_that_isnt_what_was_signed_is_refused(db, env): + status, body = _signed_put(env, "Collections/Talks/x.pdf", [PDF], hashlib.sha256(b"something else").hexdigest()) + assert status == 400 and b"XAmzContentSHA256Mismatch" in body and not _recording(db, "x") + status, _ = _signed_put(env, "Collections/Talks/y.pdf", [PDF], hashlib.sha256(PDF).hexdigest()) + assert status == 200 and _recording(db, "y") + + +def test_people_with_a_role_on_a_collection_add_there_only(client, db, env): + uid = make_user(db, "co@x.io", "collection password 1") + hierarchy.give(db, env["talks"], uid, "admin") + h = login(client, "co@x.io", "collection password 1") + k = client.post("/api/v1/shares/s3/keys", headers=h, json={}).json() + env["keys"]["co"] = (k["id"], k["secret"]) + c = _s3(env, "co") + c.put_object(Bucket="pods", Key="Collections/Talks/Day 1/minutes.pdf", Body=PDF) + day = next(x for x in hierarchy.of_space(db, env["pods"]) if x["name"] == "Day 1") + assert day["parent"] == env["talks"] and _recording(db, "minutes")["collection"] == day["id"] + assert _code(lambda: c.put_object(Bucket="pods", Key="Collections/Other/x.pdf", Body=PDF))[0] == "AccessDenied" + assert _code(lambda: c.put_object(Bucket="pods", Key="x.pdf", Body=PDF))[0] == "AccessDenied" # the default collection isn't theirs diff --git a/fastapi_backend/tests/api/test_webdav.py b/fastapi_backend/tests/api/test_webdav.py index 007f9afa6..acca9bcf3 100644 --- a/fastapi_backend/tests/api/test_webdav.py +++ b/fastapi_backend/tests/api/test_webdav.py @@ -47,7 +47,7 @@ def _listing(body): def test_mount_and_read(env): status, h, _ = _call(env, "OPTIONS", "/", auth_header=False) - assert status == 200 and h["DAV"] == "1" and "PROPFIND" in h["Allow"] + assert status == 200 and h["DAV"] == "1, 2" and "PROPFIND" in h["Allow"] and "PUT" in h["Allow"] status, _, body = _call(env, "PROPFIND", "/", {"Depth": "1"}) assert status == 207 and [(x[0], x[1]) for x in _listing(body)] == [("/", True), ("/pods/", True)] status, _, body = _call(env, "PROPFIND", "/pods/Collections/General/", {"Depth": "1"}) @@ -65,6 +65,24 @@ def test_mount_and_read(env): assert status == 200 and h["Content-Length"] == str(size) and body == b"" +def test_read_only_when_files_are_not_taken(db, cfg, folder): + seed(db, cfg, folder) + uid = make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + from app.domain import s3 + + kid, secret = s3.create_key(db, cfg, uid) + off = {**cfg, "shares": {**cfg["shares"], "add_files": False}} + server = shares.Server("127.0.0.1", 0, db, lambda: off, handler=webdav.Handler).start() + env = {"port": server.port, "auth": "Basic " + base64.b64encode(f"{kid}:{secret}".encode()).decode()} + try: + status, h, _ = _call(env, "OPTIONS", "/") + assert h["DAV"] == "1" and "PUT" not in h["Allow"] + assert _call(env, "PUT", "/pods/Collections/General/x.pdf", body=b"%PDF-1.4")[0] == 403 + assert _call(env, "LOCK", "/pods/Collections/General/x.pdf")[0] == 403 + finally: + server.stop() + + def test_what_webdav_refuses(env): assert _call(env, "PROPFIND", "/", {"Depth": "1"}, auth_header=False)[0] == 401 assert _call(env, "PROPFIND", "/calls/", {"Depth": "1"})[0] == 404 # not theirs diff --git a/fastapi_backend/tests/api/test_webdav_writes.py b/fastapi_backend/tests/api/test_webdav_writes.py new file mode 100644 index 000000000..1f016685a --- /dev/null +++ b/fastapi_backend/tests/api/test_webdav_writes.py @@ -0,0 +1,177 @@ +"""Adding files over WebDAV (docs/file-tree.md#adding-files): what Finder, Windows Explorer and rclone send when +they copy files and folders onto the drive, and the files are imported where they were put.""" + +from __future__ import annotations + +import base64 +import http.client +import xml.etree.ElementTree as ET + +import pytest + +from app.domain import auth, hierarchy, share_import, shares, store, webdav +from tests.helpers import login, make_user, seed, text_pdf, web_relay + +D = "{DAV:}" +PDF = text_pdf([["Harbour minutes", "The lamp was lit at dusk."]]) + + +@pytest.fixture +def env(client, db, cfg, folder): + seed(db, cfg, folder) + pods = store.ns_id(db, "pods") + talks = hierarchy.create(db, pods, "Talks") + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + keys = {} + for who, pw in (("ed", "editor password 1"), ("vi", "viewer password 1")): + k = client.post("/api/v1/shares/s3/keys", headers=login(client, f"{who}@x.io", pw), json={}).json() + keys[who] = "Basic " + base64.b64encode(f"{k['id']}:{k['secret']}".encode()).decode() + conf = {"cfg": cfg} + server = shares.Server("127.0.0.1", 0, db, lambda: conf["cfg"], handler=webdav.Handler).start() + auth._FAILS.clear() + share_import.REGISTRY.items.clear() + yield {"port": server.port, "keys": keys, "talks": talks, "pods": pods, "conf": conf} + server.stop() + + +def _call(env, method, path, headers=None, body=None, who="ed", chunked=False, port=None): + c = http.client.HTTPConnection("127.0.0.1", port or env["port"], timeout=10) + h = {"Authorization": env["keys"][who], **(headers or {})} + if chunked: + c.putrequest(method, path) + for k, v in {**h, "Transfer-Encoding": "chunked"}.items(): + c.putheader(k, v) + c.endheaders() + for i in range(0, len(body), 100): + piece = body[i : i + 100] + c.send(f"{len(piece):x}\r\n".encode() + piece + b"\r\n") + c.send(b"0\r\n\r\n") + else: + c.request(method, path, body=body, headers=h) + r = c.getresponse() + out = (r.status, dict(r.getheaders()), r.read()) + c.close() + return out + + +def _names(env, path, who="ed"): + status, _, body = _call(env, "PROPFIND", path, {"Depth": "1"}, who=who) + assert status == 207, body + return [r.find(f".//{D}displayname").text for r in ET.fromstring(body).findall(f"{D}response")][1:] + + +def _recording(db, title): + return db.one("SELECT record::id(id) AS id, collection, size FROM recording WHERE title = $t", t=title) + + +def test_finder_copies_a_file(db, env): + """Finder: an empty file, a lock, the bytes chunked, its ._ file, PROPPATCH, unlock.""" + path = "/pods/Collections/Talks/minutes.pdf" + assert _call(env, "PUT", path, body=b"")[0] == 201 + status, h, body = _call(env, "LOCK", path, {"Timeout": "Second-600"}, body=b'<?xml version="1.0"?><D:lockinfo xmlns:D="DAV:"/>') + assert status == 200 and h["Lock-Token"].startswith("<opaquelocktoken:") and b"activelock" in body + status, h, _ = _call(env, "PUT", path, {"X-Expected-Entity-Length": str(len(PDF)), "If": f"({h['Lock-Token']})"}, PDF, chunked=True) + assert status == 204 and h["ETag"].strip('"') + assert _call(env, "PUT", "/pods/Collections/Talks/._minutes.pdf", body=b"\0\5\26\7AppleDouble")[0] == 201 + status, _, body = _call( + env, + "PROPPATCH", + path, + body=b'<?xml version="1.0"?><D:propertyupdate xmlns:D="DAV:" xmlns:Z="urn:schemas-microsoft-com:"><D:set><D:prop><Z:Win32LastModifiedTime>Thu, 15 Oct 2026 10:00:00 GMT</Z:Win32LastModifiedTime></D:prop></D:set></D:propertyupdate>', + ) + assert status == 207 and b"Win32LastModifiedTime" in body and b"200 OK" in body + assert _call(env, "UNLOCK", path, {"Lock-Token": "<opaquelocktoken:1>"})[0] == 204 + rec = _recording(db, "minutes") + assert rec["collection"] == env["talks"] and rec["size"] == len(PDF) + names = _names(env, "/pods/Collections/Talks/") + assert {"minutes.pdf", "._minutes.pdf", f"minutes [{rec['id']}].pdf"} <= set(names) + assert _call(env, "GET", path)[2] == PDF + assert "minutes.pdf" not in _names(env, "/pods/Collections/Talks/", "vi") # their own view only + assert _call(env, "DELETE", "/pods/Collections/Talks/._minutes.pdf")[0] == 204 # Finder tidies up + assert _call(env, "DELETE", path)[0] == 403 # an import isn't undone from here + + +def test_windows_copies_a_file(db, env): + """Windows Explorer: a lock on a new name makes it (empty), then the bytes, its times, unlock.""" + path = "/pods/notes.pdf" + status, h, _ = _call(env, "LOCK", path, {"Depth": "0"}, body=b'<?xml version="1.0"?><D:lockinfo xmlns:D="DAV:"/>') + assert status == 201 + status, h2, _ = _call(env, "LOCK", path, {"If": f"({h['Lock-Token']})"}) # renewed + assert status == 200 and h2["Lock-Token"] == h["Lock-Token"] + assert _call(env, "PUT", path, body=PDF)[0] == 204 + assert _recording(db, "notes")["collection"] == store.default_collection(db, env["pods"]) + + +def test_a_folder_dragged_in_becomes_collections(db, env): + assert _call(env, "MKCOL", "/pods/Collections/Trips/")[0] == 201 + assert _call(env, "MKCOL", "/pods/Collections/Trips/Lisbon/")[0] == 201 + assert _call(env, "MKCOL", "/pods/Collections/Trips/")[0] == 405 # there already + assert _call(env, "PUT", "/pods/Collections/Trips/Lisbon/day1.pdf", body=PDF)[0] == 201 + lisbon = next(c for c in hierarchy.of_space(db, env["pods"]) if c["name"] == "Lisbon") + assert _recording(db, "day1")["collection"] == lisbon["id"] + assert _call(env, "MKCOL", "/pods/Topics/New/")[0] == 403 + assert _call(env, "MKCOL", "/pods/Collections/Trips/x/", body=b"<x/>")[0] == 415 + + +def test_an_upload_under_a_temporary_name_is_imported_when_renamed(db, env): + assert _call(env, "PUT", "/pods/Collections/Talks/talk.pdf.filepart", body=PDF)[0] == 201 + assert not _recording(db, "talk") + status, _, _ = _call(env, "MOVE", "/pods/Collections/Talks/talk.pdf.filepart", + {"Destination": f"http://127.0.0.1:{env['port']}/pods/Collections/Talks/talk.pdf"}) # fmt: skip + assert status == 201 and _recording(db, "talk")["collection"] == env["talks"] + assert "talk.pdf.filepart" not in _names(env, "/pods/Collections/Talks/") + status, _, _ = _call(env, "MOVE", "/pods/Collections/General/ep1%20%5B1%5D.txt", {"Destination": "/pods/Collections/Talks/ep1.txt"}) + assert status == 403 # Lens's own files stay where they are + + +def test_files_added_at_the_web_apps_address(db, cfg, env): + """Through the web app's relay (/dav, docs/file-tree.md#at-the-web-apps-address) files are added as at the port, + once an admin opens it there (shares.remote); paths and links carry the /dav the client sees.""" + with web_relay(env["port"], "/dav") as port: + assert _call(env, "PUT", "/dav/pods/Collections/Talks/far.pdf", body=PDF, port=port)[0] == 403 + env["conf"]["cfg"] = {**cfg, "shares": {**cfg["shares"], "remote": True}} + assert _call(env, "PUT", "/dav/pods/Collections/Talks/far.pdf.part", body=PDF, port=port)[0] == 201 + status, _, _ = _call(env, "MOVE", "/dav/pods/Collections/Talks/far.pdf.part", + {"Destination": f"http://127.0.0.1:{port}/dav/pods/Collections/Talks/far.pdf"}, port=port) # fmt: skip + assert status == 201 + status, _, body = _call( + env, "LOCK", "/dav/pods/Collections/Talks/near.pdf", body=b'<?xml version="1.0"?><D:lockinfo xmlns:D="DAV:"/>', port=port + ) + assert status == 201 and b"<D:href>/dav/pods/Collections/Talks/near.pdf</D:href>" in body + assert _recording(db, "far")["collection"] == env["talks"] + + +def test_what_may_not_be_added_is_refused(db, cfg, env): + assert _call(env, "PUT", "/pods/Collections/Talks/x.pdf", body=PDF, who="vi")[0] == 403 + assert _call(env, "LOCK", "/pods/Collections/Talks/x.pdf", who="vi")[0] == 403 + assert _call(env, "PUT", "/pods/Collections/General/ep1%20%5B1%5D.json", body=b"{}")[0] == 403 + status, _, body = _call(env, "PUT", "/pods/tool.exe", body=b"MZ") + assert status == 415 and b"can't be uploaded here" in body + assert _call(env, "PUT", "/pods/Somewhere/x.pdf", body=PDF)[0] == 409 + assert _call(env, "PUT", "/calls/x.pdf", body=PDF)[0] == 409 # not theirs: no such folder, as for one that isn't there + assert _call(env, "PUT", "/x.pdf", body=PDF)[0] == 403 + assert _call(env, "COPY", "/pods/Collections/General/ep1%20%5B1%5D.txt", {"Destination": "/pods/x.txt"})[0] == 403 + env["conf"]["cfg"] = {**cfg, "uploads": {**cfg["uploads"], "max_mb": 1}} + # refused from what the client says it will send, before a byte of it is read + assert _ask(env, "PUT /pods/big.pdf", {"Content-Length": str(2 << 20), "Expect": "100-continue"}).startswith(b"HTTP/1.1 413") + assert _ask(env, "PUT /pods/big.pdf", {"X-Expected-Entity-Length": str(2 << 20), "Transfer-Encoding": "chunked"}).startswith( + b"HTTP/1.1 413" + ) + # and a chunked body that turns out larger than it said, once it has arrived; the client, still sending the rest, + # reads the answer (the server drops what follows rather than reset the connection) + big = PDF + b"\0" * (4 << 20) + assert _call(env, "PUT", "/pods/big.pdf", {"X-Expected-Entity-Length": "10"}, big, chunked=True)[0] == 413 + assert _call(env, "PUT", "/pods/big.pdf", {"Content-Length": str(len(big))}, big)[0] == 413 + assert _ask(env, "PUT /pods/x.pdf", {"Content-Length": "5", "Expect": "100-continue"}).startswith(b"HTTP/1.1 100") + assert not db.values("SELECT VALUE id FROM recording WHERE source = 'document'") + + +def _ask(env, line, headers): + """Send a request's headers only, and the first answer's status line and headers.""" + import socket + + with socket.create_connection(("127.0.0.1", env["port"]), timeout=10) as s: + h = {"Host": "x", "Authorization": env["keys"]["ed"], **headers} + s.sendall((f"{line} HTTP/1.1\r\n" + "".join(f"{k}: {v}\r\n" for k, v in h.items()) + "\r\n").encode()) + return s.recv(4096) diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 29b5fd8c8..17c90acc7 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -18492,6 +18492,12 @@ export type S3Status = { * what the process running it says; none while it's off */ service?: S3Service | null; + /** + * Add Files + * + * whether files people add through it are imported (shares.add_files) + */ + add_files?: boolean; webdav: ShareServer; /** * Remote @@ -20094,6 +20100,12 @@ export type ShareServer = { * what the process running it says; none while it's off */ service?: S3Service | null; + /** + * Add Files + * + * whether files people add through it are imported (shares.add_files) + */ + add_files?: boolean; [key: string]: unknown; }; diff --git a/nextjs-frontend/components/account/s3-keys.tsx b/nextjs-frontend/components/account/s3-keys.tsx index c6f1c8e2d..e7b2c0e04 100644 --- a/nextjs-frontend/components/account/s3-keys.tsx +++ b/nextjs-frontend/components/account/s3-keys.tsx @@ -54,8 +54,11 @@ export function S3Keys() { <div className="flex min-w-0 flex-1 flex-col gap-1"> <h2 className="text-[18px] font-bold leading-[1.2] text-fg">File share access keys</h2> <p className="text-[13.5px] leading-[1.4] text-fg-secondary"> - Read the archive as folders and files from any S3 client (rclone, Cyberduck, the AWS CLI). A key reads as - you; nothing can be changed through it. + Read the archive as folders and files from any S3 client (rclone, Cyberduck, the AWS CLI), or mount it as a + drive over WebDAV. A key acts as you + {status.data?.add_files + ? ": files you copy into a collection’s folder are imported there, where you may add files." + : "; nothing can be changed through it."} {status.data && !status.data.enabled && " An admin turns sharing on in Settings, File shares."} {status.data?.enabled && status.data.remote && diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 999c4c408..6245e0f6b 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -355,7 +355,7 @@ export const SECTIONS: SectionSpec[] = [ label: "File shares", backend: ["shares"], description: - "Off unless you turn it on. The archive’s assets as folders and files (namespaces and collections, a metadata file beside each asset), served read-only over S3 and WebDAV by Lens’s workers; people make their own access keys.", + "Off unless you turn it on. The archive’s assets as folders and files (namespaces and collections, a metadata file beside each asset), served over S3 and WebDAV by Lens’s workers; people make their own access keys, and files they add are imported where they put them.", }, { id: "uploads", @@ -2653,12 +2653,19 @@ export const FIELDS: FieldSpec[] = [ kind: "switch", hint: "Off: nothing listens. The API and the MCP tools read the tree either way", }, + { + section: "shares", + key: "add_files", + label: "Import files people add", + kind: "switch", + hint: "A file copied into a collection’s folder is imported there, as an upload; a new folder under Collections is a new collection. Off: read-only", + }, { section: "shares", key: "s3", label: "S3", kind: "switch", - hint: "Read-only; each person signs in with their own access key", + hint: "Each person signs in with their own access key", }, { section: "shares", key: "s3_port", label: "S3 port", kind: "int", min: 1, max: 65535 }, { @@ -2666,7 +2673,7 @@ export const FIELDS: FieldSpec[] = [ key: "webdav", label: "WebDAV", kind: "switch", - hint: "Read-only; mount it as a drive in Finder, Windows Explorer or a file manager", + hint: "Mount it as a drive in Finder, Windows Explorer or a file manager", }, { section: "shares", key: "webdav_port", label: "WebDAV port", kind: "int", min: 1, max: 65535 }, { diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index ec8c4b0f4..7ae00dacf 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -62844,6 +62844,12 @@ ], "description": "what the process running it says; none while it's off" }, + "add_files": { + "type": "boolean", + "title": "Add Files", + "description": "whether files people add through it are imported (shares.add_files)", + "default": false + }, "webdav": { "$ref": "#/components/schemas/ShareServer" }, @@ -65702,6 +65708,12 @@ } ], "description": "what the process running it says; none while it's off" + }, + "add_files": { + "type": "boolean", + "title": "Add Files", + "description": "whether files people add through it are imported (shares.add_files)", + "default": false } }, "additionalProperties": true, diff --git a/threat-model.json b/threat-model.json index 5e49187cd..8375d6f6f 100644 --- a/threat-model.json +++ b/threat-model.json @@ -570,7 +570,7 @@ "parameter": "any method, path, Host and body, passed to the share servers with X-Forwarded-Prefix and X-Forwarded-For", "attacker_controllable": "yes", "component": "optional", - "caller_must_enforce": "admin enables shares.enabled and shares.remote; per-person access keys, read-only; failed WebDAV sign-ins throttled per visitor from X-Forwarded-For as trusted proxies report it", + "caller_must_enforce": "admin enables shares.enabled and shares.remote; per-person access keys, files added imported as at the ports (shares.add_files); failed WebDAV sign-ins throttled per visitor from X-Forwarded-For as trusted proxies report it", "provenance": "documented", "source": "docs/file-tree.md \"At the web app's address\"" }, @@ -1252,7 +1252,7 @@ "property": "P-SHARES", "violation_symptom": "info-leak, integrity-bypass", "severity_tier": "security", - "conditions": "the file shares (S3, WebDAV) are off until an admin enables them, and reach the web app's address (/s3, /dav) only with shares.remote too; every request carries a person's own access key (SigV4 or Basic), reads as them and never writes", + "conditions": "the file shares (S3, WebDAV) are off until an admin enables them, and reach the web app's address (/s3, /dav) only with shares.remote too; every request carries a person's own access key (SigV4 or Basic) and reads as them; files written are imported only where that person may import, checked and size-capped as uploads, never over an asset or a file Lens made; shares.add_files off: read-only", "provenance": "documented", "source": "docs/file-tree.md \"Off until an admin turns it on\"" }, diff --git a/threat-model.md b/threat-model.md index 3e1ad5581..c933291a5 100644 --- a/threat-model.md +++ b/threat-model.md @@ -105,7 +105,7 @@ optional listeners (MQTT, syslog, tunnel, local model server) run inside workers | Ingest & processing (Ingest) | `routes/{uploads,files,imports}.py`, `domain/{ingest,documents,convert,video,uploads}.py`, `templates.py`, `render.py` | child processes (ffmpeg, LibreOffice, Chromium, poppler, Tesseract), files | In | | Query engines (Query) | `domain/cypher.py`, `domain/rdf.py` (SPARQL), `domain/rdf_import.py`, `domain/search.py` | DB | In | | Assistant & extensions (Assistant) | `domain/{chat,ai_tools,ops_tools,extensions,tool_nodes,flow,custom_nodes,code_tools,mcp_client,mcp_servers,marketplace}.py`, MCP write tools | network (model servers, admins' MCP servers, the MCP runner), child process (code tools) | In | -| Optional services, off by default (Optional) | `domain/{sensors,mqtt,syslog,shares,s3,webdav}.py`, `tunnel.py`, `https.py`, `local_llm.py`, `components.py`, `telemetry.py` | listening sockets, downloads, child processes | In | +| Optional services, off by default (Optional) | `domain/{sensors,mqtt,syslog,shares,s3,webdav,share_import,http_body}.py`, `tunnel.py`, `https.py`, `local_llm.py`, `components.py`, `telemetry.py` | listening sockets, downloads, child processes | In | | Packaging variants | `docker-compose.prod.yml`, `install.sh`, `cloudron/`, `packaging/{synology,qnap}`, `proxmox/` | host | §1.6 | | Dev and CI tooling, samples | `docker-compose.yml`, `make dev`, `.github/`, `ci/`, `examples/`, `overrides/`, `prod-frontend-deploy.yml`, `watcher.py`, `commands/`, `fastapi_backend/tests/` | host | Out (§1.3) | @@ -241,7 +241,7 @@ machine requires HTTPS, and a changed DB credential when SurrealDB runs as its o | `documents.web_networks`, `LENS_WEB_NETWORKS`; `notifications.networks` | empty | widen the outbound guard to listed private networks. `documents.web_networks` is in Settings → Documents for admins (and agents with an admin token), who can't list loopback, `0.0.0.0/8` or link-local (cloud metadata) networks there; only archive.yaml and `LENS_WEB_NETWORKS`, on the server, can | Yes (operator and admin input) | *(documented, docs/configuration.md "The web app can't list this machine (loopback) or link-local addresses")* | | Chromium sandbox | off as root or without user namespaces | the `netguard` proxy is the only barrier | Yes | *(documented, docs/configuration.md "runs with its sandbox where it can")* | | `sensors.enabled`, `sensors.mqtt_anonymous` | off (`bind` `0.0.0.0`); off | opens MQTT and syslog listeners; anonymous MQTT publish | Yes; anonymous is the admin's choice | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | -| `shares.enabled` | off (`bind` `0.0.0.0`) | opens the read-only S3 and WebDAV servers for the asset tree | Yes | *(documented, docs/file-tree.md "Off until an admin turns it on")* | +| `shares.enabled` | off (`bind` `0.0.0.0`) | opens the S3 and WebDAV servers for the asset tree; with `shares.add_files` (on) files people add through them are imported | Yes | *(documented, docs/file-tree.md "Off until an admin turns it on")* | | `shares.remote` | off | the web app relays the file shares at its own address (`/s3`, `/dav`; `nextjs-frontend/lens-relay.js`), so the tunnel and HTTPS reach them from the internet; until it's on, the share servers refuse requests carrying `X-Forwarded-Prefix` before any sign-in | Yes | *(documented, docs/file-tree.md "At the web app's address")* | | `tunnel.mode`, `local_llm.enabled`, `telemetry`, `analytics.enabled` | off | outbound tunnel (Cloudflare terminates TLS, Q30); local model server; OTLP export; GA4 Measurement Protocol export of daily counts | Yes | *(documented, docs/remote-access.md "It's off until an admin turns it on")* | | `https.enabled` (and install.sh's `LENS_HTTPS=1`, which Settings › HTTPS can also ask the upgrade agent for) | off | Caddy, fetched at run time, terminates TLS on ports 443/80 in the worker and proxies to the web app; the DNS provider's credentials are written to `data_dir/https/caddy.json` (mode 600), not a command line | Yes | *(documented, docs/https.md "Off by default")* | @@ -304,8 +304,8 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | automations' params and prompts, `POST /automations/preview` and `/automations/variables` | template text, a draft automation, an activity id | yes, anyone signed in (preview, the picker); the namespace editors who manage an automation (its params) | callback/code (declarative), data, x-prompt-injection | rendered in the Jinja sandbox (2 MB of output, `*` and `**` guarded; Q23); previews read only activities, namespaces and automations the person may see; prompts mark outside text as data | *(documented, docs/automations.md "The variable picker and preview")* | | NFS share (port 12049 on the host, 2049 in the worker) | NFS and MOUNT RPCs, client address | yes, once enabled, from `nfs.allowed_networks` | data, x-network-source | admin enables and picks the account; operator firewalls | *(documented, docs/file-shares.md "NFS has no login")* | | SMB share (port 1445 on the host, 4445 in the worker) | SMB2 requests, NTLM sign-in with a file access key, client address | yes, once enabled, from `smb.allowed_networks` | data, x-credential, x-network-source | admin enables; operator firewalls | *(documented, docs/smb.md "What keeps it safe")* | -| S3 9555 and WebDAV 9556 (file shares) | signed S3 requests (SigV4 header or presigned link); WebDAV with an access key as Basic sign-in | yes, once enabled | x-credential, resource-name | admin enables; each request carries a person's own access key; read-only; failed WebDAV sign-ins throttled per address | *(documented, docs/file-tree.md "Off until an admin turns it on")* | -| web app `/s3` and `/dav` (file shares relayed by `lens-relay.js`, ahead of Next) | any method, path, `Host` and body passed to the share servers with `X-Forwarded-Prefix` and `X-Forwarded-For` | yes, once `shares.enabled` and `shares.remote`; from the internet through the tunnel or HTTPS | x-credential, resource-name, x-network-source | admin enables both; the same per-person access keys, read-only; failed WebDAV sign-ins throttled per visitor from `X-Forwarded-For` as trusted proxies report it (`server.trusted_proxies`, `LENS_TRUSTED_PROXY_HOSTS`, loopback); `X-Forwarded-Prefix` sent straight to a share port only gets the sender refused or signed and linked under that path | *(documented, docs/file-tree.md "At the web app's address")* | +| S3 9555 and WebDAV 9556 (file shares) | signed S3 requests (SigV4 header or presigned link); WebDAV with an access key as Basic sign-in | yes, once enabled | x-credential, resource-name | admin enables; each request carries a person's own access key; files added are imported only where that person may import (`shares.add_files` off: read-only), checked as uploads, size-capped while streamed, never over Lens's own files; failed WebDAV sign-ins throttled per address | *(documented, docs/file-tree.md "Off until an admin turns it on")* | +| web app `/s3` and `/dav` (file shares relayed by `lens-relay.js`, ahead of Next) | any method, path, `Host` and body passed to the share servers with `X-Forwarded-Prefix` and `X-Forwarded-For` | yes, once `shares.enabled` and `shares.remote`; from the internet through the tunnel or HTTPS | x-credential, resource-name, x-network-source | admin enables both; the same per-person access keys, files added imported as at the ports (`shares.add_files`); failed WebDAV sign-ins throttled per visitor from `X-Forwarded-For` as trusted proxies report it (`server.trusted_proxies`, `LENS_TRUSTED_PROXY_HOSTS`, loopback); `X-Forwarded-Prefix` sent straight to a share port only gets the sender refused or signed and linked under that path | *(documented, docs/file-tree.md "At the web app's address")* | | `.env`, `archive.yaml`, `LENS_*`, `ARCHIVE_SECRET_KEY`, `RCLONE_BINARY` | process configuration | no (operator) | x-env-config | keep unreadable to others | *(maintainer, 2026-10)* | | workers and DB peers | DB credential, shared database rows | no (trusted peer) | x-peer | run peers on trusted hosts | *(inferred, Q9)* | | `.lens-control/inbox/upgrade` (host upgrade agent), `/metrics`, `.lens-control/prometheus-token` | `ref` (release tag or this install's branch; empty: the version running), with `https=`, `public=` (`0`/`1`) and `target=` (`full`/`lean`); `action=services` with `monitoring=on`/`off`; the scrape token | the inbox is writable by the containers (and local users); `/metrics` by anyone reaching the web app | x-env-config, credential | the agent installs only a release tag or the install's branch, changes only the HTTPS ports, the web app's network reach and the edition (any other value is dropped) and only toggles the `monitoring` profile, nothing else from the file, so a writer of the inbox can at most restart Lens with those choices or open it to the network; `/metrics` answers only the scrape token (constant-time compare); the token file is world-readable on the host and grants metrics only, which carry no personal content | *(documented, docs/updates.md, docs/prometheus.md "Privacy")* | @@ -547,7 +547,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-SENSORS | Sensors are off until an admin enables them. MQTT needs a hub login unless `mqtt_anonymous` is on. Syslog hears only listed networks. | integrity-bypass | security-critical | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | Voided by: `sensors.mqtt_anonymous` — `domain/sensors.py:984`; `sensors.syslog_networks`, `sensors.bind`. | | P-NFS | The NFS share is off until an admin turns it on. It is read-only, shows only what one chosen account sees in the web app, and turns away every client outside `nfs.allowed_networks` before rclone reads a byte. | info-leak | security-critical | *(documented, docs/file-shares.md "NFS has no login")* | Voided by: `nfs.allowed_networks` set wide, or a client address spoofed on the local network (NFS has no login) — `domain/share_relay.py`; Docker publishing that rewrites client addresses. | | P-SMB | The SMB share is off until an admin turns it on. It is read-only, signs people in with their own file access keys and shows each only what they see in the web app, refuses guests and SMB1 sessions, gives every connection its own NTLM challenge, and closes connections from outside `smb.allowed_networks` before reading a byte. | info-leak | security-critical | *(documented, docs/smb.md "What keeps it safe")* | Voided by: `smb.allowed_networks` set wide; SMB 2.0 has no encryption on the wire, so a key's NTLM exchange and the files read can be watched on the network — `domain/smb_share.py`. | -| P-SHARES | The file shares (S3 and WebDAV) are off until an admin enables them. Every request is signed (S3: AWS Signature Version 4) or signed in (WebDAV: Basic) with an access key someone made while signed in to the web app; it reads as them, sees only what the web app shows them, and never writes. Keys expire as API keys do, stop with a disabled account, and are revoked at once. | info-leak, integrity-bypass | security-critical | *(documented, docs/file-tree.md "Off until an admin turns it on")* | Through the web app (`/s3`, `/dav`) only with `shares.remote` on as well. Voided by: `shares.enabled` with `LENS_BIND=0.0.0.0` sends files, S3 signatures and WebDAV's Basic secret in clear over plain HTTP on the network, as does `shares.remote` when the web app itself is reached over plain HTTP; key secrets are sealed with the server key (`secret.key`), not hashed, since SigV4 signs with the secret itself — `domain/s3.py`, `domain/webdav.py`. | +| P-SHARES | The file shares (S3 and WebDAV) are off until an admin enables them. Every request is signed (S3: AWS Signature Version 4) or signed in (WebDAV: Basic) with an access key someone made while signed in to the web app; it reads as them and sees only what the web app shows them. A file it writes is imported as an upload by that person would be (the import permission or an editor's role on the collection; making a collection as the web app allows; the upload type and size checks, with the body streamed to disk and cut off at the size limit), never replaces or deletes an asset or a file Lens made, and with `shares.add_files` off nothing is written. Keys expire as API keys do, stop with a disabled account, and are revoked at once. | info-leak, integrity-bypass | security-critical | *(documented, docs/file-tree.md "Off until an admin turns it on")* | Through the web app (`/s3`, `/dav`) only with `shares.remote` on as well. Voided by: `shares.enabled` with `LENS_BIND=0.0.0.0` sends files, S3 signatures and WebDAV's Basic secret in clear over plain HTTP on the network, as does `shares.remote` when the web app itself is reached over plain HTTP; key secrets are sealed with the server key (`secret.key`), not hashed, since SigV4 signs with the secret itself; a leaked key can add files as its owner can — `domain/s3.py`, `domain/webdav.py`, `domain/share_import.py`. | | P-CHILD-SECRETS | The tunnel token reaches cloudflared in its environment, not its command line; llama-server gets its own random API key. | info-leak | correctness-only | *(documented, docs/remote-access.md "in its environment, not on its command line")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-REMOTE-NAMES | An IMAP message is named by mailbox validity and uid, so a rebuilt mailbox never serves another message under an old name. | wrong-output | correctness-only | *(documented, docs/processing.md "an old one never reads another message")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-NOTIFY | Each notification event is claimed once across processes and retried with backoff. | wrong-output | correctness-only | *(documented, docs/notifications.md "claimed once")* | Voided by: none identified (not required for this symptom; no exhaustive search) | diff --git a/threat-model.yaml b/threat-model.yaml index 72edbd42f..3fd9e4843 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:b41fc97af52afeb4402f75c826637ee297dee746ce119b8d5e8e009632f55775" +prose_version: "threat-model.md@sha256:35bef40c8c965ae68f3e5d5f839c22c8528f5526f251c4e5384deb840575ca60" model_status: under-review triage_policy: strict confidence: {documented: 204, maintainer: 46, inferred: 91, assumption: 8} @@ -502,7 +502,7 @@ entry_points: attacker_controllable: true x-condition: "once shares.enabled and shares.remote; from the internet through the tunnel or HTTPS" control_kinds: [x-credential, resource-name, x-network-source] - caller_must_enforce: "admin enables both; per-person access keys, read-only; failed WebDAV sign-ins throttled per visitor from X-Forwarded-For as trusted proxies report it" + caller_must_enforce: "admin enables both; per-person access keys, files added imported as at the ports (shares.add_files); failed WebDAV sign-ins throttled per visitor from X-Forwarded-For as trusted proxies report it" provenance: {kind: documented, source: 'docs/file-tree.md "At the web app''s address"'} - id: ".env, archive.yaml, LENS_*, ARCHIVE_SECRET_KEY, RCLONE_BINARY" component: outbound @@ -1227,7 +1227,7 @@ build_flags: default: "off" security_relevant: true maintainer_stance: supported - x-scope: "opens the read-only S3 and WebDAV servers for the asset tree on shares.s3_port and shares.webdav_port" + x-scope: "opens the S3 and WebDAV servers for the asset tree on shares.s3_port and shares.webdav_port; with shares.add_files (on) files people add are imported where they may import" affects_properties: - {property_id: P-SHARES, effect: narrows} provenance: {kind: documented, source: 'docs/file-tree.md "Off until an admin turns it on"'} @@ -1691,9 +1691,9 @@ properties_claimed: kind: authentication components: [optional] tier: security-critical - conditions: "the file shares (S3, WebDAV) are off until an admin enables them, and reach the web app's address (/s3, /dav) only with shares.remote too; every request carries a person's own access key (SigV4 or Basic), reads as them and never writes" + conditions: "the file shares (S3, WebDAV) are off until an admin enables them, and reach the web app's address (/s3, /dav) only with shares.remote too; every request carries a person's own access key (SigV4 or Basic) and reads as them; files written are imported only where that person may import, checked and size-capped as uploads, never over an asset or a file Lens made; shares.add_files off: read-only" violation_symptoms: [info-leak, integrity-bypass] - x-voided-by: "plain HTTP on the network with LENS_BIND=0.0.0.0, or shares.remote with the web app itself on plain HTTP (WebDAV's Basic secret in clear); secrets sealed with the server key, not hashed (domain/s3.py)" + x-voided-by: "plain HTTP on the network with LENS_BIND=0.0.0.0, or shares.remote with the web app itself on plain HTTP (WebDAV's Basic secret in clear); secrets sealed with the server key, not hashed (domain/s3.py); a leaked key adds files as its owner can (domain/share_import.py)" provenance: {kind: documented, source: 'docs/file-tree.md "Off until an admin turns it on"'} - id: P-CHILD-SECRETS kind: confidentiality