Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
sign-in link from email or an admin. Now Profile and sign-in › Passkeys lists Lens's other addresses with **Open
signed in**, which opens one with a 15-minute sign-in link of your own, where one fingerprint adds a passkey for it
([Authentication](docs/authentication.md#signing-in-passkeys)).
- **Files added over NFS are imported.** A file copied into a collection's folder on the NFS share is imported a moment
after it's closed, as the account the share shows the tree as (`nfs.account`), with the same checks as an upload.
rclone keeps files being written in a cache on Lens's disk and puts them to Lens once they're closed; a file Lens
can't take shows in the share's log. **Import files people add** in Settings, NFS share (`nfs.add_files`, on) turns it
off and serves the share read-only again ([Adding files](docs/file-shares.md#adding-files)).
- **Export a conversation.** `GET /api/v1/chats/<id>/export.md`, `.pdf` or `.json` gives the version showing, one
message (`?message=`) or, as JSON, every version (`?all=true`). The Markdown has each question and answer, its
citations as footnotes linking back to the moment in Lens and the tools it used; the PDF is that Markdown printed by
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
answer OpenAI's chat completions API for any app, cited from that namespace alone, each off until its owner turns it
on, with API keys that open that one namespace and nothing else. The same folders and files can be mounted over S3,
WebDAV, NFS or SMB (S3 and WebDAV also at the web app's address, through remote access), so file managers and scripts
read Lens too, and a file copied in over S3, WebDAV or SMB is imported where it was put. The web app runs on the same
REST API, so an agent can do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP
read Lens too, and a file copied into any of them is imported where it was put. The web app runs on the same REST
API, so an agent can do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP
server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app),
[API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md);
[checks](docs/requirements.md#connect-your-agents))
Expand Down
39 changes: 29 additions & 10 deletions docs/file-shares.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,22 @@ and assets are files inside them.
## NFS

Settings → NFS share → **Share the file tree over NFS**. It's off until you turn it on, and then a worker starts
serving within seconds. The share is **read-only**.
serving within seconds. A file copied into a collection's folder is imported there, as the share's account
([Adding files](#adding-files)); turn **Import files people add** off and the share is read-only.

Mount it on Linux (Settings shows the command with your port filled in):
Mount it on Linux (Settings shows the command with your port filled in; add `ro` to only read):

```sh
sudo mount -t nfs -o port=12049,mountport=12049,tcp,vers=3,ro <lens-machine>:/ /mnt/lens
sudo mount -t nfs -o port=12049,mountport=12049,tcp,vers=3 <lens-machine>:/ /mnt/lens
```

On macOS: `sudo mount -t nfs -o port=12049,mountport=12049,tcp,vers=3,ro,resvport <lens-machine>:/ /Volumes/lens`.
On macOS: `sudo mount -t nfs -o port=12049,mountport=12049,tcp,vers=3,resvport <lens-machine>:/ /Volumes/lens`.

| Setting | Default | What it does |
| --- | --- | --- |
| `nfs.enabled` | off | Serve the share |
| `nfs.account` | the first admin | Whose view of the tree the share shows: it holds exactly what that person sees in the web app |
| `nfs.account` | the first admin | Whose view of the tree the share shows: it holds exactly what that person sees in the web app, and files written to it are added as that person |
| `nfs.add_files` | on | Import the files written to the share (**Import files people add**); off, rclone serves it read-only |
| `nfs.allowed_networks` | private networks (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`, `fc00::/7`, `::1/128`) | Machines anywhere else are turned away before they read anything; changes apply without a restart |
| `nfs.port` | 2049 | The port inside Lens (with Docker, inside the worker container) |

Expand All @@ -34,18 +36,34 @@ on its own while the web app stays local.
NFS (version 3) trusts the network: there is no password, passkey or token. So Lens:

- serves nothing until an admin turns it on;
- serves it read-only, so nothing reaching the share can change the archive;
- adds files only as one account may (`nfs.account`), checked as uploads are, and never writes over, renames or
deletes what Lens made; with `nfs.add_files` off it serves the tree read-only, so nothing reaching the share can
change the archive;
- shows one account's view (`nfs.account`), never more than that person may see; pick an account that sees only what
the share should show (a member of just the namespaces or collections to share), not an admin, when the network has
people who shouldn't see everything;
people who shouldn't see everything, or add to everything;
- turns away every client outside `nfs.allowed_networks`, and counts them in Settings.

Anyone on an allowed network can read the share, and an address on the local network can be spoofed, so list only your
own networks, and keep the port off the internet.
Anyone on an allowed network can read the share and add files to it as its account, and an address on the local
network can be spoofed, so list only your own networks, and keep the port off the internet.

### Adding files

Copy files into a collection's folder (`cp` onto the mount, or a file manager) and each is imported as an upload is,
as the share's account: the same checks of its type and size, the namespace's pipeline, and the same file twice is
added once. A file anywhere else in a namespace goes to its default collection, and one written inside a new folder
under `Collections` makes that collection ([Adding files](file-tree.md#adding-files) has the rest).

rclone keeps a file being written in a cache on Lens's disk (`data_dir/cache/nfs`) and puts it to Lens a couple of
seconds after it's closed, so it shows up in Lens a moment later. NFS has no way to tell the client that Lens turned a
file down after it was written, so a file Lens can't take (its type, its size) shows in Settings → NFS share's log,
and rclone tries it again later. An empty new folder lasts as long as rclone remembers it; it becomes a
collection when a file is written into it.

### How it works

A worker runs `rclone serve nfs --read-only` (rclone is in Lens's image already; MIT licensed) over Lens's own S3
A worker runs `rclone serve nfs` (rclone is in Lens's image already; MIT licensed; `--read-only` while
`nfs.add_files` is off) over Lens's own S3
server (a copy on 127.0.0.1 inside the worker, so the S3 share itself needn't be on), signed in with an S3 key
made for `nfs.account`, which is revoked when the share stops. rclone listens on 127.0.0.1 inside the container; a small
relay in Lens (`domain/share_relay.py`) holds the published port and hands rclone only the connections from allowed
Expand All @@ -58,3 +76,4 @@ state at `GET /api/v1/settings/nfs/status` and change it with `PUT /api/v1/setti
- NFS version 4 and Kerberos (sec=krb5) for a share with a real login.
- A share per namespace or collection, each with its own account and networks.
- Packaging for the Synology and QNAP compose files, which don't publish the port yet.
- Tell the client when Lens turns a written file down, and make empty new folders collections at once.
15 changes: 9 additions & 6 deletions docs/file-tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are
folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the
folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read
Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3, WebDAV or
SMB is imported there ([Adding files](#adding-files)).
Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3, WebDAV,
SMB or NFS is imported there ([Adding files](#adding-files)).

| Where | Status |
| --- | --- |
| The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built |
| An [S3 server](#s3) and a [WebDAV server](#webdav) (off until turned on); files added through them are imported | built |
| An [NFS share](file-shares.md) (read-only, off until turned on, allowed networks only) | built |
| An [NFS share](file-shares.md) (off until turned on, allowed networks only); files added through it are imported | built |
| An [SMB share](smb.md) (off until turned on); files added through it are imported | built |

Every way in serves the same tree, built by `app/domain/filetree.py`.
Expand Down Expand Up @@ -47,7 +47,8 @@ Names are made safe for every file system: `/ \ : * ? " < > |` and control chara
Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces
they have a role in, and in a namespace where they only have roles on some collections, those collections and the
topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The API only
reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers and the [SMB share](smb.md).
reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers and the [SMB](smb.md) and
[NFS](file-shares.md) shares.

## The API

Expand Down Expand Up @@ -118,7 +119,8 @@ behind TLS on a network you don't trust.

## Adding files

A file copied into the tree over S3, WebDAV or [SMB](smb.md) is imported, as an upload in the web app is: the same checks of its type
A file copied into the tree over S3, WebDAV, [SMB](smb.md) or [NFS](file-shares.md) is imported, as an upload in the
web app is: the same checks of its type
and size (Settings, Uploads), the namespace's pipeline, and the same file twice is added once. Where it's put decides
where it goes:

Expand All @@ -144,7 +146,8 @@ kept on the uploads disk, in memory per worker, so a restart forgets them.

Adding files is on whenever the shares are; turn it off in Settings, File shares (**Import files people add**,
`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The SMB
share has its own switch (`smb.add_files`, on). The [NFS](file-shares.md) share stays read-only for now.
and NFS shares have their own switches (`smb.add_files` and `nfs.add_files`, on); NFS adds files as the account it
shows the tree as, since NFS has no login.

## At the web app's address

Expand Down
2 changes: 1 addition & 1 deletion docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ request, to another check that proves the same thing; loosening a requirement or
| API keys that open one namespace and nothing else | [test_namespace_keys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_namespace_keys.py) `test_a_key_for_one_namespace_opens_its_servers_alone`, `test_it_has_the_persons_role_there_and_no_more` |
| The graph's folders and files mounted read-only over S3, WebDAV, NFS and SMB | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_people_read_their_namespaces_over_s3`, `test_what_s3_refuses`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_mount_and_read`, `test_what_webdav_refuses`; [test_nfs_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_nfs_share.py) `test_the_runner_follows_the_setting`; [test_smb_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_share.py) `test_reads_the_tree_as_its_key_sees_it`, `test_nothing_can_be_written` |
| S3 and WebDAV at the web app's address, through remote access, once an admin allows it | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`, `test_through_the_web_app_each_visitor_has_its_own_throttle` |
| A file copied in over S3, WebDAV or SMB is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused`; [test_smb_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_writes.py) `test_a_file_copied_into_a_collection_is_imported_there`, `test_new_folders_under_collections_are_new_collections`, `test_what_may_not_be_added_is_refused`, `test_the_share_is_read_only_while_add_files_is_off` |
| A file copied in over S3, WebDAV, SMB or NFS is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused`; [test_smb_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_writes.py) `test_a_file_copied_into_a_collection_is_imported_there`, `test_new_folders_under_collections_are_new_collections`, `test_what_may_not_be_added_is_refused`, `test_the_share_is_read_only_while_add_files_is_off`; [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_the_nfs_shares_key_follows_nfs_add_files` (NFS adds through rclone's S3 client) |
| The web app runs on the same REST API, through a client generated from its OpenAPI schema | [ci.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/ci.yml) `OpenAPI schema is up to date` |

## Ask, and get cited answers
Expand Down
31 changes: 20 additions & 11 deletions fastapi_backend/app/domain/nfs_share.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
"""The file tree over NFS (docs/file-tree.md, NFS): graph nodes as folders and assets as files, read-only, for the
machines on the networks you allow (Settings → NFS share).

Off by default. Turned on, one worker runs `rclone serve nfs --read-only` (rclone is in Lens's image already) over
Lens's own S3 server, signed in with an S3 key made for the account the share shows the tree as (`nfs.account`;
nobody picked: the first admin), so the share shows exactly what that person sees in the web app and nothing else.
"""The file tree over NFS (docs/file-tree.md, NFS): graph nodes as folders and assets as files, for the machines on
the networks you allow (Settings → NFS share).

Off by default. Turned on, one worker runs `rclone serve nfs` (rclone is in Lens's image already) over Lens's own S3
server, signed in with an S3 key made for the account the share shows the tree as (`nfs.account`; nobody picked: the
first admin), so the share shows exactly what that person sees in the web app and nothing else. While `nfs.add_files`
is on, rclone keeps files being written in a cache on disk and puts each one to the S3 server once it's closed, where
it's imported as that account adds it (share_import.py); off, rclone serves it --read-only.
NFS has no sign-in of its own, so rclone listens on 127.0.0.1 only and a relay (share_relay.py) on the published port
lets in only `nfs.allowed_networks`.

Expand All @@ -16,6 +18,7 @@
import datetime as dt
import json
import os
import pathlib
import shutil
import socket
import subprocess
Expand Down Expand Up @@ -68,8 +71,9 @@ def status(db, cfg):
"refused": s.get("refused") or 0,
"log": s.get("log") or [],
"process": s.get("owner") if live else None,
"mount": f"sudo mount -t nfs -o port={published},mountport={published},tcp,vers=3,ro <this machine>:/ /mnt/lens",
"mount": f"sudo mount -t nfs -o port={published},mountport={published},tcp,vers=3{'' if c['add_files'] else ',ro'} <this machine>:/ /mnt/lens",
"rclone": bool(rclone_binary()),
"add_files": bool(c["add_files"]),
}


Expand Down Expand Up @@ -198,8 +202,13 @@ def _start(self, cfg, c):
account = _account(self.db, c)
remote, env_extra = self.remote(self.db, self.cfg_fn, account)
inner = _free_port()
cmd = [exe, "serve", "nfs", remote, "--addr", f"127.0.0.1:{inner}", "--read-only", "--vfs-cache-mode", "off",
"--dir-cache-time", "30s", "--no-checksum"] # fmt: skip
cmd = [exe, "serve", "nfs", remote, "--addr", f"127.0.0.1:{inner}", "--dir-cache-time", "30s", "--no-checksum"]
if c["add_files"]: # files are written to a cache on disk, then put to the S3 server (imported) once closed
cache = pathlib.Path(cfg["data_dir"]) / "cache" / "nfs"
cache.mkdir(parents=True, exist_ok=True, mode=0o700)
cmd += ["--vfs-cache-mode", "writes", "--cache-dir", str(cache), "--vfs-write-back", "2s", "--vfs-cache-max-age", "10m"]
else:
cmd += ["--read-only", "--vfs-cache-mode", "off"]
env = {**os.environ, **env_extra} # the key in the environment, not on the command line where `ps` shows it
self.proc = subprocess.Popen(cmd, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1)
threading.Thread(target=self._read, args=(self.proc,), daemon=True, name="nfs-rclone-log").start()
Expand All @@ -218,7 +227,7 @@ def _start(self, cfg, c):
raise ShareError(f"port {c['port']}: {e.strerror or e}") from None
self.account, self.account_id = account.get("email"), account["id"]
self.phase, self.error, self.fails = "running", None, 0
self.log(f"nfs: serving the file tree as {self.account} on port {self.relay.port}")
self.log(f"nfs: serving the file tree as {self.account} on port {self.relay.port}" + ("" if c["add_files"] else " (read-only)"))

def _drop_key(self):
aid, self.account_id = self.account_id, None
Expand All @@ -243,7 +252,7 @@ def tick(self):
if self.proc or self.relay:
self.stop_serving()
return
sig = json.dumps([c["port"], c["account"], c["bind"]])
sig = json.dumps([c["port"], c["account"], c["bind"], c["add_files"]])
if sig != self.sig:
self.stop_serving()
self._drop_key()
Expand Down
Loading
Loading