Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **The graph as files.** Graph nodes are folders, nested along its links: collections inside collections, topics
inside their broader topics, entities by type. Each folder holds its assets' files, with a `.json` sidecar of
metadata and a `.txt` of the text; everyone sees what the web app shows them. Browse and read it through the API
(`/api/v1/tree`) and the MCP tools `list_files` and `read_file` ([The graph as files](docs/file-tree.md)).
S3, WebDAV, SMB and NFS servers that share it are in progress.
- **Budget meter and Costs page.** The top bar shows what the namespace picked spent against its budget, used and
left, highlighted when near or over, and follows the namespace switcher; it opens the details for a day, week or
month. **Costs** in the left menu shows spend by day, model and resource, and sets the namespace's budget
Expand Down
10 changes: 10 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -594,6 +594,16 @@ page was before each change, and `restore` puts an earlier version back (what it
lists the files and images attached to a page; `blobs` uploads (in chunks) and reads them, and `DELETE …/files?key=`
removes one.

## the graph as files

```
GET /api/v1/tree?path=
GET /api/v1/tree/file?path=
```

The archive as folders and files ([The graph as files](file-tree.md)): `tree` lists a folder (an empty path lists your
namespaces; in each, `Collections`, `Topics` and `Entities`), `tree/file` reads a file with byte ranges. Read-only; everyone sees what they can read in the web app.

## comments

```
Expand Down
62 changes: 62 additions & 0 deletions docs/file-tree.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# The graph as files

Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are
folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the
folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read Lens without learning its API.

| Where | Status |
| --- | --- |
| The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built |
| S3 and WebDAV servers (read-only, off until turned on) | in progress |
| SMB and NFS shares | in progress |

Every way in serves the same tree, built by `app/domain/filetree.py`.

## Layout

```text
<namespace>/
Collections/<collection>/<collection inside it>/… every asset, in its collection's folder
Topics/<topic>/<narrower topic>/… the assets about each topic (accepted ones)
Entities/<type>/<name>/ the assets that mention each entity
```

A folder holds the folders under it and its assets' files side by side. The same asset shows in every folder it
belongs to, and a topic with several broader topics shows under each of them.

An asset's files are named after it, `<title> [<id>]`, so names are unique and stable:

| File | What it holds |
| --- | --- |
| `<title> [<id>].<ext>` | its own file as it was imported: audio, video, PDF, Word file, image… (`.original.txt` for an imported transcript) |
| `<title> [<id>].json` | its metadata: Dublin Core terms (`identifier`, `title`, `type`, `format`, `created`, `date`, `subject` for its topics), its collection, its entities, and the description saved on it |
| `<title> [<id>].txt` | its text: the transcript (`Speaker: …` lines) or what was read off its pages |
| `<title> [<id>].pdf` | the PDF made of a document that isn't one (a Word file, an email…) |

Files stored encrypted ([Encryption](encryption.md)) are read decrypted, and a locked vault's files can't be read until
one of its owners unlocks it. Files kept on a storage connection are read from there.

Names are made safe for every file system: `/ \ : * ? " < > |` and control characters become `_`, names are cut to
120 characters, and two names in a folder that differ only in case get the second one's id added (`Talks [12]`).

## Who sees what

Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces
they have a role in, and in a namespace where they only have roles on some collections, those collections and the
topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The tree only reads.

## The API

`GET /api/v1/tree?path=pods/Topics/Gene therapy` lists a folder: `{path, entries: [{name, folder, modified, size,
content_type}]}`. An empty path lists your namespaces. `GET /api/v1/tree/file?path=…` reads a file, with byte ranges
(`Range: bytes=0-1023`) and an `ETag`. Asking for a file as a folder (or the other way round) answers 400.

The MCP server has `list_files` and `read_file` ([MCP server](mcp.md)): `read_file` returns the text of `.txt` and
`.json` files, a page at a time, and the download link of everything else.

## Refine later

- Show supplementary files (captions, translations, attachments) beside each asset.
- Cache an asset's text and metadata so large folders list faster.
- A sidecar for each collection, topic and entity, and a folder of notes.
- Nest entities by how they relate (people under organisations) once the graph records relations between them.
2 changes: 2 additions & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ front of the web app also needs `TRUST_PROXY_HEADERS=true` ([Authentication](aut
| `write_note` | write a note or a thing's page, marked as written by an assistant. Needs a write-scope token and editor access |
| `create_podcast` | make a two-host learning [episode](podcasts.md) about recordings or lines in them, with an angle, a length and a style; it's made in the background. Needs a write-scope token and editor access where it goes |
| `get_podcast` | an episode: how far it has got, its script with each line's speaker and the sources it cites (links), and what the fact-check changed |
| `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files |
| `read_file` | a text file of that tree (an asset's `.txt` text or `.json` metadata), a page at a time; other files' download link |
| `file_storage` | where Lens keeps its own files (notes' images and attachments) and the connections it could use; `check` tests it. Admins only |
| `set_file_storage` | keep new files on this machine or a storage connection (folder, rclone crypt); checked before it's saved. Admins only, with a write-scope token |
| `propose_graph_change` | ask for two entities to be merged or linked; it waits in Proposed changes unless `apply` makes it at once (it can be undone). Needs a write-scope token and editor access |
Expand Down
57 changes: 56 additions & 1 deletion fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import html
import json
import re
import urllib.parse
from collections.abc import Callable
from dataclasses import dataclass, field
from typing import Any, Literal
Expand All @@ -26,7 +27,7 @@
from app.api.v1.routes import recordings as recording_routes
from app.api.v1.routes import search as search_routes
from app.api.v1.routes import topics as topic_routes
from app.domain import analyze, auth, blobs, feeds, graph_history, library, notebook, rdf, render, settings, sources, store
from app.domain import analyze, auth, blobs, feeds, filetree, graph_history, library, notebook, rdf, render, settings, sources, store
from app.domain import entities as ents
from app.domain import speakers as spk
from app.domain import topics as topicmod
Expand Down Expand Up @@ -1436,6 +1437,60 @@ def get_podcast(ctx: Context, episode_id: int) -> dict[str, Any]:
)


# ---------- the graph as files ----------
FILE_CHARS = 100_000 # the most of a file read_file returns at once; offset pages through the rest


def _tree(ctx: Context) -> filetree.Tree:
return filetree.Tree(ctx.db, ctx.cfg, filetree.Viewer(ctx.user.admin, ctx.user.roles, ctx.user.collections))


@tool(
"list_files",
"List a folder of the file tree",
"The graph as folders and files: '' lists your namespaces; each has Collections/<collection>/<inside it>/…, "
"Topics/<topic>/<narrower topic>/… and Entities/<type>/<name>/, each folder holding its assets' files, named "
"'<title> [<id>]': its own file (.mp3, .pdf, …), .json with its metadata, topics and entities, and .txt with its text.",
Arg("path", "string", "the folder, from the top (e.g. pods/Topics/Gene therapy)", default="", max_length=4000),
)
def list_files(ctx: Context, path: str) -> dict[str, Any]:
t = _tree(ctx)
try:
entries = t.list(path)
except filetree.NotFound:
raise KeyError(path) from None
except filetree.NotAFolder:
raise ToolError("that's a file: read it with read_file") from None
return {"path": "/".join(t.split(path)), "entries": [e.out() for e in entries]}


@tool(
"read_file",
"Read a file of the file tree",
"A text file of the tree (an asset's .txt or .json), from list_files. Media and documents' own "
f"files are only their size and type here: download them from the API (/api/v1/tree/file?path=). At most "
f"{FILE_CHARS} characters at a time; next_offset continues.",
Arg("path", "string", "the file, from the top", required=True, max_length=4000),
Arg("offset", "integer", "start at this character", default=0, minimum=0),
)
def read_file(ctx: Context, path: str, offset: int) -> dict[str, Any]:
try:
e = _tree(ctx).stat(path)
except filetree.NotFound:
raise KeyError(path) from None
if e.folder:
raise ToolError("that's a folder: list it with list_files")
out: dict[str, Any] = {"path": path, "size": e.size, "content_type": e.content_type, "modified": e.modified}
if e.data is None:
out["download"] = f"{store.API}/tree/file?path={urllib.parse.quote(path)}"
return out
text = e.data.decode("utf-8", "replace")
out["text"] = text[offset : offset + FILE_CHARS]
if offset + FILE_CHARS < len(text):
out["next_offset"] = offset + FILE_CHARS
return out


# ---------- where files are kept (admins) ----------
def _admin(ctx: Context) -> None:
if not ctx.user.admin:
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
fedora,
fields,
files,
filetree,
graph,
graph_history,
hierarchy,
Expand Down Expand Up @@ -87,6 +88,7 @@
notebook,
comments,
files,
filetree,
fields,
imports,
uploads,
Expand Down
65 changes: 65 additions & 0 deletions fastapi_backend/app/api/v1/routes/filetree.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
"""The graph as folders and files (docs/file-tree.md): browse the tree the S3, WebDAV, SMB and NFS servers serve, and
read its files. Everyone sees what the web app shows them; the tree only reads."""

from __future__ import annotations

from typing import Any

from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import Response, StreamingResponse

from app.api.deps import Cfg, CurrentUser, Db, Principal
from app.api.streaming import LOCKED, byte_range
from app.domain import filetree, keyring
from app.domain.store import DB
from app.schemas.filetree import TreeEntry, TreeFolder

router = APIRouter(prefix="/tree", tags=["tree"])
PATH = Query("", max_length=4000, description="a folder or file, from the top: namespace/Collections|Topics|Entities/…")


def tree_of(db: DB, cfg: dict[str, Any], user: Principal) -> filetree.Tree:
return filetree.Tree(db, cfg, filetree.Viewer(user.admin, user.roles, user.collections))


@router.get("")
def list_folder(user: CurrentUser, db: Db, cfg: Cfg, path: str = PATH) -> TreeFolder:
"""A folder's files and folders: '' lists your namespaces, then their collections, each holding its assets'
files. A file answers 400 (read it at /tree/file)."""
t = tree_of(db, cfg, user)
try:
entries = t.list(path)
except filetree.NotFound:
raise HTTPException(404, "not found") from None
except filetree.NotAFolder:
raise HTTPException(400, "that's a file: read it at /tree/file") from None
return TreeFolder(path="/".join(t.split(path)), entries=[TreeEntry(**e.out()) for e in entries])


@router.get(
"/file",
response_class=Response,
responses={200: {"content": {"application/octet-stream": {}}}, 206: {"description": "a byte range"}},
)
def read_file(request: Request, user: CurrentUser, db: Db, cfg: Cfg, path: str = PATH) -> Response:
"""A file of the tree, with byte ranges."""
t = tree_of(db, cfg, user)
try:
e = t.stat(path)
except filetree.NotFound:
raise HTTPException(404, "not found") from None
if e.folder:
raise HTTPException(400, "that's a folder: list it at /tree")
r = byte_range(e.size, request) if e.size else (0, -1, 200)
if r is None:
return Response(status_code=416, headers={"Content-Range": f"bytes */{e.size}"})
start, end, status = r
try:
if e.encrypted: # a locked vault says so now, not halfway through the body
keyring.plain_size(db, cfg, e.path)
except keyring.Locked:
raise HTTPException(423, LOCKED) from None
headers = {"Accept-Ranges": "bytes", "ETag": f'"{e.version}"', "Content-Length": str(max(0, end - start + 1))}
if status == 206:
headers["Content-Range"] = f"bytes {start}-{end}/{e.size}"
return StreamingResponse(e.read(db, cfg, start, end), status_code=status, media_type=e.content_type, headers=headers)
Loading
Loading