Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,29 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Evidence PDFs hold everything Lens found.** After each resource's own file come pages of what Lens knows about
it: its details, metadata and custom fields, tags, topics, the summary and analysis models wrote, its sections,
speakers, emotion and sounds (overall, by speaker, and marked on each transcript line, such as "(Happy, Laughter)"),
entities, keywords, the faces, objects, descriptions and text on screen seen in it, and the places and dates found
in it. Notes, highlights and comments people shared go in when picked. Export › Choose what goes in it… (and the
button beside a collection's Evidence PDF) picks the parts, and `include` does the same in the API
([Evidence PDFs](docs/api.md#evidence-pdfs)).
- **Roles with permissions.** Viewer, editor and owner are now sets of permissions (read, edit, import, process,
curate, share, publish, remove, members, manage), and admins make custom roles from them in Admin → Roles, given per
namespace like the built-in ones. Existing access is unchanged. Someone who manages members but isn't an admin gives
only roles they hold all of ([Access](docs/access.md#roles-and-permissions)).

- **Invite and remove people.** Making an account in Admin → People now emails the sign-in link as an invitation when
mail is set up, and the link is still shown to share yourself. **Remove for good** ends every way someone signs in,
takes their roles and their private chats, notes and views, and keeps what they shared under their name; their email
can be invited again ([Authentication](docs/authentication.md#removing-someone)). The assistant gets `list_people`,
`invite_person` and `change_person`, which always wait for an admin's approval.

- **NSFW classification, with a model on your server.** Turn it on in Settings → NSFW for every namespace, or on a
namespace's page for that one (off by default). The new `nsfw` step scores a video's sampled frames, a document's
pages or an image with an Apache-2.0 model (fetched on first use, about 105 MB with ONNX Runtime) and flags the
resource when a picture scores at least the threshold. Flagged resources are blurred until someone chooses to see
them, left out of the Library, or shown with a label; editors can mark one NSFW or not by hand, and **Classify
existing** classifies what was there before. Agents use it through the API and MCP (`nsfw_settings`, `set_nsfw`,
`mark_nsfw`) ([NSFW classification](docs/nsfw.md)).

- **Apache Tika.** Lens can use an Apache Tika server (`--profile tika` in the compose files, or one of your own, set
in Settings → Documents). Documents nothing here can make into a PDF, such as Word files on a lean server or a
Raspberry Pi, are read by Tika as text, and every document's own title, authors, language and dates are read as
Expand Down
29 changes: 28 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -565,11 +565,38 @@ Manifest with a Canvas per page ([IIIF](iiif.md)), and its public page shows its
collection you can read (a list in its own order, a filter's oldest first; 50 at most), as one PDF to hand on, made by
[anytopdf](configuration.md#anytopdf). Each resource goes in as its own file: a document or an image as it is, a
captured web page as captured, audio or video with Lens's transcript of it (its speakers' names on each line), and a
transcript without media as its text. The PDF is searchable PDF/A-3 with its text and chunks embedded, and its last
transcript without media as its text. After it come the pages of what Lens found about it, a text file of its own
named `<title> - what Lens found.txt`. The PDF is searchable PDF/A-3 with its text and chunks embedded, and its last
page is a provenance page giving every file's SHA-256 and size, so whoever gets it can check each part against the
file it came from. It holds no paths of the server (anytopdf's `share` profile), is made when asked for, and nothing
of it is kept. Anyone who can read the resources can have it (share links and signed links can't).

`include` picks its parts (repeat it for several, `?include=file&include=summary`); without it, every part but
`notes` goes in. In the web app, a resource's Export menu has Evidence PDF (the default parts) and Choose what goes in
it…, and a collection's page has the same beside its Evidence PDF button; the picks are remembered in the browser.

| Part | What goes in |
| --- | --- |
| `file` | its own file, as above |
| `details` | what it is, when it was recorded and added, its length or pages, size, language, words, content type, where it came from, its status and NSFW classification (where that is on); an email's subject, sender, recipients and date; a web page's address and when it was captured; what a document says about itself (Tika) and how its PDF was made |
| `metadata` | its description as saved over what it says about itself: title, description, date, language, creators, contributors, subjects, identifiers, rights, attribution, provider, related links, access, Dublin Core terms and other statements, and its custom fields |
| `tags` | its tags |
| `topics` | the topics it's about: accepted ones, then suggested ones, each saying how it came |
| `summary` | the summary a model wrote, its key points and follow-ups (each with where it comes from), the people and topics it names, its overall tone and importance |
| `analysis` | what its pipeline's model steps saved, by key, with the model |
| `sections` | its sections, with where each starts |
| `speakers` | who spoke, for how long and what share of the speech, their turns, words and pace, and how each was recognised |
| `emotion` | the share of the speech time of each emotion, overall and by speaker, and the sounds heard; each transcript line is marked with its emotion and sound, such as "(Happy, Laughter)" (neutral speech isn't marked) |
| `entities` | the names it mentions by type (people, organisations, places, dates …), each with how often |
| `keywords` | its keywords |
| `visual` | its shots, the faces found (and who, when recognised), the objects seen, what a model said each page or shot shows, and the text on screen, with where each is |
| `places` | the places and dates anytopdf found in it and where a photo was taken (members of its namespace only, as on its page) |
| `notes` | the highlights, shared notes and comments on it, with who wrote them (names only); never anyone's private notes |

What a model wrote (the summary, analysis and descriptions) says so on its page. A list longer than 1,000 lines (text
on screen in a long video, say) shows the first 1,000 and says how many more there are. Making these pages calls no
model: they're read from what Lens already stored.

anytopdf runs as it does for conversions (no network, plugins or config file of its own); if it isn't here it's
fetched on first use, unless Settings › Documents says never to use it (503). A conversion node takes one file per
job and its own options: a single document goes as it is, and more go as one zip, with audio and video as their
Expand Down
17 changes: 13 additions & 4 deletions fastapi_backend/app/api/evidence.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,28 @@
import urllib.error
from typing import Any

from fastapi import HTTPException
from fastapi import HTTPException, Query
from fastapi.responses import Response

from app.api.streaming import LOCKED
from app.domain import anytopdf, evidence, keyring
from app.domain import anytopdf, evidence, evidence_report, keyring
from app.domain.store import DB
from app.schemas.recordings import EvidencePart

PDF: dict[int | str, dict[str, Any]] = {200: {"content": {"application/pdf": {}}}}
INCLUDE = Query(
None,
description="the parts of the PDF; repeat for several. Without it: every part but notes. "
+ "; ".join(f"{k}: {v}" for k, v in evidence_report.PARTS.items()),
)


def download(db: DB, cfg: dict[str, Any], rids: list[int], title: str) -> Response:
def download(
db: DB, cfg: dict[str, Any], rids: list[int], title: str, include: list[EvidencePart] | None = None, places: set[int] | None = None
) -> Response:
"""The evidence PDF of these resources as a download. `places`: the ones whose places the reader may see."""
try:
data = evidence.make(db, cfg, rids)
data = evidence.make(db, cfg, rids, include, places)
except keyring.Locked:
raise HTTPException(423, LOCKED) from None
except FileNotFoundError as e:
Expand Down
18 changes: 15 additions & 3 deletions fastapi_backend/app/api/v1/routes/collections.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from app.domain.store import DB
from app.schemas.collections import Collection, CollectionCreate, CollectionDetail, CollectionUpdate
from app.schemas.common import Created, Ok
from app.schemas.recordings import EvidencePart

router = APIRouter(prefix="/collections", tags=["collections"])

Expand Down Expand Up @@ -65,14 +66,25 @@ def get_collection(cid: int, user: CurrentUser, acl: Acl, db: Db) -> CollectionD


@router.get("/{cid}/evidence.pdf", response_class=Response, responses=evidence_api.PDF)
def collection_evidence(cid: int, user: CurrentUser, acl: Acl, db: Db, cfg: Cfg) -> Response:
def collection_evidence(
cid: int, user: CurrentUser, acl: Acl, db: Db, cfg: Cfg, include: list[EvidencePart] | None = evidence_api.INCLUDE
) -> Response:
"""The collection's recordings you can read (a list in its order, a filter's oldest first) as one evidence PDF made by anytopdf (a download): each
one's pages, then a provenance page with every file's SHA-256 and size. At most 50 recordings."""
one's pages and what Lens found about it, then a provenance page with every file's SHA-256 and size. `include` picks the parts, as for a
resource's evidence PDF. At most 50 recordings."""
col = own_collection(db, cid, user)
ids = recsets.members(db, col, set(acl.roles))
if len(ids) > evidence.MAX:
raise HTTPException(400, f"an evidence PDF holds {evidence.MAX} recordings at most; this collection has {len(ids)}")
return evidence_api.download(db, cfg, ids if col.get("recordings") is not None else ids[::-1], col["name"])
recs = (
db.rows(
"SELECT record::id(id) AS id, space, collection FROM recording WHERE id IN $ids", ids=[store.R("recording", i) for i in ids]
)
if ids
else []
)
places = {r["id"] for r in recs if acl.member(r)}
return evidence_api.download(db, cfg, ids if col.get("recordings") is not None else ids[::-1], col["name"], include, places)


@router.patch("/{cid}")
Expand Down
13 changes: 9 additions & 4 deletions fastapi_backend/app/api/v1/routes/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from app.schemas.recordings import (
AccessRequest,
EmbedLink,
EvidencePart,
JobQueued,
LanguageCount,
MediaKind,
Expand Down Expand Up @@ -691,12 +692,16 @@ def export_recording(rid: int, fmt: str, acl: Acl, db: Db) -> Response:


@router.get("/{rid}/evidence.pdf", response_class=Response, responses=evidence_api.PDF)
def evidence_pdf(rid: int, acl: Acl, db: Db, cfg: Cfg) -> Response:
def evidence_pdf(rid: int, acl: Acl, db: Db, cfg: Cfg, include: list[EvidencePart] | None = evidence_api.INCLUDE) -> Response:
"""The resource as an evidence PDF, made by anytopdf (a download): its own file's pages (audio and video with
their transcript), searchable, then a provenance page with the file's SHA-256 and size. 503 when anytopdf can't
run here (it's fetched on first use unless Settings › Documents says never to use it)."""
their transcript, each line marked with its emotion and sound), searchable, then the pages of what Lens found
about it (details, metadata, tags, topics, summary, analysis, sections, speakers, emotion, entities, keywords,
what is seen, places and dates; notes when asked for), then a provenance page with each file's SHA-256 and size.
`include` picks the parts. 503 when anytopdf can't run here (it's fetched on first use unless Settings ›
Documents says never to use it)."""
rec = acl.recording(rid)
return evidence_api.download(db, cfg, [rid], rec.get("title") or f"recording {rid}")
places = {rid} if acl.member(rec) else set()
return evidence_api.download(db, cfg, [rid], rec.get("title") or f"recording {rid}", include, places)


@router.patch("/{rid}/segments/{idx}")
Expand Down
46 changes: 34 additions & 12 deletions fastapi_backend/app/domain/evidence.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,12 @@
SHA-256 and size, so whoever gets it can check each page against the file it came from. Its text is searchable and its
chunks are embedded (PDF/A-3), so what an agent cites from it stays cited.

A resource without a file of its own (an imported transcript) goes in as its text. Nothing is kept: the PDF is made
when asked for, from plain copies in a folder of its own that is removed straight after.
A resource without a file of its own (an imported transcript) goes in as its text. After each resource's file come
the pages of what Lens found about it (evidence_report.py): its details, metadata, tags, topics, summary, analysis,
sections, speakers, emotion and sounds, entities, keywords, what is seen in it and the places and dates in it, each a
part that can be left out (`parts`); the notes, highlights and comments people shared on it go in when asked for.
Nothing is kept: the PDF is made when asked for, from plain copies in a folder of its own that is removed straight
after.
"""

from __future__ import annotations
Expand All @@ -15,7 +19,7 @@
import shutil
import tempfile

from . import anytopdf, ingest, keyring, render, store
from . import anytopdf, evidence_report, ingest, keyring, render, store

R = store.R
MAX = 50 # recordings in one evidence PDF, at most
Expand Down Expand Up @@ -43,12 +47,27 @@ def ready(cfg, say=None):
anytopdf.fetch(cfg, say)


def _gather(db, cfg, rid, i, into):
"""Put one recording's file (and its transcript, for audio or video) into the folder: (input, transcript or None)."""
def _gather(db, cfg, rid, i, into, parts, places):
"""Put one recording's file (and its transcript, for audio or video) and what Lens found about it into the folder:
([its files, in order], transcript or None)."""
rec = db.one("SELECT space, path, remote, source, title, media FROM $r", r=R("recording", rid))
if not rec:
raise KeyError(rid)
kind, name = render.kind(rec), _name(i, rec.get("title"))
d = render.player_data(db, rid)
out, said = [], None
if "file" in parts:
out, said = _own(db, cfg, rec, rid, kind, name, into, evidence_report.marked(d) if "emotion" in parts else d)
if evidence_report.has_findings(parts):
found = into / f"{name} - what Lens found.txt"
found.write_text(evidence_report.write(db, cfg, rid, parts, d, places), encoding="utf-8")
out.append(found)
return out, said


def _own(db, cfg, rec, rid, kind, name, into, d):
"""The recording's own file copied into the folder, or its text when it has none: ([that file], its transcript or
None)."""
src = None
if kind != "transcript" and (rec.get("path") or rec.get("remote")):
try:
Expand All @@ -59,28 +78,31 @@ def _gather(db, cfg, rid, i, into):
if kind in ("document", "image"):
raise FileNotFoundError(f"the file of “{rec.get('title') or rid}” is missing on the server")
src = None
d = render.player_data(db, rid)
if src is None: # its text is all there is
out = into / f"{name}.txt"
out.write_text(render.export_text(d, "txt") or "(no text)\n", encoding="utf-8")
return out, None
return [out], None
out = into / f"{name}{pathlib.PurePosixPath(str((rec.get('remote') or {}).get('path') or src)).suffix.lower()}"
shutil.copyfile(src, out)
said = None
if kind in ("audio", "video") and d["segments"]:
said = into / "transcripts" / f"{name}.srt"
said.parent.mkdir(exist_ok=True)
said.write_text(render.export_text(d, "srt"), encoding="utf-8")
return out, said
return [out], said


def make(db, cfg, rids):
"""The evidence PDF of these recordings, in this order, as bytes."""
def make(db, cfg, rids, include=None, places=None):
"""The evidence PDF of these recordings, in this order, as bytes. `include`: its parts (evidence_report.PARTS;
None: everything but people's notes). `places`: the recordings whose places may go in (None: all of them)."""
rids = list(dict.fromkeys(int(r) for r in rids))
if not rids:
raise ValueError("there's nothing to put in it")
if len(rids) > MAX:
raise ValueError(f"an evidence PDF holds {MAX} recordings at most; this has {len(rids)}")
parts = evidence_report.parts_of(include)
if not parts:
raise ValueError("there's nothing to put in it: include its file or something Lens found")
ready(cfg)
work = pathlib.Path(cfg["data_dir"]) / "tmp"
work.mkdir(parents=True, exist_ok=True)
Expand All @@ -89,8 +111,8 @@ def make(db, cfg, rids):
into.mkdir(mode=0o700)
inputs, transcripts = [], []
for i, rid in enumerate(rids, 1):
f, t = _gather(db, cfg, rid, i, into)
inputs.append(f)
files, t = _gather(db, cfg, rid, i, into, parts, places is None or rid in places)
inputs += files
transcripts += [t] if t else []
out = pathlib.Path(tmp) / "evidence.pdf"
anytopdf.evidence(cfg, inputs, transcripts, out)
Expand Down
Loading
Loading