From f00cf771dc33c3035683329de82b7f8a7ce7524c Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Thu, 15 Oct 2026 11:16:58 +0000 Subject: [PATCH 1/2] feat(find): find resources by face, voice, entity, topic, type, content type and date One filter for the API (POST /find), the MCP server (find) and the assistant (find_resources): who is seen in a resource (faces) and heard (speakers), the entities it mentions, the topics it is about (narrower topics included), its kind and content type, when it was made (a day, a month or a year), tags, language, objects seen, emotions heard, length, status, collection, origin and words said in it. Faces, speakers, entities and topics take ids or names; `resolved` says what each name matched, and a name that matches nothing is an error naming the closest. Each result says why it matched, with up to three moments per reason to cite. The Library list (GET /resources) takes face, entity, topic, content_type and emotion ids as filters too. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N --- CHANGELOG.md | 7 + docs/api.md | 20 + docs/assistant.md | 6 +- docs/mcp.md | 1 + docs/processing.md | 32 + fastapi_backend/app/api/mcp_tools.py | 102 ++ fastapi_backend/app/api/v1/router.py | 2 + fastapi_backend/app/api/v1/routes/find.py | 29 + .../app/api/v1/routes/recordings.py | 14 + fastapi_backend/app/domain/ai_tools.py | 66 + fastapi_backend/app/domain/content_types.py | 20 + fastapi_backend/app/domain/find.py | 428 +++++++ fastapi_backend/app/domain/library.py | 63 +- fastapi_backend/app/schemas/find.py | 102 ++ fastapi_backend/tests/api/test_find.py | 222 ++++ nextjs-frontend/app/openapi-client/index.ts | 12 + nextjs-frontend/app/openapi-client/sdk.gen.ts | 26 + .../app/openapi-client/types.gen.ts | 509 ++++++++ nextjs-frontend/openapi.json | 1115 +++++++++++++++++ 19 files changed, 2761 insertions(+), 15 deletions(-) create mode 100644 fastapi_backend/app/api/v1/routes/find.py create mode 100644 fastapi_backend/app/domain/find.py create mode 100644 fastapi_backend/app/schemas/find.py create mode 100644 fastapi_backend/tests/api/test_find.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d7513f7c..5185932a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased +- **Find anything by who, what and when.** Agents and the API can find resources by the faces seen in them, the + voices heard, the entities they mention, the topics they're about, their kind and content type, when they were made + (a day, a month or a year), tags, language, objects seen, emotions heard, length and words said, together or alone, + naming people and things instead of looking up ids. Each result says why it matched, with moments to cite + (`POST /find`, the MCP tool `find`, the assistant's `find_resources`; [Finding by any + attribute](docs/processing.md#finding-by-any-attribute)). The Library list takes faces, entities, topics, content + types and emotions as filters too. - **The assistant always answers.** When a question used up its tool steps (`ai.max_steps`), the assistant used to stop with "I ran out of steps before finishing". Now it's asked to answer from what it found, with the tools off, and says what it didn't get to check; a reply with nothing in it is asked again the same way. It also knows its step budget up diff --git a/docs/api.md b/docs/api.md index eab222305..70d726eb9 100644 --- a/docs/api.md +++ b/docs/api.md @@ -417,6 +417,11 @@ repeat a parameter that takes several values (`?status=new&status=error`) to mat | `origin` | where they came from: `source:` (a connected source), `upload`, `paste`, `iiif`, `folder` (the archive's own folders) or `file` (another file imported by path) | | `language` | language codes, ignoring case; `none` for recordings whose language isn't known | | `object` | kinds of object seen in them (`person`, `car` …; [Objects](#objects)), ignoring case | +| `face` | face identity ids: the recordings they're seen in ([Video](video.md)) | +| `entity` | entity ids: the recordings that mention them | +| `topic` | topic ids: the recordings accepted as about them (not their narrower topics; `POST /find` brings those) | +| `content_type` | content type keys (`podcast`, `scan`, `email` …; [content types](#content-types-and-custom-nodes)) | +| `emotion` | emotions heard in them, ignoring case | | `edited_by` | `me`: recordings you edited (corrected a line of the transcript, changed the catalogue record, or renamed) | | `sensitive` | where [NSFW classification](nsfw.md) is on: `only` the ones flagged NSFW, `exclude` them, or `include` them. Without it, flagged ones are left out where their namespace hides them (`display: hide`) | | `sort` | `date`, `title`, `duration`, `speakers`, `status` or `importance`; `-` in front for descending (default `-date`). Recordings without the value come last either way | @@ -909,6 +914,7 @@ recording later leaves its uploaded file in place, like any other media file. ``` GET /api/v1/search GET /api/v1/search/terms +POST /api/v1/find GET /api/v1/similar GET /api/v1/graph GET /api/v1/mentions @@ -949,6 +955,20 @@ key words marked), the key `words` it says, and the `topics` and `entities` in c when an embedding model compared them too, else `words`; `terms` are the key words looked for and `meaning` says why it wasn't by meaning when search by meaning is set up. A resource is never listed as like itself. +`POST /find` finds the resources you can read by any attribute ([finding by any attribute](processing.md#finding-by-any-attribute)). +The body takes `faces`, `speakers`, `entities` and `topics` (ids or names), `types`, `content_types` (keys or labels), +`date_from` and `date_to` (`2026-09-30`, `2026-09` or `2026`), `min_duration` and `max_duration` (seconds), `tags`, +`languages`, `objects`, `emotions`, `status`, `origins`, `collection`, `sensitive`, `title` (words in the title, the +namespace or a speaker's name), `text` (words said or written, as `GET /search` takes them) and `namespace`; `sort` +(`relevance`, the default with `text`, else `-date`; or `date`, `title`, `duration`, `-` for descending), `limit` +(default 20, at most 100) and `offset`. Filters combine with AND, the values of one with OR. The reply has `total`, +the page's `items` (rows as `GET /resources` lists them, each with `why`: `[{by, name, moments, count}]`, `by` being +`words`, `face`, `voice`, `entity`, `topic`, `content_type`, `object` or `emotion`, and each moment a `t0` and `t1` in +ms or a `page` from 0, with its `text` when it has words), `resolved` (what each name matched: `[{id, name, namespace}]` +per filter, and `narrower_topics`) and `text` (how the words were matched, as `GET /search` says, and whether the +moments looked through were `capped`). A name that matches nothing in the namespaces you can read is a 400 that names +the closest ones; an unknown field is a 422. + Search has no prefix search (`interp*` looks for the word "interp"). `GET /search/terms?prefix=interp` lists whole words said in the namespaces you can read (`ns` for one) that start with it, the most said first, with how often and in how many recordings (`limit`, default 8, at most 20); the web app offers them as "Try …". diff --git a/docs/assistant.md b/docs/assistant.md index b99b1d390..07a5fc24d 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -92,8 +92,10 @@ engine), `POST /voice/transcribe` (a clip as the raw body; nothing is kept), `PO When the configured model supports function calling, chat becomes an agent. - **What it can look things up with:** transcripts (including on-screen text), recordings, transcript passages, - outputs, entities and their mentions and timelines, the knowledge graph, speakers, and content similar to a text, - a recording, a topic or an entity (`find_similar`). + outputs, entities and their mentions and timelines, the knowledge graph, speakers, content similar to a text, + a recording, a topic or an entity (`find_similar`), and resources by any attribute: faces seen, speakers heard, + entities, topics, kind, content type, dates and the rest, each with the moments that show why it matched + (`find_resources`, [Finding by any attribute](processing.md#finding-by-any-attribute)). - **Limits:** everything is limited to what the person can read and to the conversation's scope, and capped by `ai.max_steps` (Settings → AI assistant, Tool steps per answer; 6 by default) and `ai.max_transcript_reads`. - **Always an answer:** the model is told its step budget. When the steps run out, or it replies with nothing, it is diff --git a/docs/mcp.md b/docs/mcp.md index 9b8e56ab7..80c5de52d 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -44,6 +44,7 @@ front of the web app also needs `TRUST_PROXY_HEADERS=true` ([Authentication](aut |---|---| | `search` | moments where words are said, shown on screen or written, best first: the recording, speaker, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, speaker or recording | | `find_similar` | the recordings, documents and images most like a text, a recording, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links | +| `find` | resources by any attribute: faces seen, speakers heard, entities, topics, kind, content type, dates (a day, month or year), tags, language, objects, emotions, length, collection and words said; ids or names. Each says why it matched, with moments and links to cite ([Finding by any attribute](processing.md#finding-by-any-attribute)) | | `list_namespaces` | the namespaces you can read, with their recordings, hours, speakers and your role | | `list_recordings` | recordings, newest first, by namespace, title words, dates, speaker, tag or kind, and whether they're flagged NSFW (`sensitive`) | | `list_speakers` | the speakers of your namespaces, the ones who talk most first | diff --git a/docs/processing.md b/docs/processing.md index adbfa7260..b0c1572d1 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -256,6 +256,38 @@ In the web app: * a topic's and an entity's drawer, and a resource, topic or entity selected on the graph canvas, list similar content too. +### Finding by any attribute + +The API (`POST /find`), the assistant (`find_resources`) and the MCP server (`find`) find resources by what is known +about them, one filter for all three: + +* who is **seen** in them (`faces`: face identities, [Video](video.md)) and **heard** (`speakers`: voices); +* the **entities** they mention and the **topics** they were accepted as about (a topic brings its narrower topics); +* their **kind** (`types`: audio, video, transcript, document, image) and **content type** (`content_types`: podcast, + interview, scan, photo, email … [Content types](#content-types)); +* **when** they were made (`date_from`, `date_to`: a day, a month or a year, so `2025` is the whole year) and how + long they are; +* their tags, language, the kinds of object seen in them, the emotions heard, their status, collection and where they + came from; +* and words said or written in them (`text`, matched as a search is, by meaning too when that is set up). + +Filters combine with AND, the values of one filter with OR. Faces, speakers, entities and topics take an id or a name: +a name matches every face, speaker, entity (or one of the other ways it's said) or topic label called that in the +namespaces you can read, ignoring case, and `resolved` says what it matched. A name that matches nothing is an error +that lists the closest names, so an agent can correct itself. + +Each resource found says **why** it matched, with up to three moments per reason to cite: where the words are, when the +face is on screen (or the pages it's on), the speaker's first lines, the lines that mention the entity, and the topics, +content type, kinds of object and emotions that matched. With `text`, the resource whose moment matches best comes +first; otherwise the newest. Nothing here calls a model, apart from `text` searching by meaning when it is set up +(one embedding request, logged in the activity ledger). + +The Library list (`GET /resources`) takes the same attributes as ids: `face`, `entity`, `topic` (as it is, without its +narrower topics), `content_type` and `emotion`, besides its other filters ([API](api.md#resources)). + +Planned, not built yet: these filters in the Library's filter bar and in saved collections, finding by what a photo +shows (`GET /search/photos`) inside the same filter, and ranking by more than the best matching moment. + ## Background work Imports, pipeline runs and folder scans return at once and run as jobs stored in SurrealDB. The server runs diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 3b1386cf1..3532c9340 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -53,6 +53,7 @@ updates, ) from app.domain import entities as ents +from app.domain import find as findmod from app.domain import speakers as spk from app.domain import topics as topicmod from app.domain.store import DB @@ -602,6 +603,107 @@ def list_recordings( return out +FIND_SORTS = ("relevance", "-date", "date", "-title", "title", "-duration", "duration") + + +@tool( + "find", + "Find resources by any attribute", + "Recordings, videos, documents and images you can read by who is seen in them (faces) and heard (speakers), the " + "entities they mention, the topics they're about, their kind and content type, when they were made, tags, language, " + "objects seen, emotions heard, length, collection, and words said or written in them. Filters combine with AND, the " + "values of one with OR. Faces, speakers, entities and topics are ids or names (ignoring case); `resolved` says what " + "each name matched, and one that matches nothing comes back as an error naming the closest. Each result says `why` " + "it matched, with up to three moments per reason (a time or a page, the words there, and a url that opens it), to cite.", + Arg("namespace", "string", "only this namespace"), + Arg("text", "string", 'words said or written in them (words, "phrases", OR); by meaning too when that is set up', max_length=300), + Arg("title", "string", "words that must all appear in the title, the namespace or a speaker's name", max_length=200), + Arg("faces", "string[]", "people seen in them: face ids or names"), + Arg("speakers", "string[]", "voices heard in them: speaker ids or names"), + Arg("entities", "string[]", "entities they mention (people, organisations, places …): ids or names"), + Arg("topics", "string[]", "topics they're about: ids or labels; narrower topics count too"), + Arg("types", "string[]", "their kind", enum=MEDIA), + Arg("content_types", "string[]", "content types by key or label: podcast, interview, meeting_audio, scan, photo, email, document …"), + Arg("date_from", "string", "made on or after this day, month or year (2026-09-30, 2026-09, 2026)"), + Arg("date_to", "string", "made on or before this day, month or year"), + Arg("min_duration", "number", "at least this many seconds long", minimum=0), + Arg("max_duration", "number", "shorter than this many seconds", minimum=0), + Arg("tags", "string[]", "tags (ignoring case)"), + Arg("languages", "string[]", "language codes (en, de …); none for those whose language isn't known"), + Arg("objects", "string[]", "kinds of object seen in them (person, car, dog …)"), + Arg("emotions", "string[]", "emotions heard in them (happy, sad, angry …)"), + Arg("collection_id", "integer", "only what's in this collection (and the collections inside it)"), + Arg( + "sensitive", + "string", + "where NSFW classification is on: only those flagged NSFW, exclude them, or include them", + enum=("include", "only", "exclude"), + ), + Arg("sort", "string", "relevance (the default with text), else date, title or duration; - first for descending", enum=FIND_SORTS), + Arg("limit", "integer", "how many", default=20, minimum=1, maximum=100), + Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000), +) +def find( + ctx: Context, + namespace: str | None, + limit: int, + offset: int, + sort: str | None, + collection_id: int | None, + **spec: Any, +) -> dict[str, Any]: + spaces, within = ctx.acl.scope(namespace) + if collection_id is not None: + spec["collection"] = collection_id + res = findmod.find(ctx.db, ctx.cfg, spaces, within, spec, sort=sort, limit=limit, offset=offset) + items = [] + for r in res["items"]: + rid = r["id"] + why = [] + for w in r["why"]: + moments = [ + store.clean( + { + **({"page": m["page"] + 1} if m.get("page") is not None else {}), + **({"to_page": m["last_page"] + 1} if m.get("last_page") is not None else {}), + **({"time": store.tc(m["t0"])} if m.get("t0") is not None else {}), + **({"until": store.tc(m["t1"])} if m.get("t1") is not None and w["by"] == "face" else {}), + "speaker": m.get("speaker"), + "text": _cut(m["text"], 400) if m.get("text") else None, + "url": ctx.link(rid, m.get("t0")), + } + ) + for m in w.get("moments") or [] + ] + why.append(store.clean({"by": w["by"], "name": w.get("name"), "count": w.get("count"), "moments": moments or None})) + items.append( + store.clean( + { + "recording_id": rid, + "title": r.get("title"), + "namespace": r.get("namespace"), + "recorded_at": r.get("recorded_at"), + "duration": store.tc(r.get("duration_ms")) if r.get("duration_ms") else None, + "media": r.get("media_kind"), + "speakers": r.get("speakers") or None, + "tags": r.get("tags") or None, + "language": r.get("language"), + "nsfw": True if (r.get("nsfw") or {}).get("flagged") else None, + "url": ctx.link(rid), + "why": why, + } + ) + ) + out: dict[str, Any] = {"total": res["total"], "results": items} + if res["resolved"]: + out["resolved"] = res["resolved"] + if res["text"]: + out["text"] = res["text"] + if offset + len(items) < res["total"]: + out["next_offset"] = offset + len(items) + return out + + @tool( "list_speakers", "List speakers", diff --git a/fastapi_backend/app/api/v1/router.py b/fastapi_backend/app/api/v1/router.py index 311028d4a..9d82abf09 100644 --- a/fastapi_backend/app/api/v1/router.py +++ b/fastapi_backend/app/api/v1/router.py @@ -24,6 +24,7 @@ fields, files, filetree, + find, graph, graph_history, hierarchy, @@ -95,6 +96,7 @@ imports, uploads, search, + find, searches, speakers, entities, diff --git a/fastapi_backend/app/api/v1/routes/find.py b/fastapi_backend/app/api/v1/routes/find.py new file mode 100644 index 000000000..f3918fa75 --- /dev/null +++ b/fastapi_backend/app/api/v1/routes/find.py @@ -0,0 +1,29 @@ +"""Find resources by any attribute: faces, voices, entities, topics, kind, content type, dates and the rest (domain/find.py).""" + +from __future__ import annotations + +from fastapi import APIRouter + +from app.api.deps import Acl, Cfg, CurrentUser, Db, domain_errors +from app.api.media import sign_urls +from app.domain import find as findmod +from app.schemas.find import FindQuery, FindResults + +router = APIRouter(tags=["search"]) + + +@router.post("/find") +def find_resources(body: FindQuery, acl: Acl, user: CurrentUser, db: Db, cfg: Cfg) -> FindResults: + """Resources you can read that match every filter given: the people seen in them (faces) and heard (speakers), the + entities they mention, the topics they're about, their kind and content type, when they were made, their tags, + language, objects seen, emotions heard, length, status, collection and origin, and words said or written in them. + Faces, speakers, entities and topics can be named instead of numbered; `resolved` says what each name matched, and + a name that matches nothing is a 400 naming the closest ones. Each resource says `why` it matched, with up to three + moments per reason to cite (a time, or a page). With `text`, the best matching come first.""" + spaces, within = acl.scope(body.namespace) + spec = body.model_dump(exclude={"namespace", "sort", "limit", "offset"}, exclude_none=True) + with domain_errors(): + res = findmod.find(db, cfg, spaces, within, spec, sort=body.sort, limit=body.limit, offset=body.offset) + for x in res["items"]: + x["role"] = acl.role_in(x["space"], x.get("collection")) + return FindResults.model_validate(sign_urls(res, full=True)) diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py index 9c9c0dbbc..04ce3b107 100644 --- a/fastapi_backend/app/api/v1/routes/recordings.py +++ b/fastapi_backend/app/api/v1/routes/recordings.py @@ -146,6 +146,15 @@ def list_recordings( description="where NSFW classification is on (docs/nsfw.md): only those flagged NSFW, exclude them, or include them. " "Without it, flagged ones are left out where their namespace hides them (display hide)", ), + face: list[int] | None = Query(None, description="face identity ids: the recordings they're seen in; repeat for several"), + entity: list[int] | None = Query(None, description="entity ids: the recordings mentioning them; repeat for several"), + topic: list[int] | None = Query( + None, description="topic ids: the recordings accepted as about them (not their narrower topics); repeat for several" + ), + content_type: list[str] | None = Query( + None, description="content type keys (podcast, scan, email …; GET /content-types): their recordings; repeat for several" + ), + emotion: list[str] | None = Query(None, description="emotions heard in them (ignoring case); repeat for several"), collection: int | None = Query(None, description="a collection: the recordings in it and in the collections inside it"), field: int | None = Query(None, description="a custom field (of resources): the recordings with a value for it, or with `value`"), value: str | None = Query( @@ -206,6 +215,11 @@ def list_recordings( field=by_field, objects=object, sensitive=sensitive, + faces=face, + entities=entity, + topics=topic, + content_types=content_type, + emotions=emotion, ) response.headers["X-Total-Count"] = str(total) for r in rows: diff --git a/fastapi_backend/app/domain/ai_tools.py b/fastapi_backend/app/domain/ai_tools.py index 041c1dff8..041177286 100644 --- a/fastapi_backend/app/domain/ai_tools.py +++ b/fastapi_backend/app/domain/ai_tools.py @@ -17,6 +17,7 @@ entities, extensions, entity_map, + find as findmod, graph_ask, graph_model, entity_setup, @@ -41,6 +42,7 @@ log = logging.getLogger("lens") _I = {"type": "integer"} _S = {"type": "string"} +_LIST = {"type": "array", "items": {"type": ["string", "integer"]}} TOOLS = [ ( "search_transcripts", @@ -65,6 +67,35 @@ [], False, ), + ( + "find_resources", + "Find recordings, videos, documents and images in scope by any attribute: who is seen in them (faces) or heard " + "(speakers), entities they mention, topics they're about, their type (audio, video, transcript, document, image), " + "content type (podcast, interview, scan, photo, email …), dates (a day, month or year), tags, language, objects " + "seen, emotions heard, length, and words said in them (text). Faces, speakers, entities and topics take ids or " + "names. Each result says why it matched, with numbered moments to cite.", + { + "text": _S, + "faces": _LIST, + "speakers": _LIST, + "entities": _LIST, + "topics": _LIST, + "types": _LIST, + "content_types": _LIST, + "date_from": _S, + "date_to": _S, + "min_duration": {"type": "number"}, + "max_duration": {"type": "number"}, + "tags": _LIST, + "languages": _LIST, + "objects": _LIST, + "emotions": _LIST, + "namespace": _S, + "limit": _I, + }, + [], + False, + ), ( "read_transcript", "Read part of one recording's transcript, by seconds.", @@ -870,6 +901,41 @@ def t_list_recordings(self, namespace=None, limit=20, speaker_id=None, entity_id ] return {"total": len(ids), "recordings": out}, f"Listed {len(ids)} recording(s)" + def t_find_resources(self, namespace=None, limit=10, **spec): + spaces = sorted(self.readable) + if namespace: + spaces = [s for s in spaces if store.space_names(self.db).get(s) == namespace] + if not spaces: + raise ValueError(f"namespace {namespace} isn't available in this conversation") + res = findmod.find(self.db, self.cfg, spaces, None, spec, limit=min(int(limit or 10), 25), ids=self.allowed) + out = [] + for x in res["items"]: + why = [] + for w in x["why"]: + moments = [] + for m in w.get("moments") or []: + text = m.get("text") or f"{w['name']} ({w['by']})" + n = self.ref( + x["id"], m.get("t0") or 0, text, m.get("speaker"), x.get("title"), m.get("source") or "said", m.get("page") + ) + where = {"page": m["page"] + 1} if m.get("page") is not None else {"time": store.tc(m.get("t0"))} + moments.append(store.clean({"ref": n, **where, "text": m.get("text")})) + why.append(store.clean({"by": w["by"], "name": w.get("name"), "count": w.get("count"), "moments": moments or None})) + out.append( + store.clean( + { + "recording_id": x["id"], + "title": x.get("title"), + "date": (x.get("recorded_at") or "")[:10] or None, + "type": x.get("media_kind"), + "namespace": x.get("namespace"), + "speakers": x.get("speakers") or None, + "why": why, + } + ) + ) + return {"total": res["total"], "resolved": res["resolved"] or None, "results": out}, f"Found {res['total']} resource(s)" + def t_read_transcript(self, recording_id, start_seconds=0, end_seconds=None, max_lines=60): cap = self.cfg["ai"].get("max_transcript_reads") or 20 if self.reads >= cap: diff --git a/fastapi_backend/app/domain/content_types.py b/fastapi_backend/app/domain/content_types.py index 43850ac13..a9ad637fd 100644 --- a/fastapi_backend/app/domain/content_types.py +++ b/fastapi_backend/app/domain/content_types.py @@ -285,6 +285,26 @@ def of_recording(db, rid): return recognise(db, rec, types), False +TYPE_FIELDS = "record::id(id) AS id, source, media, path, title, duration_ms, form, content_type" + + +def keys_of(db, rows): + """{recording id: its subtype's key} for recording rows (with TYPE_FIELDS), as of_recording tells one, with the + vocabulary read once.""" + types = all_types(db) + by_key = {t["key"]: t for t in types} + out = {} + for rec in rows: + chosen = by_key.get(rec.get("content_type") or "") + if chosen and chosen["base"] == base_of(rec): + out[rec["id"]] = chosen["key"] + continue + t = recognise(db, rec, types) + if t: + out[rec["id"]] = t["key"] + return out + + def choose(db, rid, key): """Set a resource's subtype (None: recognise it from the file again). It has to be of the resource's base type.""" rec = db.one("SELECT source, media, path FROM $r", r=R("recording", int(rid))) diff --git a/fastapi_backend/app/domain/find.py b/fastapi_backend/app/domain/find.py new file mode 100644 index 000000000..0180a55c7 --- /dev/null +++ b/fastapi_backend/app/domain/find.py @@ -0,0 +1,428 @@ +"""Find resources by any attribute: who is seen in them (faces) and heard (voices), the entities they mention, the +topics they're about, their kind (audio, video, transcript, document, image) and content type (podcast, scan, email …), +when they were made, and the rest of what's known about them (tags, language, kinds of object seen, emotions heard, +length, status, collection, where they came from), alone or together with words said or written in them. + +One filter for every way in: the API (POST /find), the MCP server (find), the assistant (find_resources) and the tool +graphs that call it. People and things can be named instead of numbered (faces=["Adeel"], topics=["Health"]): a name +matches the faces, speakers, entities (or the other ways an entity is said) and topics (any of their labels) called +that in the namespaces the asker reads, ignoring case, and `resolved` says which ones it matched. A topic brings its +narrower topics with it. A name that matches nothing is an error that says the closest names there are. + +Every resource found says why it matched (`why`), with moments to cite: where the words are said or written, when a +face is on screen (or the pages it's on), a speaker's first lines, the lines an entity is mentioned in, and the topics, +content type, kinds of object and emotions that matched. Filters combine with AND, the values of one filter with OR. +Nothing here calls a model, except `text` searching by meaning when that's set up (one embedding, in the activity +ledger), as search does. +""" + +from __future__ import annotations + +import datetime as dt +import difflib +import html +import re +from collections import defaultdict + +from . import analyze, content_types as ctypes, hierarchy, library, search as searchmod, store, topics as topicmod + +R = store.R +NAMED = ("faces", "speakers", "entities", "topics") # ids, or names resolved in the namespaces someone reads +WORDS = ("types", "content_types", "tags", "languages", "objects", "emotions", "status", "origins") +KEYS = ("text", "title", *NAMED, *WORDS, "date_from", "date_to", "min_duration", "max_duration", "collection", "sensitive") +SORTS = ("relevance", "-date", "date", "-title", "title", "-duration", "duration") +MAX_VALUES = 20 # values in one filter +MOMENTS = 3 # moments shown per reason +TEXT_HITS = 1000 # moments a text search looks through for the resources it finds +SPAN_RX = re.compile(r"^(\d{4})(?:-(\d{2}))?(?:-(\d{2}))?$") + + +def _key(s): + return " ".join(str(s or "").split()).casefold() + + +def _list(spec, k): + v = spec.get(k) + if v in (None, "", []): + return [] + vals = v if isinstance(v, (list, tuple)) else [v] + out = [] + for x in vals: + if isinstance(x, bool) or not isinstance(x, (str, int)): + raise ValueError(f"{k} is a list of names or ids") + if isinstance(x, str) and not x.strip(): + continue + out.append(x.strip() if isinstance(x, str) else x) + if len(out) > MAX_VALUES: + raise ValueError(f"{k} has at most {MAX_VALUES} values") + return list(dict.fromkeys(out)) + + +def day(value, end=False, name="date"): + """A day from 2026-09-30, a month (2026-09) or a year (2026): its first day, or with `end` its last.""" + if value in (None, ""): + return None + if isinstance(value, dt.date): + return value + m = SPAN_RX.match(str(value).strip()) + if not m: + raise ValueError(f"{name} is a day (2026-09-30), a month (2026-09) or a year (2026)") + y, mo, d = int(m.group(1)), m.group(2), m.group(3) + try: + if d: + return dt.date(y, int(mo), int(d)) + if mo: + first = dt.date(y, int(mo), 1) + return (first.replace(day=28) + dt.timedelta(days=4)).replace(day=1) - dt.timedelta(days=1) if end else first + return dt.date(y, 12, 31) if end else dt.date(y, 1, 1) + except ValueError: + raise ValueError(f"{name} {value} isn't a date") from None + + +def _closest(want, names): + near = difflib.get_close_matches(want, sorted(names), n=5, cutoff=0.5) + near += [n for n in sorted(names) if want in n and n not in near][: 5 - len(near)] + return f"; the closest: {', '.join(near)}" if near else "" + + +def _pick(what, values, rows, keys_of, norm=_key): + """The rows these values name: an id, or a name (`norm`ed) that one of `keys_of(row)` is.""" + by_id = {r["id"]: r for r in rows} + picked = {} + for v in values: + if isinstance(v, int) or (isinstance(v, str) and v.isdigit()): + r = by_id.get(int(v)) + if not r: + raise ValueError(f"{what} {v} isn't in a namespace you can read") + picked[r["id"]] = r + continue + want = norm(v) + hits = [r for r in rows if want in keys_of(r)] + if not hits: + known = {k for r in rows for k in keys_of(r)} + raise ValueError(f'no {what} called "{v}" in the namespaces you can read{_closest(want, known)}') + picked.update({r["id"]: r for r in hits}) + return list(picked.values()) + + +def _faces(db, spaces, values): + rows = db.rows("SELECT record::id(id) AS id, name, label, space FROM face WHERE space IN $s", s=spaces) + return _pick("face", values, rows, lambda r: {_key(r.get("name")), _key(r.get("label"))} - {""}) + + +def _speakers(db, spaces, values): + rows = db.rows("SELECT record::id(id) AS id, name, label, space FROM speaker WHERE space IN $s", s=spaces) + return _pick("speaker", values, rows, lambda r: {_key(r.get("name")), _key(r.get("label"))} - {""}) + + +def _entities(db, spaces, values): + names = [v for v in values if not (isinstance(v, int) or str(v).isdigit())] + ids = [int(v) for v in values if isinstance(v, int) or str(v).isdigit()] + keys = sorted({analyze.ent_key(v) for v in names}) + aliased = db.rows("SELECT key, entity FROM entity_alias WHERE key IN $k", k=keys) if keys else [] + rows = db.rows( + "SELECT record::id(id) AS id, name, key, space FROM entity WHERE space IN $s AND (key IN $k OR id IN $ids)", + s=spaces, + k=keys, + ids=[R("entity", i) for i in ids + [a["entity"] for a in aliased]], + ) + said = defaultdict(set) + for a in aliased: + said[a["entity"]].add(a["key"]) + out, missing = {}, [] + for v in values: + if isinstance(v, int) or str(v).isdigit(): + hit = [r for r in rows if r["id"] == int(v)] + if not hit: + raise ValueError(f"entity {v} isn't in a namespace you can read") + else: + k = analyze.ent_key(v) + hit = [r for r in rows if r["key"] == k or k in said[r["id"]]] + if not hit: + missing.append(v) + out.update({r["id"]: r for r in hit}) + if missing: + want = analyze.ent_key(missing[0]) + near = db.values( + "SELECT VALUE name FROM entity WHERE space IN $s AND string::contains(key, $w) LIMIT 5", s=spaces, w=want.split(" ")[0] + ) + hint = f"; the closest: {', '.join(dict.fromkeys(near))}" if near else "" + raise ValueError(f'no entity called "{missing[0]}" in the namespaces you can read{hint}') + return list(out.values()) + + +def _topics(db, spaces, values): + rows = db.rows("SELECT record::id(id) AS id, label, alt, broader, space FROM topic WHERE space IN $s", s=spaces) + named = _pick("topic", values, rows, lambda r: {topicmod.key_of(x) for x in [r["label"], *(r.get("alt") or [])]}, topicmod.key_of) + narrower = defaultdict(list) + for r in rows: + for b in r.get("broader") or []: + narrower[b].append(r["id"]) + ids, todo = {r["id"] for r in named}, [r["id"] for r in named] + while todo: + for c in narrower[todo.pop()]: + if c not in ids: + ids.add(c) + todo.append(c) + return named, sorted(ids) + + +def _content_types(db, values): + types = ctypes.all_types(db) + out = [] + for v in values: + want = _key(v) + t = next((t for t in types if t["key"] == want or _key(t["label"]) == want), None) + if not t: + raise ValueError(f'no content type "{v}"; there are {", ".join(t["key"] for t in types)}') + out.append(t) + return out + + +def clean(spec): + """The filter as given, checked: ValueError naming the first problem.""" + spec = {k: v for k, v in dict(spec or {}).items() if v not in (None, "", [])} + extra = sorted(set(spec) - set(KEYS)) + if extra: + raise ValueError(f"find by {', '.join(KEYS)}; not {extra[0]}") + out = {k: _list(spec, k) for k in NAMED + WORDS if k in spec} + for k in ("text", "title"): + if k in spec: + if not isinstance(spec[k], str): + raise ValueError(f"{k} is words") + out[k] = " ".join(spec[k].split())[:300] + for k in ("date_from", "date_to"): + if k in spec: + out[k] = day(spec[k], end=k == "date_to", name=k) + for k in ("min_duration", "max_duration"): + if k in spec: + if isinstance(spec[k], bool) or not isinstance(spec[k], (int, float)) or spec[k] < 0: + raise ValueError(f"{k} is a number of seconds") + out[k] = spec[k] + if "collection" in spec: + if isinstance(spec["collection"], bool) or not str(spec["collection"]).isdigit(): + raise ValueError("collection is a collection's id") + out["collection"] = int(spec["collection"]) + if "sensitive" in spec: + if spec["sensitive"] not in ("include", "only", "exclude"): + raise ValueError("sensitive is include, only or exclude") + out["sensitive"] = spec["sensitive"] + return out + + +def resolve(db, spaces, within, f): + """(the library filters for this find, what its names resolved to).""" + seen = sorted(set(spaces) | set(within or {})) + names = store.space_names(db) + shown = {} + filters = {"q": f.get("title")} + if f.get("faces"): + rows = _faces(db, seen, f["faces"]) + filters["faces"] = [r["id"] for r in rows] + shown["faces"] = [{"id": r["id"], "name": r.get("name") or r["label"], "namespace": names.get(r["space"])} for r in rows] + if f.get("speakers"): + rows = _speakers(db, seen, f["speakers"]) + filters["speakers"] = [r["id"] for r in rows] + shown["speakers"] = [{"id": r["id"], "name": r.get("name") or r["label"], "namespace": names.get(r["space"])} for r in rows] + if f.get("entities"): + rows = _entities(db, seen, f["entities"]) + filters["entities"] = [r["id"] for r in rows] + shown["entities"] = [{"id": r["id"], "name": r["name"], "namespace": names.get(r["space"])} for r in rows] + if f.get("topics"): + rows, ids = _topics(db, seen, f["topics"]) + filters["topics"] = ids + shown["topics"] = [{"id": r["id"], "name": r["label"], "namespace": names.get(r["space"])} for r in rows] + if len(ids) > len(rows): + shown["narrower_topics"] = len(ids) - len(rows) + if f.get("content_types"): + types = _content_types(db, f["content_types"]) + filters["content_types"] = [t["key"] for t in types] + shown["content_types"] = [{"key": t["key"], "name": t["label"]} for t in types] + if f.get("types"): + bad = [t for t in f["types"] if t not in library.MEDIA] + if bad: + raise ValueError(f"types are {', '.join(library.MEDIA)}; not {bad[0]}") + filters["media"] = f["types"] + for k, lib in (("tags", "tags"), ("languages", "languages"), ("objects", "objects"), ("emotions", "emotions"), ("origins", "origins")): + if f.get(k): + filters[lib] = [str(x) for x in f[k]] + if f.get("status"): + filters["status"] = [str(x) for x in f["status"]] + for k in ("date_from", "date_to", "min_duration", "max_duration", "sensitive"): + if f.get(k) is not None: + filters[k] = f[k] + if f.get("collection") is not None: + c = db.one("SELECT space FROM $r", r=R("collection", f["collection"])) + if not c or (c["space"] not in spaces and f["collection"] not in (within or {}).get(c["space"], [])): + raise ValueError(f"collection {f['collection']} isn't one you can read") + filters["collections"] = hierarchy.subtree(db, c["space"], f["collection"]) + return {k: v for k, v in filters.items() if v not in (None, "", [])}, shown + + +def _plain(snippet): + return html.unescape(re.sub(r"", "", snippet or "")) + + +def _words(db, cfg, text, spaces, within): + """A text search's moments, best first, for the resources it finds.""" + also = hierarchy.recordings_in(db, [c for cs in (within or {}).values() for c in cs]) + return searchmod.search( + db, text, None, limit=TEXT_HITS, spaces=set(spaces), also=also, files=True, objects=True, described=True, cfg=cfg, mode="auto" + ) + + +def find(db, cfg, spaces, within=None, spec=None, sort=None, limit=20, offset=0, ids=None): + """Resources matching the filter (KEYS), a page of them, each as the library lists it with `why` it matched: + {total, items, resolved, text}. `sort` is one of SORTS; relevance (the default with `text`) puts the resources + whose moments match the words best first, else they're newest first. `ids` keeps to these recordings (such as a + conversation's scope).""" + f = clean(spec) + sort = sort or ("relevance" if f.get("text") else "-date") + if sort not in SORTS: + raise ValueError(f"sort by {', '.join(SORTS)}") + if sort == "relevance" and not f.get("text"): + sort = "-date" + limit, offset = max(1, min(int(limit or 20), 100)), max(0, int(offset or 0)) + filters, shown = resolve(db, spaces, within, f) + hits, said = [], None + if ids is not None: + filters["only"] = {int(i) for i in ids} + if f.get("text"): + said = _words(db, cfg, f["text"], spaces, within) + hits = said["hits"] + found = {h["recording_id"] for h in hits} + filters["only"] = found if ids is None else found & filters["only"] + if not spaces and not within: + return {"total": 0, "items": [], "resolved": shown, "text": _said(said)} + if sort == "relevance": + cond, p = library.where(db, spaces, within=within, cfg=cfg, **filters) + found = set(db.values(f"SELECT VALUE record::id(id) FROM recording WHERE {cond}", **p)) + ranked = [i for i in dict.fromkeys(h["recording_id"] for h in hits) if i in found] + page = ranked[offset : offset + limit] + rows = db.rows(f"SELECT {library.FIELDS} FROM recording WHERE id IN $ids", ids=library.recs(page)) if page else [] + rows.sort(key=lambda r: page.index(r["id"])) + items, total = library.summaries(db, rows, cfg), len(ranked) + else: + items, total = library.list_recordings(db, spaces, sort=sort, limit=limit, offset=offset, within=within, cfg=cfg, **filters) + reasons = why(db, [x["id"] for x in items], filters, shown, hits, f.get("text")) + for x in items: + x["why"] = reasons.get(x["id"], []) + return {"total": total, "items": items, "resolved": shown, "text": _said(said)} + + +def _said(res): + if res is None: + return None + return {"mode": res["mode"], "meaning": res["meaning"], "moments": res["total"], "capped": res["capped"] or res["total"] >= TEXT_HITS} + + +def why(db, rids, filters, shown, hits=(), text=None): + """{recording id: [{by, name, moments?, count?}]}: why each of these resources matched, with moments to cite. A + moment has t0 and t1 (ms) on a timeline, or page (from 0) in a document or an image, and the words when it has any.""" + out = defaultdict(list) + if not rids: + return out + rr = library.recs(rids) + if text: + per = defaultdict(list) + for h in hits: + if h["recording_id"] in rids and len(per[h["recording_id"]]) < MOMENTS: + per[h["recording_id"]].append( + store.clean( + { + "t0": None if h.get("page") is not None else h.get("t0"), + "t1": None if h.get("page") is not None else h.get("t1"), + "page": h.get("page"), + "text": _plain(h.get("snippet")), + "speaker": h.get("speaker"), + "source": h.get("source"), + } + ) + ) + for rid, ms in per.items(): + out[rid].append({"by": "words", "name": text, "moments": ms}) + if filters.get("faces"): + names = {x["id"]: x["name"] for x in shown.get("faces", [])} + for t in db.rows( + "SELECT recording, face, spans, paged, first_ms FROM face_track WHERE recording IN $r AND face IN $f ORDER BY first_ms", + r=rids, + f=filters["faces"], + ): + spans = (t.get("spans") or [])[:MOMENTS] + moments = [{"page": a, "last_page": b - 1} if t.get("paged") else {"t0": a, "t1": b} for a, b in spans] + out[t["recording"]].append({"by": "face", "name": names.get(t["face"]), "moments": moments}) + if filters.get("speakers"): + names = {x["id"]: x["name"] for x in shown.get("speakers", [])} + for rid in rids: + lines = db.rows( + "SELECT speaker, idx, t0, t1, page, text FROM segment WHERE recording = $r AND speaker IN $s ORDER BY idx LIMIT 50", + r=rid, + s=filters["speakers"], + ) + first = defaultdict(list) + for s in lines: + if len(first[s["speaker"]]) < MOMENTS: + first[s["speaker"]].append(_line(s)) + for sid, ms in first.items(): + out[rid].append({"by": "voice", "name": names.get(sid), "moments": ms}) + if filters.get("entities"): + names = {x["id"]: x["name"] for x in shown.get("entities", [])} + ms = db.rows( + "SELECT recording, entity, record::id(in) AS segment FROM mentions WHERE recording IN $r AND entity IN $e", + r=rids, + e=filters["entities"], + ) + ms.sort(key=lambda m: m["segment"]) + keep = defaultdict(list) + for m in ms: + if len(keep[(m["recording"], m["entity"])]) < MOMENTS: + keep[(m["recording"], m["entity"])].append(m["segment"]) + segs = { + s["id"]: s + for s in ( + db.rows( + "SELECT record::id(id) AS id, t0, t1, page, text FROM segment WHERE id IN $ids", + ids=[R("segment", i) for v in keep.values() for i in v], + ) + if keep + else [] + ) + } + for (rid, eid), ids in keep.items(): + out[rid].append({"by": "entity", "name": names.get(eid), "moments": [_line(segs[i]) for i in ids if i in segs]}) + if filters.get("topics"): + labels = { + t["id"]: t["label"] + for t in db.rows("SELECT record::id(id) AS id, label FROM topic WHERE id IN $t", t=[R("topic", i) for i in filters["topics"]]) + } + for a in db.rows( + "SELECT recording, topic FROM topic_about WHERE recording IN $r AND topic IN $t AND status = 'accepted'", + r=rids, + t=filters["topics"], + ): + out[a["recording"]].append({"by": "topic", "name": labels.get(a["topic"])}) + if filters.get("content_types"): + labels = {x["key"]: x["name"] for x in shown.get("content_types", [])} + keys = ctypes.keys_of(db, db.rows(f"SELECT {ctypes.TYPE_FIELDS} FROM recording WHERE id IN $r", r=rr)) + for rid, k in keys.items(): + if k in labels: + out[rid].append({"by": "content_type", "name": labels[k]}) + if filters.get("objects"): + want = {_key(o) for o in filters["objects"]} + for r in db.rows("SELECT record::id(id) AS id, objects FROM recording WHERE id IN $r", r=rr): + for o in sorted(want & set(r.get("objects") or [])): + out[r["id"]].append({"by": "object", "name": o}) + if filters.get("emotions"): + want = {_key(e) for e in filters["emotions"]} + for r in db.rows("SELECT record::id(id) AS id, stats.emotions AS e FROM recording WHERE id IN $r", r=rr): + for k, n in sorted((r.get("e") or {}).items()): + if n and k.casefold() in want: + out[r["id"]].append({"by": "emotion", "name": k, "count": n}) + return out + + +def _line(s): + paged = s.get("page") is not None + return store.clean( + {"t0": None if paged else s.get("t0"), "t1": None if paged else s.get("t1"), "page": s.get("page"), "text": s.get("text")} + ) diff --git a/fastapi_backend/app/domain/library.py b/fastapi_backend/app/domain/library.py index 9609795d0..f3d6f0987 100644 --- a/fastapi_backend/app/domain/library.py +++ b/fastapi_backend/app/domain/library.py @@ -8,7 +8,7 @@ import re from collections import Counter, defaultdict -from . import access as acc, metadata, nsfw, render, store +from . import access as acc, content_types as ctypes, metadata, nsfw, render, store R = store.R STATUSES = ("new", "transcribed", "diarized", "analyzed", "error") @@ -40,6 +40,13 @@ TAG_MAX, TAGS_MAX = 40, 20 # characters in a tag, tags on a recording MONTHS_MAX = 240 # months in a namespace's stats: the latest twenty years DAY = re.compile(r"\d{4}-\d{2}-\d{2}") +MEDIA_WHERE = { + "video": "media.kind = 'video'", + "audio": "(media.kind = 'audio' OR (media.kind = NONE AND source = 'audio'))", + "transcript": "(media.kind = 'transcript' OR (media.kind = NONE AND source NOT IN ['audio', 'document', 'image']))", + "document": "source = 'document'", + "image": "source = 'image'", +} def recs(ids): @@ -196,12 +203,19 @@ def where( field=None, objects=None, sensitive=None, + faces=None, + entities=None, + topics=None, + content_types=None, + emotions=None, + only=None, ): """The WHERE clause and its parameters for these filters. Filters combine with AND, the values of one filter with OR. q: every word in the title, the namespace's name or a speaker's name. status: recording statuses and job states (STATES). attention: errored, latest job failed, or a voice match to review. processing: a job queued or running. speakers: speaker ids. date_from/date_to: the recording date, inclusive. min/max_duration: seconds, max exclusive. + media: a kind (MEDIA), or a list of them. access: levels (public, restricted, private), a namespace's default counting for recordings without their own. featured: true or false. tags: any of these tags (ignoring case). origins: where they came from (Origins; needs cfg). languages: language codes (ignoring case), "none" for recordings whose language isn't known. edited: recording ids @@ -210,6 +224,9 @@ def where( from fields.filter_condition. objects: any of these kinds of object seen in it (ignoring case; objects.py). sensitive (nsfw.py, where NSFW classification is on): only those flagged NSFW, exclude them, or include them; without it (and with cfg), flagged ones are left out where their namespace hides them. + faces: face identity ids, seen in it (faces.py). entities: entity ids, mentioned in it. topics: topic ids, that it + was accepted as about (topics.py). content_types: subtype keys, its content type (content_types.py). emotions: + emotions heard in it (ignoring case). only: recording ids (such as those another search found). """ base, p = in_scope(spaces, within) w = [base] @@ -261,18 +278,12 @@ def where( if max_duration is not None: p["dmax"] = int(max_duration * 1000) w.append("duration_ms != NONE AND duration_ms < $dmax") - if media: - if media not in MEDIA: - raise ValueError(f"media is one of {', '.join(MEDIA)}") - w.append( - { - "video": "media.kind = 'video'", - "audio": "(media.kind = 'audio' OR (media.kind = NONE AND source = 'audio'))", - "transcript": "(media.kind = 'transcript' OR (media.kind = NONE AND source NOT IN ['audio', 'document', 'image']))", - "document": "source = 'document'", - "image": "source = 'image'", - }[media] - ) + kinds_wanted = [media] if isinstance(media, str) else list(dict.fromkeys(media or [])) + if any(m not in MEDIA for m in kinds_wanted): + raise ValueError(f"media is one of {', '.join(MEDIA)}") + if kinds_wanted: + conds = [MEDIA_WHERE[m] for m in kinds_wanted] + w.append(conds[0] if len(conds) == 1 else "(" + " OR ".join(conds) + ")") levels = list(dict.fromkeys(access or [])) if levels: if any(x not in acc.LEVELS for x in levels): @@ -318,6 +329,32 @@ def where( if edited is not None: p["edited"] = recs(sorted(edited)) w.append("id IN $edited") + if only is not None: + p["only"] = recs(sorted({int(i) for i in only})) + w.append("id IN $only") + if faces: + p["by_face"] = recs(db.values("SELECT VALUE recording FROM face_track WHERE face IN $f", f=sorted({int(x) for x in faces}))) + w.append("id IN $by_face") + if entities: + ents = sorted({int(x) for x in entities}) + p["by_entity"] = recs(db.values("SELECT VALUE recording FROM mentions WHERE entity IN $e", e=ents)) + w.append("id IN $by_entity") + if topics: + tids = sorted({int(x) for x in topics}) + p["by_topic"] = recs(db.values("SELECT VALUE recording FROM topic_about WHERE topic IN $t AND status = 'accepted'", t=tids)) + w.append("id IN $by_topic") + if content_types: + keys = set(content_types) + rows = db.rows(f"SELECT {ctypes.TYPE_FIELDS} FROM recording WHERE space IN $s", s=spaces) + p["by_type"] = recs(sorted(i for i, k in ctypes.keys_of(db, rows).items() if k in keys)) + w.append("id IN $by_type") + moods = {" ".join(e.split()).casefold() for e in emotions or [] if e and e.strip()} + if moods: + heard = db.rows( + "SELECT record::id(id) AS id, stats.emotions AS e FROM recording WHERE space IN $s AND stats.emotions != NONE", s=spaces + ) + p["by_emotion"] = recs(sorted(r["id"] for r in heard if moods & {k.casefold() for k, n in (r.get("e") or {}).items() if n})) + w.append("id IN $by_emotion") if collections is not None: p["cols"] = sorted(collections) w.append("collection IN $cols") diff --git a/fastapi_backend/app/schemas/find.py b/fastapi_backend/app/schemas/find.py new file mode 100644 index 000000000..1d3ae9960 --- /dev/null +++ b/fastapi_backend/app/schemas/find.py @@ -0,0 +1,102 @@ +"""Finding resources by any attribute (domain/find.py).""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import Field + +from app.schemas.common import RequestModel, ResponseModel +from app.schemas.recordings import MediaKind, RecordingState, RecordingSummary + +Named = list[int | str] + + +class FindQuery(RequestModel): + """Filters combine with AND, the values of one filter with OR. Faces, speakers, entities and topics are ids or names + (ignoring case; an entity's other names count, and any of a topic's labels).""" + + namespace: str | None = Field(None, description="one namespace (default: every namespace you can read)") + text: str | None = Field( + None, + max_length=300, + description='words said or written in them, as search takes them (words, "phrases", OR); by meaning too when that is set up', + ) + title: str | None = Field( + None, max_length=200, description="words that must all appear in the title, the namespace or a speaker's name" + ) + faces: Named | None = Field(None, max_length=20, description="people seen in them: face identities, by id or name") + speakers: Named | None = Field(None, max_length=20, description="voices heard in them: speakers, by id or name") + entities: Named | None = Field(None, max_length=20, description="entities they mention, by id or name") + topics: Named | None = Field( + None, max_length=20, description="topics they were accepted as about, by id or label; a topic's narrower topics count too" + ) + types: list[MediaKind] | None = Field(None, max_length=5, description="their kind: audio, video, transcript, document or image") + content_types: list[str] | None = Field( + None, max_length=20, description="content types (podcast, interview, scan, photo, email …, GET /content-types), by key or label" + ) + date_from: str | None = Field(None, description="made on or after this day, month or year (2026-09-30, 2026-09, 2026)") + date_to: str | None = Field(None, description="made on or before this day, month or year") + min_duration: float | None = Field(None, ge=0, description="at least this many seconds long") + max_duration: float | None = Field(None, ge=0, description="shorter than this many seconds") + tags: list[str] | None = Field(None, max_length=20, description="tags (ignoring case)") + languages: list[str] | None = Field(None, max_length=20, description="language codes; none for those whose language isn't known") + objects: list[str] | None = Field(None, max_length=20, description="kinds of object seen in them (person, car, dog …)") + emotions: list[str] | None = Field(None, max_length=20, description="emotions heard in them (ignoring case)") + status: list[RecordingState] | None = Field(None, max_length=10, description="statuses, or processing and failed") + origins: list[str] | None = Field( + None, max_length=20, description="where they came from: source:, upload, paste, iiif, folder or file" + ) + collection: int | None = Field(None, description="a collection: what's in it and in the collections inside it") + sensitive: Literal["include", "only", "exclude"] | None = Field( + None, description="where NSFW classification is on: only those flagged, exclude them, or include them" + ) + sort: Literal["relevance", "-date", "date", "-title", "title", "-duration", "duration"] | None = Field( + None, description="relevance (the default with text: best matching words first), else newest first" + ) + limit: int = Field(20, ge=1, le=100) + offset: int = Field(0, ge=0, le=10000) + + +class FindMoment(ResponseModel): + t0: int | None = Field(None, description="ms from the start, on a timeline") + t1: int | None = Field(None, description="ms, where it ends") + page: int | None = Field(None, description="in a document or an image: the page (from 0)") + last_page: int | None = Field(None, description="a face over several pages: the last of them") + text: str | None = Field(None, description="the words there, when it has any") + speaker: str | None = None + source: str | None = Field(None, description="words: said, screen, file, page, object or described (as search has it)") + + +class FindReason(ResponseModel): + by: Literal["words", "face", "voice", "entity", "topic", "content_type", "object", "emotion"] + name: str | None = Field(None, description="what matched: the words, the person's or the entity's name, the topic's label …") + moments: list[FindMoment] = Field(default_factory=list, description="up to 3 moments that show it, to cite") + count: int | None = Field(None, description="emotion: how many lines have it") + + +class FindItem(RecordingSummary): + why: list[FindReason] = Field(default_factory=list, description="why it matched") + + +class FindNamed(ResponseModel): + id: int | None = None + key: str | None = None + name: str | None = None + namespace: str | None = None + + +class FindText(ResponseModel): + mode: str = Field(description="how the words were matched: keyword, semantic or hybrid") + meaning: str | None = Field(None, description="why not by meaning, when it was asked and couldn't be") + moments: int = Field(description="moments the words were found in") + capped: bool = Field(description="more moments matched than were looked through, so some resources may be missing") + + +class FindResults(ResponseModel): + total: int = Field(description="how many resources match, on all pages") + items: list[FindItem] + resolved: dict[str, list[FindNamed] | int] = Field( + default_factory=dict, description="what the names given matched, by filter (and how many narrower topics came with them)" + ) + text: FindText | None = None diff --git a/fastapi_backend/tests/api/test_find.py b/fastapi_backend/tests/api/test_find.py new file mode 100644 index 000000000..38f75a432 --- /dev/null +++ b/fastapi_backend/tests/api/test_find.py @@ -0,0 +1,222 @@ +"""Finding resources by any attribute (domain/find.py): faces, voices, entities, topics, kinds, content types, dates and +the rest, by id or by name, through the API, the library list, the MCP server and the assistant, each result saying why +it matched.""" + +from __future__ import annotations + +import pytest + +from app.domain import ai_tools, content_types, faces, find, store, topics +from tests.api.test_mcp import rpc +from tests.api.test_oauth import _bearer, _grant +from tests.helpers import login, make_user, seed + +R = store.R + + +@pytest.fixture +def env(client, db, cfg, folder): + """ep1 and ep2 in pods, call in calls; Adeel's face on ep2 and on the call; topics Health > Gene therapy, ep1 about + the narrower one; the call chosen as an email; dates across two years.""" + ep1, ep2, call = seed(db, cfg, folder) + pods, calls = store.ns_id(db, "pods"), store.ns_id(db, "calls") + for rid, day in ((ep1, "2026-09-30T09:00:00"), (ep2, "2026-03-02T10:00:00"), (call, "2025-11-20T08:00:00")): + db.q("UPDATE $r SET recorded_at = $d", r=R("recording", rid), d=day) + adeel = faces.new_face(db, pods, name="Adeel") + adeel_too = faces.new_face(db, calls, name="Adeel") + stranger = faces.new_face(db, pods) + db.q( + "INSERT INTO face_track $rows", + rows=[ + { + "recording": ep2, + "space": pods, + "local": "P1", + "face": adeel, + "spans": [[5000, 9000], [20000, 25000]], + "screen_ms": 9000, + "first_ms": 5000, + }, + {"recording": ep1, "space": pods, "local": "P1", "face": stranger, "spans": [[0, 1000]], "screen_ms": 1000, "first_ms": 0}, + { + "recording": call, + "space": calls, + "local": "P1", + "face": adeel_too, + "spans": [[1000, 4000]], + "screen_ms": 3000, + "first_ms": 1000, + }, + ], + ) + health = topics.create(db, pods, "Health") + gene = topics.create(db, pods, "Gene therapy", alt=["AAV"], broader=[health]) + topics.tag(db, gene, [ep1]) + content_types.choose(db, call, "email") + make_user(db, "root@x.io", "root password 1", admin=True) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + return { + "ep1": ep1, + "ep2": ep2, + "call": call, + "pods": pods, + "calls": calls, + "adeel": adeel, + "adeel_too": adeel_too, + "gene": gene, + "health": health, + "h": login(client, "root@x.io", "root password 1"), + "hv": login(client, "vi@x.io", "viewer password 1"), + } + + +def found(client, h, **body): + r = client.post("/api/v1/find", json=body, headers=h) + assert r.status_code == 200, r.text + return r.json() + + +def ids(res): + return [x["id"] for x in res["items"]] + + +def brief(why): + """Reasons without the fields they leave empty.""" + return [ + { + k: v + for k, v in {**w, "moments": [{k: v for k, v in m.items() if v is not None} for m in w["moments"]]}.items() + if v not in (None, []) + } + for w in why + ] + + +def test_by_face_voice_entity_and_date(client, db, env): + h, ep1, ep2, call = env["h"], env["ep1"], env["ep2"], env["call"] + # a face by name matches that name in every namespace you read; each result says when it's on screen + res = found(client, h, faces=["adeel"]) + assert ids(res) == [ep2, call] # newest first + assert {x["namespace"] for x in res["resolved"]["faces"]} == {"pods", "calls"} + why = brief(res["items"][0]["why"]) + assert why == [{"by": "face", "name": "Adeel", "moments": [{"t0": 5000, "t1": 9000}, {"t0": 20000, "t1": 25000}]}] + # by id, too, and an unnamed face by its label + assert ids(found(client, h, faces=[env["adeel"]])) == [ep2] + assert ids(found(client, h, faces=["face 2"])) == [ep1] + # a voice: the speaker's first lines are the moments + res = found(client, h, speakers=["Carol"]) + assert ids(res) == [ep2] + (w,) = res["items"][0]["why"] + assert w["by"] == "voice" and w["name"] == "Carol" and "exploit benchmark" in w["moments"][0]["text"] + # an entity by any of its names; the lines it's mentioned in + res = found(client, h, entities=["dyno therapeutics"]) + assert set(ids(res)) == {ep1, ep2, call} + first = next(x for x in res["items"] if x["id"] == ep1)["why"][0] + assert first["by"] == "entity" and len(first["moments"]) == 3 and "Dyno Therapeutics" in first["moments"][0]["text"] + # AND across filters, OR inside one; dates as a day, a month or a year + assert ids(found(client, h, entities=["Dyno Therapeutics"], speakers=["Dave"])) == [call] + assert ids(found(client, h, faces=["Adeel"], date_from="2026")) == [ep2] + assert ids(found(client, h, date_from="2026-03", date_to="2026-03")) == [ep2] + assert ids(found(client, h, date_to="2025")) == [call] + assert ids(found(client, h, speakers=["Bob", "Dave"], sort="date")) == [call, ep1] + + +def test_by_topic_type_content_type_and_text(client, db, env): + h, ep1, ep2, call = env["h"], env["ep1"], env["ep2"], env["call"] + # a topic brings its narrower topics; any label works + res = found(client, h, topics=["health"]) + assert ids(res) == [ep1] and res["resolved"]["narrower_topics"] == 1 + assert brief(res["items"][0]["why"]) == [{"by": "topic", "name": "Gene therapy"}] + assert ids(found(client, h, topics=["aav"])) == [ep1] + # content types by key or label; kinds as a list + res = found(client, h, content_types=["Email"]) + assert ids(res) == [call] and brief(res["items"][0]["why"]) == [{"by": "content_type", "name": "Email"}] + assert ids(found(client, h, content_types=["text"])) == [ep1, ep2] + db.q("UPDATE $r SET source = 'audio'", r=R("recording", ep2)) + assert ids(found(client, h, types=["audio", "video"])) == [ep2] + # words said in them: best matching first, with the moments + res = found(client, h, text="capsid") + best = client.get("/api/v1/search", params={"q": "capsid"}, headers=h).json()["hits"][0]["recording_id"] + assert ids(res)[0] == best and set(ids(res)) == {ep1, call} # the best matching moment's resource first + w = next(x for x in res["items"] if x["id"] == ep1)["why"][0] + assert w["by"] == "words" and w["name"] == "capsid" and len(w["moments"]) == 3 and "capsid" in w["moments"][0]["text"] + assert res["text"]["mode"] == "keyword" and res["text"]["capped"] is False + # words and attributes together + assert ids(found(client, h, text="capsid", namespace="calls")) == [call] + assert ids(found(client, h, text="capsid", faces=["Adeel"])) == [call] + # emotions heard + assert ids(found(client, h, emotions=["fear"])) == [ep2] + + +def test_names_that_match_nothing_and_scope(client, db, env): + h, hv = env["h"], env["hv"] + r = client.post("/api/v1/find", json={"speakers": ["Carl"]}, headers=h) + assert r.status_code == 400 and 'no speaker called "Carl"' in r.json()["detail"] and "carol" in r.json()["detail"] + r = client.post("/api/v1/find", json={"entities": ["Dino Therapeutics"]}, headers=h) + assert r.status_code == 400 and "no entity called" in r.json()["detail"] + r = client.post("/api/v1/find", json={"content_types": ["vlog"]}, headers=h) + assert r.status_code == 400 and "podcast" in r.json()["detail"] + r = client.post("/api/v1/find", json={"whatever": 1}, headers=h) + assert r.status_code == 422 + # a viewer of pods finds nothing of calls, even by a name only calls has + assert ids(found(client, hv, faces=["Adeel"])) == [env["ep2"]] + r = client.post("/api/v1/find", json={"speakers": ["Dave"]}, headers=hv) + assert r.status_code == 400 + r = client.post("/api/v1/find", json={"faces": [env["adeel_too"]]}, headers=hv) + assert r.status_code == 400 and "isn't in a namespace you can read" in r.json()["detail"] + assert client.post("/api/v1/find", json={"namespace": "calls"}, headers=hv).status_code == 404 + + +def test_library_list_filters_by_the_same(client, db, env): + h, ep1, ep2, call = env["h"], env["ep1"], env["ep2"], env["call"] + + def listing(**params): + r = client.get("/api/v1/recordings", params=params, headers=h) + assert r.status_code == 200, r.text + return [x["id"] for x in r.json()] + + assert listing(face=[env["adeel"], env["adeel_too"]]) == [ep1, ep2, call][1:] + dyno = db.values("SELECT VALUE record::id(id) FROM entity WHERE key = 'dyno therapeutics' AND space = $s", s=env["calls"]) + assert listing(entity=dyno) == [call] + assert listing(topic=[env["gene"]]) == [ep1] + assert listing(topic=[env["health"]]) == [] # the list takes a topic as it is; find brings the narrower ones + assert listing(content_type="email") == [call] + assert listing(emotion="Happy") == [ep1] + + +def test_find_for_agents(client, db, cfg, env): + # MCP: names, results with links to each moment + hv = env["hv"] + _, tokens = _grant(client, hv, scope="read") + res = rpc(client, _bearer(tokens), "tools/call", {"name": "find", "arguments": {"faces": ["Adeel"], "types": ["transcript"]}})["result"] + assert res["isError"] is False + out = res["structuredContent"] + assert out["total"] == 1 and out["results"][0]["recording_id"] == env["ep2"] + (w,) = out["results"][0]["why"] + assert w["by"] == "face" and w["moments"][0] == { + "time": "0:05", + "until": "0:09", + "url": f"http://localhost:3000/resources/{env['ep2']}?t=5", + } + bad = rpc(client, _bearer(tokens), "tools/call", {"name": "find", "arguments": {"topics": ["Wealth"]}})["result"] + assert bad["isError"] is True and "no topic called" in bad["content"][0]["text"] + # the assistant: numbered moments to cite, within the conversation's scope + me = {"id": 1, "email": "root@x.io"} + box = ai_tools.Toolbox(db, cfg, me, {env["pods"], env["calls"]}, set(), {"recordings": [env["ep1"], env["call"]]}, None) + assert "find_resources" in {t["function"]["name"] for t in box.specs()} + got, summary = box.t_find_resources(entities=["Dyno Therapeutics"]) + assert summary == "Found 2 resource(s)" + assert {x["recording_id"] for x in got["results"]} == {env["ep1"], env["call"]} + refs = [m["ref"] for x in got["results"] for w in x["why"] for m in w["moments"]] + assert refs == list(range(1, len(refs) + 1)) and len(box.refs) == len(refs) + + +def test_clean_checks_the_filter(): + assert find.day("2024-02", end=True).isoformat() == "2024-02-29" + assert find.day("2026", end=True).isoformat() == "2026-12-31" + with pytest.raises(ValueError, match="a day"): + find.day("last week") + with pytest.raises(ValueError, match="not colour"): + find.clean({"colour": "red"}) + with pytest.raises(ValueError, match="at most 20"): + find.clean({"tags": [str(i) for i in range(21)]}) diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index af490230d..f1663d971 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -937,11 +937,23 @@ export type { FindGraphPathErrors, FindGraphPathResponse, FindGraphPathResponses, + FindItem, + FindMoment, + FindNamed, + FindQuery, + FindReason, + FindResourcesData, + FindResourcesError, + FindResourcesErrors, + FindResourcesResponse, + FindResourcesResponses, + FindResults, FindSimilarData, FindSimilarError, FindSimilarErrors, FindSimilarResponse, FindSimilarResponses, + FindText, FinishData, FinishError, FinishErrors, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index e3bfc4787..6f85dc207 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -433,6 +433,9 @@ import type { FindGraphPathData, FindGraphPathErrors, FindGraphPathResponses, + FindResourcesData, + FindResourcesErrors, + FindResourcesResponses, FindSimilarData, FindSimilarErrors, FindSimilarResponses, @@ -6103,6 +6106,29 @@ export class Search { ...options, }); } + + /** + * Find Resources + * + * Resources you can read that match every filter given: the people seen in them (faces) and heard (speakers), the + * entities they mention, the topics they're about, their kind and content type, when they were made, their tags, + * language, objects seen, emotions heard, length, status, collection and origin, and words said or written in them. + * Faces, speakers, entities and topics can be named instead of numbered; `resolved` says what each name matched, and + * a name that matches nothing is a 400 naming the closest ones. Each resource says `why` it matched, with up to three + * moments per reason to cite (a time, or a page). With `text`, the best matching come first. + */ + public static findResources( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/find", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } } export class Searches { diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index b0d2a2f70..b1f221edd 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -5759,6 +5759,460 @@ export type FileUpdate = { description?: string | null; }; +/** + * FindItem + */ +export type FindItem = { + /** + * Id + */ + id: number; + /** + * Title + */ + title?: string | null; + /** + * Recorded At + */ + recorded_at?: string | null; + /** + * Duration Ms + */ + duration_ms?: number | null; + /** + * Status + */ + status?: string | null; + /** + * Error + */ + error?: string | null; + /** + * Source + */ + source?: string | null; + /** + * Space + */ + space: number; + /** + * Namespace + */ + namespace?: string | null; + /** + * Collection + * + * the collection it lives in + */ + collection?: number | null; + /** + * Collection Name + */ + collection_name?: string | null; + /** + * Media Kind + * + * audio, video, transcript (text without media), document or image + */ + media_kind: string; + /** + * Pages + * + * a document's or an image's pages + */ + pages?: number | null; + /** + * Poster + * + * signed link to a video's first frame, or a document's or image's first page + */ + poster?: string | null; + /** + * Emotions + */ + emotions?: { + [key: string]: unknown; + }; + /** + * Words + */ + words?: number | null; + /** + * Importance + */ + importance?: unknown; + /** + * Sentiment + */ + sentiment?: unknown; + /** + * Speakers + * + * speaker names, comma separated + */ + speakers?: string; + /** + * Access + * + * its own access, or its namespace's default + */ + access?: "public" | "restricted" | "private"; + /** + * Open + * + * the parts anyone may use when it is public + */ + open?: Array<"media" | "transcript" | "index">; + /** + * Featured + */ + featured?: boolean; + /** + * Tags + */ + tags?: Array; + /** + * Language + * + * its language code, when known + */ + language?: string | null; + /** + * Origin + * + * where it came from: source: (a connected source), upload, paste, iiif, folder or file + */ + origin?: string | null; + /** + * Origin Name + * + * what to call its origin: the source's name, or e.g. Uploaded + */ + origin_name?: string | null; + /** + * Role + * + * your role on it: through its namespace or its collection (an admin of the collection is an owner) + */ + role?: "viewer" | "editor" | "owner" | null; + /** + * NSFW classification, where it's on in its namespace and it was classified + */ + nsfw?: NsfwState | null; + /** + * Why + * + * why it matched + */ + why?: Array; + [key: string]: unknown; +}; + +/** + * FindMoment + */ +export type FindMoment = { + /** + * T0 + * + * ms from the start, on a timeline + */ + t0?: number | null; + /** + * T1 + * + * ms, where it ends + */ + t1?: number | null; + /** + * Page + * + * in a document or an image: the page (from 0) + */ + page?: number | null; + /** + * Last Page + * + * a face over several pages: the last of them + */ + last_page?: number | null; + /** + * Text + * + * the words there, when it has any + */ + text?: string | null; + /** + * Speaker + */ + speaker?: string | null; + /** + * Source + * + * words: said, screen, file, page, object or described (as search has it) + */ + source?: string | null; + [key: string]: unknown; +}; + +/** + * FindNamed + */ +export type FindNamed = { + /** + * Id + */ + id?: number | null; + /** + * Key + */ + key?: string | null; + /** + * Name + */ + name?: string | null; + /** + * Namespace + */ + namespace?: string | null; + [key: string]: unknown; +}; + +/** + * FindQuery + * + * Filters combine with AND, the values of one filter with OR. Faces, speakers, entities and topics are ids or names + * (ignoring case; an entity's other names count, and any of a topic's labels). + */ +export type FindQuery = { + /** + * Namespace + * + * one namespace (default: every namespace you can read) + */ + namespace?: string | null; + /** + * Text + * + * words said or written in them, as search takes them (words, "phrases", OR); by meaning too when that is set up + */ + text?: string | null; + /** + * Title + * + * words that must all appear in the title, the namespace or a speaker's name + */ + title?: string | null; + /** + * Faces + * + * people seen in them: face identities, by id or name + */ + faces?: Array | null; + /** + * Speakers + * + * voices heard in them: speakers, by id or name + */ + speakers?: Array | null; + /** + * Entities + * + * entities they mention, by id or name + */ + entities?: Array | null; + /** + * Topics + * + * topics they were accepted as about, by id or label; a topic's narrower topics count too + */ + topics?: Array | null; + /** + * Types + * + * their kind: audio, video, transcript, document or image + */ + types?: Array<"audio" | "video" | "transcript" | "document" | "image"> | null; + /** + * Content Types + * + * content types (podcast, interview, scan, photo, email …, GET /content-types), by key or label + */ + content_types?: Array | null; + /** + * Date From + * + * made on or after this day, month or year (2026-09-30, 2026-09, 2026) + */ + date_from?: string | null; + /** + * Date To + * + * made on or before this day, month or year + */ + date_to?: string | null; + /** + * Min Duration + * + * at least this many seconds long + */ + min_duration?: number | null; + /** + * Max Duration + * + * shorter than this many seconds + */ + max_duration?: number | null; + /** + * Tags + * + * tags (ignoring case) + */ + tags?: Array | null; + /** + * Languages + * + * language codes; none for those whose language isn't known + */ + languages?: Array | null; + /** + * Objects + * + * kinds of object seen in them (person, car, dog …) + */ + objects?: Array | null; + /** + * Emotions + * + * emotions heard in them (ignoring case) + */ + emotions?: Array | null; + /** + * Status + * + * statuses, or processing and failed + */ + status?: Array<"new" | "transcribed" | "diarized" | "analyzed" | "error" | "processing" | "failed"> | null; + /** + * Origins + * + * where they came from: source:, upload, paste, iiif, folder or file + */ + origins?: Array | null; + /** + * Collection + * + * a collection: what's in it and in the collections inside it + */ + collection?: number | null; + /** + * Sensitive + * + * where NSFW classification is on: only those flagged, exclude them, or include them + */ + sensitive?: "include" | "only" | "exclude" | null; + /** + * Sort + * + * relevance (the default with text: best matching words first), else newest first + */ + sort?: "relevance" | "-date" | "date" | "-title" | "title" | "-duration" | "duration" | null; + /** + * Limit + */ + limit?: number; + /** + * Offset + */ + offset?: number; +}; + +/** + * FindReason + */ +export type FindReason = { + /** + * By + */ + by: "words" | "face" | "voice" | "entity" | "topic" | "content_type" | "object" | "emotion"; + /** + * Name + * + * what matched: the words, the person's or the entity's name, the topic's label … + */ + name?: string | null; + /** + * Moments + * + * up to 3 moments that show it, to cite + */ + moments?: Array; + /** + * Count + * + * emotion: how many lines have it + */ + count?: number | null; + [key: string]: unknown; +}; + +/** + * FindResults + */ +export type FindResults = { + /** + * Total + * + * how many resources match, on all pages + */ + total: number; + /** + * Items + */ + items: Array; + /** + * Resolved + * + * what the names given matched, by filter (and how many narrower topics came with them) + */ + resolved?: { + [key: string]: Array | number; + }; + text?: FindText | null; + [key: string]: unknown; +}; + +/** + * FindText + */ +export type FindText = { + /** + * Mode + * + * how the words were matched: keyword, semantic or hybrid + */ + mode: string; + /** + * Meaning + * + * why not by meaning, when it was asked and couldn't be + */ + meaning?: string | null; + /** + * Moments + * + * moments the words were found in + */ + moments: number; + /** + * Capped + * + * more moments matched than were looked through, so some resources may be missing + */ + capped: boolean; + [key: string]: unknown; +}; + /** * ForgotPasswordRequest */ @@ -24985,6 +25439,36 @@ export type ListRecordingsData = { * where NSFW classification is on (docs/nsfw.md): only those flagged NSFW, exclude them, or include them. Without it, flagged ones are left out where their namespace hides them (display hide) */ sensitive?: "include" | "only" | "exclude" | null; + /** + * Face + * + * face identity ids: the recordings they're seen in; repeat for several + */ + face?: Array | null; + /** + * Entity + * + * entity ids: the recordings mentioning them; repeat for several + */ + entity?: Array | null; + /** + * Topic + * + * topic ids: the recordings accepted as about them (not their narrower topics); repeat for several + */ + topic?: Array | null; + /** + * Content Type + * + * content type keys (podcast, scan, email …; GET /content-types): their recordings; repeat for several + */ + content_type?: Array | null; + /** + * Emotion + * + * emotions heard in them (ignoring case); repeat for several + */ + emotion?: Array | null; /** * Collection * @@ -28567,6 +29051,31 @@ export type ListMentionsResponses = { export type ListMentionsResponse = ListMentionsResponses[keyof ListMentionsResponses]; +export type FindResourcesData = { + body: FindQuery; + path?: never; + query?: never; + url: "/api/v1/find"; +}; + +export type FindResourcesErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type FindResourcesError = FindResourcesErrors[keyof FindResourcesErrors]; + +export type FindResourcesResponses = { + /** + * Successful Response + */ + 200: FindResults; +}; + +export type FindResourcesResponse = FindResourcesResponses[keyof FindResourcesResponses]; + export type ListSearchesData = { body?: never; path?: never; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 257441c20..676c5178e 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -7156,6 +7156,111 @@ }, "description": "where NSFW classification is on (docs/nsfw.md): only those flagged NSFW, exclude them, or include them. Without it, flagged ones are left out where their namespace hides them (display hide)" }, + { + "name": "face", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "integer" + } + }, + { + "type": "null" + } + ], + "description": "face identity ids: the recordings they're seen in; repeat for several", + "title": "Face" + }, + "description": "face identity ids: the recordings they're seen in; repeat for several" + }, + { + "name": "entity", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "integer" + } + }, + { + "type": "null" + } + ], + "description": "entity ids: the recordings mentioning them; repeat for several", + "title": "Entity" + }, + "description": "entity ids: the recordings mentioning them; repeat for several" + }, + { + "name": "topic", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "integer" + } + }, + { + "type": "null" + } + ], + "description": "topic ids: the recordings accepted as about them (not their narrower topics); repeat for several", + "title": "Topic" + }, + "description": "topic ids: the recordings accepted as about them (not their narrower topics); repeat for several" + }, + { + "name": "content_type", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "null" + } + ], + "description": "content type keys (podcast, scan, email \u2026; GET /content-types): their recordings; repeat for several", + "title": "Content Type" + }, + "description": "content type keys (podcast, scan, email \u2026; GET /content-types): their recordings; repeat for several" + }, + { + "name": "emotion", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "null" + } + ], + "description": "emotions heard in them (ignoring case); repeat for several", + "title": "Emotion" + }, + "description": "emotions heard in them (ignoring case); repeat for several" + }, { "name": "collection", "in": "query", @@ -13032,6 +13137,48 @@ } } }, + "/api/v1/find": { + "post": { + "tags": [ + "search" + ], + "summary": "Find Resources", + "description": "Resources you can read that match every filter given: the people seen in them (faces) and heard (speakers), the\nentities they mention, the topics they're about, their kind and content type, when they were made, their tags,\nlanguage, objects seen, emotions heard, length, status, collection and origin, and words said or written in them.\nFaces, speakers, entities and topics can be named instead of numbered; `resolved` says what each name matched, and\na name that matches nothing is a 400 naming the closest ones. Each resource says `why` it matched, with up to three\nmoments per reason to cite (a time, or a page). With `text`, the best matching come first.", + "operationId": "find_resources", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FindQuery" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FindResults" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/searches": { "get": { "tags": [ @@ -37411,6 +37558,974 @@ "type": "object", "title": "FileUpdate" }, + "FindItem": { + "properties": { + "id": { + "type": "integer", + "title": "Id" + }, + "title": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Title" + }, + "recorded_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Recorded At" + }, + "duration_ms": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Duration Ms" + }, + "status": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Status" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Source" + }, + "space": { + "type": "integer", + "title": "Space" + }, + "namespace": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Namespace" + }, + "collection": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Collection", + "description": "the collection it lives in" + }, + "collection_name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Collection Name" + }, + "media_kind": { + "type": "string", + "title": "Media Kind", + "description": "audio, video, transcript (text without media), document or image" + }, + "pages": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Pages", + "description": "a document's or an image's pages" + }, + "poster": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Poster", + "description": "signed link to a video's first frame, or a document's or image's first page" + }, + "emotions": { + "additionalProperties": true, + "type": "object", + "title": "Emotions" + }, + "words": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Words" + }, + "importance": { + "title": "Importance" + }, + "sentiment": { + "title": "Sentiment" + }, + "speakers": { + "type": "string", + "title": "Speakers", + "description": "speaker names, comma separated", + "default": "" + }, + "access": { + "type": "string", + "enum": [ + "public", + "restricted", + "private" + ], + "title": "Access", + "description": "its own access, or its namespace's default", + "default": "private" + }, + "open": { + "items": { + "type": "string", + "enum": [ + "media", + "transcript", + "index" + ] + }, + "type": "array", + "title": "Open", + "description": "the parts anyone may use when it is public" + }, + "featured": { + "type": "boolean", + "title": "Featured", + "default": false + }, + "tags": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Tags" + }, + "language": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Language", + "description": "its language code, when known" + }, + "origin": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Origin", + "description": "where it came from: source: (a connected source), upload, paste, iiif, folder or file" + }, + "origin_name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Origin Name", + "description": "what to call its origin: the source's name, or e.g. Uploaded" + }, + "role": { + "anyOf": [ + { + "type": "string", + "enum": [ + "viewer", + "editor", + "owner" + ] + }, + { + "type": "null" + } + ], + "title": "Role", + "description": "your role on it: through its namespace or its collection (an admin of the collection is an owner)" + }, + "nsfw": { + "anyOf": [ + { + "$ref": "#/components/schemas/NsfwState" + }, + { + "type": "null" + } + ], + "description": "NSFW classification, where it's on in its namespace and it was classified" + }, + "why": { + "items": { + "$ref": "#/components/schemas/FindReason" + }, + "type": "array", + "title": "Why", + "description": "why it matched" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "space", + "media_kind" + ], + "title": "FindItem" + }, + "FindMoment": { + "properties": { + "t0": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "T0", + "description": "ms from the start, on a timeline" + }, + "t1": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "T1", + "description": "ms, where it ends" + }, + "page": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Page", + "description": "in a document or an image: the page (from 0)" + }, + "last_page": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Last Page", + "description": "a face over several pages: the last of them" + }, + "text": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Text", + "description": "the words there, when it has any" + }, + "speaker": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Speaker" + }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Source", + "description": "words: said, screen, file, page, object or described (as search has it)" + } + }, + "additionalProperties": true, + "type": "object", + "title": "FindMoment" + }, + "FindNamed": { + "properties": { + "id": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Id" + }, + "key": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Key" + }, + "name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Name" + }, + "namespace": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Namespace" + } + }, + "additionalProperties": true, + "type": "object", + "title": "FindNamed" + }, + "FindQuery": { + "properties": { + "namespace": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Namespace", + "description": "one namespace (default: every namespace you can read)" + }, + "text": { + "anyOf": [ + { + "type": "string", + "maxLength": 300 + }, + { + "type": "null" + } + ], + "title": "Text", + "description": "words said or written in them, as search takes them (words, \"phrases\", OR); by meaning too when that is set up" + }, + "title": { + "anyOf": [ + { + "type": "string", + "maxLength": 200 + }, + { + "type": "null" + } + ], + "title": "Title", + "description": "words that must all appear in the title, the namespace or a speaker's name" + }, + "faces": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "string" + } + ] + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Faces", + "description": "people seen in them: face identities, by id or name" + }, + "speakers": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "string" + } + ] + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Speakers", + "description": "voices heard in them: speakers, by id or name" + }, + "entities": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "string" + } + ] + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Entities", + "description": "entities they mention, by id or name" + }, + "topics": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "string" + } + ] + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Topics", + "description": "topics they were accepted as about, by id or label; a topic's narrower topics count too" + }, + "types": { + "anyOf": [ + { + "items": { + "type": "string", + "enum": [ + "audio", + "video", + "transcript", + "document", + "image" + ] + }, + "type": "array", + "maxItems": 5 + }, + { + "type": "null" + } + ], + "title": "Types", + "description": "their kind: audio, video, transcript, document or image" + }, + "content_types": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Content Types", + "description": "content types (podcast, interview, scan, photo, email \u2026, GET /content-types), by key or label" + }, + "date_from": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Date From", + "description": "made on or after this day, month or year (2026-09-30, 2026-09, 2026)" + }, + "date_to": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Date To", + "description": "made on or before this day, month or year" + }, + "min_duration": { + "anyOf": [ + { + "type": "number", + "minimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Min Duration", + "description": "at least this many seconds long" + }, + "max_duration": { + "anyOf": [ + { + "type": "number", + "minimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Max Duration", + "description": "shorter than this many seconds" + }, + "tags": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Tags", + "description": "tags (ignoring case)" + }, + "languages": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Languages", + "description": "language codes; none for those whose language isn't known" + }, + "objects": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Objects", + "description": "kinds of object seen in them (person, car, dog \u2026)" + }, + "emotions": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Emotions", + "description": "emotions heard in them (ignoring case)" + }, + "status": { + "anyOf": [ + { + "items": { + "type": "string", + "enum": [ + "new", + "transcribed", + "diarized", + "analyzed", + "error", + "processing", + "failed" + ] + }, + "type": "array", + "maxItems": 10 + }, + { + "type": "null" + } + ], + "title": "Status", + "description": "statuses, or processing and failed" + }, + "origins": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 20 + }, + { + "type": "null" + } + ], + "title": "Origins", + "description": "where they came from: source:, upload, paste, iiif, folder or file" + }, + "collection": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Collection", + "description": "a collection: what's in it and in the collections inside it" + }, + "sensitive": { + "anyOf": [ + { + "type": "string", + "enum": [ + "include", + "only", + "exclude" + ] + }, + { + "type": "null" + } + ], + "title": "Sensitive", + "description": "where NSFW classification is on: only those flagged, exclude them, or include them" + }, + "sort": { + "anyOf": [ + { + "type": "string", + "enum": [ + "relevance", + "-date", + "date", + "-title", + "title", + "-duration", + "duration" + ] + }, + { + "type": "null" + } + ], + "title": "Sort", + "description": "relevance (the default with text: best matching words first), else newest first" + }, + "limit": { + "type": "integer", + "maximum": 100.0, + "minimum": 1.0, + "title": "Limit", + "default": 20 + }, + "offset": { + "type": "integer", + "maximum": 10000.0, + "minimum": 0.0, + "title": "Offset", + "default": 0 + } + }, + "additionalProperties": false, + "type": "object", + "title": "FindQuery", + "description": "Filters combine with AND, the values of one filter with OR. Faces, speakers, entities and topics are ids or names\n(ignoring case; an entity's other names count, and any of a topic's labels)." + }, + "FindReason": { + "properties": { + "by": { + "type": "string", + "enum": [ + "words", + "face", + "voice", + "entity", + "topic", + "content_type", + "object", + "emotion" + ], + "title": "By" + }, + "name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Name", + "description": "what matched: the words, the person's or the entity's name, the topic's label \u2026" + }, + "moments": { + "items": { + "$ref": "#/components/schemas/FindMoment" + }, + "type": "array", + "title": "Moments", + "description": "up to 3 moments that show it, to cite" + }, + "count": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Count", + "description": "emotion: how many lines have it" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "by" + ], + "title": "FindReason" + }, + "FindResults": { + "properties": { + "total": { + "type": "integer", + "title": "Total", + "description": "how many resources match, on all pages" + }, + "items": { + "items": { + "$ref": "#/components/schemas/FindItem" + }, + "type": "array", + "title": "Items" + }, + "resolved": { + "additionalProperties": { + "anyOf": [ + { + "items": { + "$ref": "#/components/schemas/FindNamed" + }, + "type": "array" + }, + { + "type": "integer" + } + ] + }, + "type": "object", + "title": "Resolved", + "description": "what the names given matched, by filter (and how many narrower topics came with them)" + }, + "text": { + "anyOf": [ + { + "$ref": "#/components/schemas/FindText" + }, + { + "type": "null" + } + ] + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "total", + "items" + ], + "title": "FindResults" + }, + "FindText": { + "properties": { + "mode": { + "type": "string", + "title": "Mode", + "description": "how the words were matched: keyword, semantic or hybrid" + }, + "meaning": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Meaning", + "description": "why not by meaning, when it was asked and couldn't be" + }, + "moments": { + "type": "integer", + "title": "Moments", + "description": "moments the words were found in" + }, + "capped": { + "type": "boolean", + "title": "Capped", + "description": "more moments matched than were looked through, so some resources may be missing" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "mode", + "moments", + "capped" + ], + "title": "FindText" + }, "ForgotPasswordRequest": { "properties": { "email": { From e6aa9c60644a9a889ed9df0ed3719939de5de3ca Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 17 Oct 2026 02:41:09 +0000 Subject: [PATCH 2/2] docs(changelog): find is unreleased Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N --- CHANGELOG.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9df017157..912cf7288 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version registries or the public web. Files attached to the conversation go in; a file it makes reaches a namespace once the person approves. Off by default; starting the gateway from Settings is in progress ([Code sandbox](docs/assistant.md#code-sandbox)). +- **Find anything by who, what and when.** Agents and the API can find assets by the faces seen in them, the people + heard, the entities they mention, the topics they're about, their kind and content type, when they were made (a + day, a month or a year), tags, language, objects seen, emotions heard, length and words in them, together or alone, + naming people and things instead of looking up ids. Each result says why it matched, with moments to cite + (`POST /find`, the MCP tool `find`, the assistant's `find_resources`; [Finding by any + attribute](docs/processing.md#finding-by-any-attribute)). The Library list takes faces, entities, topics, content + types and emotions as filters too. ## 0.13.0 October 16, 2026 {id="0.13.0"} @@ -170,13 +177,6 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version database), are left out, and a file Lens can't read is left out with a warning instead of stopping the backup. When backing up stops an upgrade, Settings now marks *Back up the database* as the step that failed ([Backups](docs/database.md#backups)). -- **Find anything by who, what and when.** Agents and the API can find assets by the faces seen in them, the people - heard, the entities they mention, the topics they're about, their kind and content type, when they were made (a - day, a month or a year), tags, language, objects seen, emotions heard, length and words in them, together or alone, - naming people and things instead of looking up ids. Each result says why it matched, with moments to cite - (`POST /find`, the MCP tool `find`, the assistant's `find_resources`; [Finding by any - attribute](docs/processing.md#finding-by-any-attribute)). The Library list takes faces, entities, topics, content - types and emotions as filters too. - **The Ask button no longer covers buttons.** The floating Ask button at the bottom right used to sit on whatever was there, such as Import on the Import page. Now, when a button or field is under it and scrolling can't move it clear (a row of buttons pinned to the bottom of the screen, or the last button on a page scrolled to its end), the Ask