From bdefed2b2f97aebde91b332b344ba61af19b20e3 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 10 Oct 2026 20:23:50 +0000 Subject: [PATCH] feat(search): find content similar to a text, a resource, a topic or an entity GET /similar, and find_similar for the assistant and MCP clients: the resources most like what was given, fused by rank from passages close in meaning (when search by meaning is set up), shared key words, shared topics and shared entities, each citing the moments that show it. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01NSG4xbWi1TYHFmWSUh1CSt --- DEVLOG.md | 23 + docs/api.md | 11 + docs/assistant.md | 3 +- docs/mcp.md | 1 + docs/processing.md | 19 + fastapi_backend/app/api/mcp_tools.py | 79 +++ fastapi_backend/app/api/v1/routes/search.py | 47 +- fastapi_backend/app/domain/ai_tools.py | 54 ++ fastapi_backend/app/domain/semantic.py | 21 +- fastapi_backend/app/domain/similar.py | 393 ++++++++++++++ fastapi_backend/app/schemas/search.py | 58 +++ fastapi_backend/tests/api/test_mcp.py | 9 + fastapi_backend/tests/api/test_similar.py | 180 +++++++ nextjs-frontend/app/openapi-client/index.ts | 10 + nextjs-frontend/app/openapi-client/sdk.gen.ts | 20 + .../app/openapi-client/types.gen.ts | 273 ++++++++++ nextjs-frontend/openapi.json | 483 ++++++++++++++++++ 17 files changed, 1680 insertions(+), 4 deletions(-) create mode 100644 fastapi_backend/app/domain/similar.py create mode 100644 fastapi_backend/tests/api/test_similar.py diff --git a/DEVLOG.md b/DEVLOG.md index e0479cb62..e402fcb81 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -2,6 +2,29 @@ Plans and progress for work in flight. Newest first. +## 2026-10-10 · Similar content + +Goal (Adeel): a Similar tab on every resource, and selecting text anywhere (a resource's text or title, notes, topics, +entities, the graph canvas) shows similar content in a popover, with a way to find more. + +Model (fastapi_backend/app/domain/similar.py, docs/processing.md#similar-content): one list for a text, a resource, a +topic or an entity, fused by reciprocal rank from four: meaning (passages near the text, or near the centre of a +resource's passages: no model call for a resource), key words (the text's words the `term` table says are rare, or +`analyze.keywords`, OR'ed through the word index), shared topics and shared entities (rarer ones weigh more). Each item +cites up to three moments. `GET /similar`, assistant and MCP `find_similar`. Without embeddings (the default) the +other three answer, with no model call. + +PRs: API, assistant and MCP tool; the Similar tab; the selection popover and the full Similar page; topics, entities and +graph nodes. + +Refine later: +- key words of a text are single words and two-word phrases; stems beyond plurals (designing/design) don't match; +- a resource not indexed for meaning is compared by its title and keywords (one embedding request), not its text; +- the centre of a long resource's passages blurs what its parts are about; comparing a few of its passages may be + sharper; +- entities in another namespace with the same name count only through the words, not as the same entity; +- no "not similar" feedback yet, and nothing is cached between requests beyond the query vectors. + ## 2026-10-10 · Podcasts from picked knowledge Goal (Adeel): pick documents, search results, topics, notes or graph concepts and get a two-host, Recast-style diff --git a/docs/api.md b/docs/api.md index c0c6afdbc..14f8076b5 100644 --- a/docs/api.md +++ b/docs/api.md @@ -838,6 +838,7 @@ recording later leaves its uploaded file in place, like any other media file. ``` GET /api/v1/search GET /api/v1/search/terms +GET /api/v1/similar GET /api/v1/graph GET /api/v1/mentions ``` @@ -867,6 +868,16 @@ when search by meaning is set up and the query has no "phrases" or OR, else keyw matched, `semantic` whether search by meaning is set up, and `meaning` why it wasn't used when asked for. Each hit's `match` is `words`, `meaning` (with its `similarity`, cosine, and shown at the passage's best line) or `both`. +`GET /similar` lists the resources most like one thing, best first, in the namespaces you can read and the +collections you were given a role on ([similar content](processing.md#similar-content)): a `text` (what was selected, +up to 2,000 characters; `exclude` leaves out the resource it was selected in), a `recording`, a `topic` or an +`entity`; give exactly one (422 otherwise, 404 for one you can't read). `limit` (default 10, at most 50) and `offset` +page through them. Each item says why it's alike in `match` (`meaning`, `words`, `topics`, `entities`), with its +`similarity` by meaning, up to three `cites` (moments to open: `source`, `idx`, `t0` or `page`, and a `snippet` with the +key words marked), the key `words` it says, and the `topics` and `entities` in common. The reply's `mode` is `meaning` +when an embedding model compared them too, else `words`; `terms` are the key words looked for and `meaning` says why +it wasn't by meaning when search by meaning is set up. A resource is never listed as like itself. + Search has no prefix search (`interp*` looks for the word "interp"). `GET /search/terms?prefix=interp` lists whole words said in the namespaces you can read (`ns` for one) that start with it, the most said first, with how often and in how many recordings (`limit`, default 8, at most 20); the web app offers them as "Try …". diff --git a/docs/assistant.md b/docs/assistant.md index b7ab32fd8..415b4058d 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -91,7 +91,8 @@ the browser should read it). When the configured model supports function calling, chat becomes an agent. - **What it can look things up with:** transcripts (including on-screen text), recordings, transcript passages, - outputs, entities and their mentions and timelines, the knowledge graph, and speakers. + outputs, entities and their mentions and timelines, the knowledge graph, speakers, and content similar to a text, + a recording, a topic or an entity (`find_similar`). - **Limits:** everything is limited to what the person can read and to the conversation's scope, and capped by `ai.max_steps` and `ai.max_transcript_reads`. - **Citations:** every moment a tool returns is numbered, so answers cite [n] across everything the assistant read. diff --git a/docs/mcp.md b/docs/mcp.md index 5a8b2b154..b72006502 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -38,6 +38,7 @@ front of the web app also needs `TRUST_PROXY_HEADERS=true` ([Authentication](aut | Tool | | |---|---| | `search` | moments where words are said, shown on screen or written, best first: the recording, speaker, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, speaker or recording | +| `find_similar` | the recordings, documents and images most like a text, a recording, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links | | `list_namespaces` | the namespaces you can read, with their recordings, hours, speakers and your role | | `list_recordings` | recordings, newest first, by namespace, title words, dates, speaker, tag or kind | | `list_speakers` | the speakers of your namespaces, the ones who talk most first | diff --git a/docs/processing.md b/docs/processing.md index bcfbe7a15..eeeed2539 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -216,6 +216,25 @@ Settings → Search shows how far it has got and can queue more now; `lens embed embeddings server can't be reached, the step is skipped (the routine tries again later) and searches go by the words, saying why. +### Similar content + +The API (`GET /similar`), the assistant and the MCP server (`find_similar`) list the resources most like a text (a +sentence, a passage, a title), a resource, a topic or an entity. + +Four lists are fused by reciprocal rank, so a resource alike in several ways comes first: + +* **meaning**, when search by meaning is set up: the passages closest to the text (a topic's labels and definition, + an entity's name and description), or for a resource to the centre of its own passages, so no model call is needed; +* **key words**: the text's words that the archive says in fewest recordings (from the analysis step's term counts), + or a resource's keywords, any of them, through the word index: resources saying more of them first; +* **topics**: resources about the same topics, a topic fewer resources are about counting for more; +* **entities**: resources mentioning the same people, organisations, places and terms, the rarer counting for more. + +Each result cites why: its closest passages and the lines with the shared words (each opens at that moment or page), +the words, and the topics and entities in common. Without an embeddings server (the default) the last three still +answer, and no language model is called; with one, a selected text costs one embedding request, logged in the +activity ledger. + ## Background work Imports, pipeline runs and folder scans return at once and run as jobs stored in SurrealDB. The server runs diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 8015a11fb..ecdb3b9a8 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -364,6 +364,85 @@ def search( return out +@tool( + "find_similar", + "Find similar content", + "The recordings, documents and images most like a text (a sentence, a passage, a title), a recording, a topic or " + "an entity: give exactly one. Best first, each with why it's alike: `match` (meaning: close in meaning, when an " + "embedding model is set up; words: shares key words; topics: about the same topics; entities: mentions the same " + "people, organisations, places...), the moments that show it (text with the key words in **bold**, and a url that " + "opens it there), and the words, topics and entities in common. A recording is never listed as like itself.", + Arg("text", "string", "what to find more like: a sentence, a passage, a title", max_length=2000), + Arg("recording_id", "integer", "a recording: the others most like it"), + Arg("topic_id", "integer", "a topic (from list_topics): what's most like its labels and definition, and about it"), + Arg("entity_id", "integer", "an entity (from list_entities): what's most like its name and description, and mentions it"), + Arg("exclude_recording_id", "integer", "leave this recording out too (the one the text is from)"), + Arg("limit", "integer", "how many", default=10, minimum=1, maximum=50), + Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000), +) +def find_similar( + ctx: Context, + text: str | None, + recording_id: int | None, + topic_id: int | None, + entity_id: int | None, + exclude_recording_id: int | None, + limit: int, + offset: int, +) -> dict[str, Any]: + if sum(x is not None for x in (text, recording_id, topic_id, entity_id)) != 1: + raise ToolError("give one of text, recording_id, topic_id or entity_id") + found = search_routes.find_similar( + ctx.acl, + ctx.user, + ctx.db, + ctx.cfg, + text=text, + recording=recording_id, + topic=topic_id, + entity=entity_id, + exclude=exclude_recording_id, + limit=limit, + offset=offset, + ) + results = [] + for x in found.items: + moments = [] + for c in x.cites: + m: dict[str, Any] = { + "found": c.match, + "at": store.tc(c.t0) if c.t0 is not None and c.page is None else None, + "seconds": _seconds(c.t0) if c.page is None else None, + "line": c.idx, + "page": c.page + 1 if c.page is not None else None, + "similarity": c.similarity, + "text": _plain(c.snippet), + "url": ctx.link(x.recording_id, c.t0 if c.page is None else None) + (f"?page={c.page}" if c.page is not None else ""), + } + moments.append({k: v for k, v in m.items() if v is not None}) + item: dict[str, Any] = { + "recording_id": x.recording_id, + "title": x.title, + "namespace": x.namespace, + "recorded_at": x.recorded_at, + "media": x.media_kind, + "match": x.match, + "similarity": x.similarity, + "moments": moments, + "words": x.words or None, + "topics": [{"id": t.id, "label": t.label} for t in x.topics] or None, + "entities": [{"id": e.id, "name": e.name} for e in x.entities] or None, + "url": ctx.link(x.recording_id), + } + results.append({k: v for k, v in item.items() if v is not None}) + out: dict[str, Any] = {"like": found.label, "by": found.mode, "key_words": found.terms, "total": found.total, "results": results} + if found.meaning: + out["note"] = found.meaning + if offset + len(results) < found.total: + out["next_offset"] = offset + len(results) + return out + + # ---------- browsing ---------- @tool( "list_namespaces", diff --git a/fastapi_backend/app/api/v1/routes/search.py b/fastapi_backend/app/api/v1/routes/search.py index 6973fadaf..47b1ce016 100644 --- a/fastapi_backend/app/api/v1/routes/search.py +++ b/fastapi_backend/app/api/v1/routes/search.py @@ -9,9 +9,9 @@ from app.api.deps import Acl, Cfg, CurrentUser, Db from app.api.media import sign_urls from app.domain import graph as graphmod -from app.domain import graph_history, hierarchy, photos, render, store +from app.domain import graph_history, hierarchy, photos, render, similar, store from app.domain import search as searchmod -from app.schemas.search import Graph, Mention, PhotoResults, SearchResults, TermSuggestion +from app.schemas.search import Graph, Mention, PhotoResults, SearchResults, SimilarResults, TermSuggestion router = APIRouter(tags=["search"]) @@ -117,6 +117,49 @@ def search_photos( return PhotoResults.model_validate(sign_urls({"hits": out}, full=True)) +@router.get("/similar") +def find_similar( + acl: Acl, + user: CurrentUser, + db: Db, + cfg: Cfg, + text: str | None = Query(None, max_length=similar.TEXT_MAX, description="what was selected: a sentence, a passage, a title"), + recording: int | None = Query(None, description="a resource: the others most like it"), + topic: int | None = Query(None, description="a topic: the resources most like its labels and definition, and about it"), + entity: int | None = Query(None, description="an entity: the resources most like its name and description, and mentioning it"), + exclude: int | None = Query(None, description="leave out this resource too (the one the text was selected in)"), + limit: int = Query(10, ge=1, le=similar.LIMIT_MAX), + offset: int = Query(0, ge=0), +) -> SimilarResults: + """The resources most like a text, a resource, a topic or an entity (give one), best first, in the namespaces you + can read and the collections you were given a role on, each with why: its closest passages by meaning (when + search by meaning is set up), the lines with the key words it shares, and the topics and entities in common. + Each cited moment can be opened. A resource is never listed as like itself.""" + given = [x for x in (text and text.strip(), recording, topic, entity) if x not in (None, "")] + if len(given) != 1: + raise HTTPException(422, "give one of text, recording, topic or entity") + if recording is not None: + acl.recording(recording) # 404 unless they may read it + spaces = set(acl.roles) + try: + res = similar.similar( + db, + cfg, + text=text, + recording=recording, + topic=topic, + entity=entity, + spaces=spaces, + also=acl.partial_recordings(), + exclude=exclude, + limit=limit, + offset=offset, + ) + except KeyError: + raise HTTPException(404, "not found") from None + return SimilarResults.model_validate(res) + + @router.get("/search/terms") def suggest_terms( acl: Acl, diff --git a/fastapi_backend/app/domain/ai_tools.py b/fastapi_backend/app/domain/ai_tools.py index c41b2f824..cf7aea864 100644 --- a/fastapi_backend/app/domain/ai_tools.py +++ b/fastapi_backend/app/domain/ai_tools.py @@ -6,6 +6,7 @@ from __future__ import annotations +import html import json import logging @@ -25,6 +26,7 @@ recsets, render, search as searchmod, + similar, speakers as spk, store, templates, @@ -43,6 +45,15 @@ ["query"], False, ), + ( + "find_similar", + "Find the recordings, documents and images in scope most like a text, a recording, a topic or an entity (give " + "one): close in meaning (when search by meaning is set up), sharing key words, about the same topics, mentioning " + "the same entities; each with the moments that show it.", + {"text": _S, "recording_id": _I, "topic_id": _I, "entity_id": _I, "limit": _I}, + [], + False, + ), ( "list_recordings", "List recordings in scope, newest first, optionally by namespace, dates (YYYY-MM-DD), speaker, entity or media.", @@ -602,6 +613,49 @@ def t_search_transcripts(self, query, namespace=None, limit=8): ) return {"total": res["total"], "results": out}, f'Searched for "{query}": {res["total"]} match(es)' + def t_find_similar(self, text=None, recording_id=None, topic_id=None, entity_id=None, limit=5): + if recording_id is not None: + self._ok(recording_id) + try: + res = similar.similar( + self.db, + self.cfg, + text=text, + recording=recording_id, + topic=topic_id, + entity=entity_id, + spaces=self.readable, + within=self.allowed, + limit=min(int(limit or 5), 20), + ) + except KeyError: + raise ValueError("that topic or entity isn't available in this conversation") from None + out = [] + for x in res["items"]: + moments = [] + for c in x["cites"]: + said = html.unescape(c["snippet"].replace("", "").replace("", "")) + n = self.ref(x["recording_id"], c["t0"] or 0, said, title=x["title"], source=c["source"], page=c["page"]) + moments.append( + {"ref": n, **({"page": c["page"] + 1} if c["page"] is not None else {"time": store.tc(c["t0"] or 0)}), "text": said} + ) + out.append( + store.clean( + { + "recording_id": x["recording_id"], + "title": x["title"], + "namespace": x["namespace"], + "alike_by": x["match"], + "moments": moments, + "words": x["words"] or None, + "topics": [t["label"] for t in x["topics"]] or None, + "entities": [e["name"] for e in x["entities"]] or None, + } + ) + ) + like = res["label"] or "" + return {"like": like, "by": res["mode"], "total": res["total"], "results": out}, f'Found {res["total"]} like "{like[:60]}"' + def t_list_recordings(self, namespace=None, limit=20, speaker_id=None, entity_id=None, media=None, **dates): f = recsets.clean_filter( { diff --git a/fastapi_backend/app/domain/semantic.py b/fastapi_backend/app/domain/semantic.py index 3412c8a00..d4dbdab2c 100644 --- a/fastapi_backend/app/domain/semantic.py +++ b/fastapi_backend/app/domain/semantic.py @@ -438,7 +438,13 @@ def nearest(db, cfg, q, where_f, params, kinds, k): text = query_text(q) if not text or not kinds: return [] - vec = query_vector(cfg, text) + return nearest_to(db, cfg, query_vector(cfg, text), where_f, params, kinds, k) + + +def nearest_to(db, cfg, vec, where_f, params, kinds, k): + """nearest() for a vector: a query's, or the centre of a recording's passages (`centre`).""" + if not kinds: + return [] dim = (state(db) or {}).get("dimension") if dim and len(vec) != dim: raise EmbedError(f"the model's vectors now have {len(vec)} dimensions, the index {dim}: index again (lens embed --force)") @@ -469,3 +475,16 @@ def nearest(db, cfg, q, where_f, params, kinds, k): rows = [r for r in rows if r.get("similarity") is not None] least = max([floor(cfg)] + [r["similarity"] - SPREAD for r in rows[:1]]) return [r for r in rows if r["similarity"] >= least] + + +def centre(db, cfg, rid, kinds=("said", "page", "described")): + """The mean of a recording's passage vectors (what it is about as a whole), or None when it has none from the + configured model yet.""" + if not available(db, cfg): + return None + rows = db.rows("SELECT embedding FROM passage WHERE recording = $r AND kind IN $k AND embedding != NONE", r=int(rid), k=list(kinds)) + vecs = [r["embedding"] for r in rows if r.get("embedding")] + if not vecs or len({len(v) for v in vecs}) != 1: + return None + n = len(vecs) + return [sum(col) / n for col in zip(*vecs)] diff --git a/fastapi_backend/app/domain/similar.py b/fastapi_backend/app/domain/similar.py new file mode 100644 index 000000000..7f623eb47 --- /dev/null +++ b/fastapi_backend/app/domain/similar.py @@ -0,0 +1,393 @@ +"""Similar content: the resources most like a text, a resource, a topic or an entity, each with why it is alike. + +Four lists, fused by reciprocal rank (as search.py fuses words and meaning): + +- meaning: the passages an embedding model finds closest (semantic.py) to the text, or to the centre of a resource's + own passages. Only when search by meaning is set up and indexed, so the default install (no embeddings server) + goes by the other three; +- words: the text's key words (the ones the archive says least often first, from the `term` table), or a resource's + keywords (analyze.keywords), any of them, through the word index (textindex.py): resources sharing more of them, + and saying them more, first; +- topics: resources about the same topics (`topic_about`), a topic fewer resources are about counting for more; +- entities: resources mentioning the same people, organisations, places... (`mentions`), the rarer counting for more. + +Every result cites why: its closest passages or the lines with the shared words (each a moment to open), the words, +and the topics and entities in common. Nothing here calls a language model; the one model call is embedding a text +(semantic.query_vector remembers it), which goes through the activity ledger like every other. +""" + +from __future__ import annotations + +import math +import re +from collections import Counter + +from . import analyze, render, semantic, store, textindex +from . import entities as ents +from . import topics as topicmod +from .search import M0, M1, _mark_plain, snippet + +R = store.R +LIMIT_MAX = 50 +TEXT_MAX = 2000 # the most of a text compared (an embedding model reads about this much) +TERMS = 12 # key words looked for +CITES = 3 # moments cited per resource +ENTITIES_MAX = 25 # a resource's most mentioned entities compared +RRF = 60 +KINDS = {"said", "page", "described"} +SNIPPET = 220 +BASES = ("text", "recording", "topic", "entity") + + +def similar( + db, cfg, *, text=None, recording=None, topic=None, entity=None, spaces=(), also=(), within=None, exclude=None, limit=10, offset=0 +): + """The resources in namespaces `spaces` (and recordings `also`) most like one of `text`, `recording`, `topic` or + `entity`, best first: {basis, label, text, terms, mode, meaning, semantic, total, items}. `within` keeps to a set of + recordings (a conversation's scope). A resource is never like itself, and `exclude` leaves out one more (the + resource a text was selected in). KeyError when the + recording, topic or entity isn't one these namespaces hold; ValueError unless exactly one of them is given.""" + given = [k for k, v in (("text", text), ("recording", recording), ("topic", topic), ("entity", entity)) if v not in (None, "")] + if len(given) != 1: + raise ValueError("give one of text, recording, topic or entity") + spaces, also = {int(s) for s in spaces}, {int(r) for r in also or ()} + b = _basis(db, cfg, given[0], text, recording, topic, entity, spaces) + can = semantic.available(db, cfg) + out = { + "basis": b["basis"], + "label": b["label"], + "text": b["text"], + "terms": b["terms"], + "mode": "words", + "meaning": None, + "semantic": can, + "total": 0, + "items": [], + } + if (not spaces and not also) or (within is not None and not within): + return out + skip = {int(x) for x in (b.get("recording"), exclude) if x not in (None, "")} + where, params = _where(spaces, also, skip, within) + limit, offset = max(1, min(int(limit), LIMIT_MAX)), max(0, int(offset)) + need = offset + limit + lists = {} + if can and (b["vector"] or b["text"]): + try: + lists["meaning"] = _by_meaning(db, cfg, b, where, params, need) + out["mode"] = "meaning" + except semantic.EmbedError: + out["meaning"] = "search by meaning is unavailable right now, so these are alike by their words, topics and entities" + elif semantic.configured(cfg): + out["meaning"] = "nothing is indexed for search by meaning with this model yet" + if b["terms"]: + lists["words"] = _by_words(db, b["terms"], where, params, need) + if b["topics"]: + lists["topics"] = _by_topics(db, b["topics"], where, params) + if b["entities"]: + lists["entities"] = _by_entities(db, b["entities"], where, params) + fused = _fuse(lists) + out["total"] = len(fused) + out["items"] = _items(db, fused[offset : offset + limit]) + return out + + +# ---------- what to compare with ---------- +def _basis(db, cfg, kind, text, rid, tid, eid, spaces): + """What the four lists look for: {basis, label, text (for meaning), vector, terms (words), topics, entities, + recording (left out of the results)}.""" + b = {"basis": kind, "label": None, "text": None, "vector": None, "terms": [], "topics": {}, "entities": {}, "recording": None} + if kind == "text": + t = " ".join(str(text).split())[:TEXT_MAX] + b.update(label=t[:120], text=t, terms=key_terms(db, t, spaces)) + return b + if kind == "recording": + rec = db.one("SELECT space, title FROM $r", r=R("recording", int(rid))) + if not rec: + raise KeyError(rid) + rid = int(rid) + terms = [w for w, _ in analyze.keywords(db, rid, top=TERMS)] + b.update( + label=rec.get("title"), + recording=rid, + terms=terms, + topics={t["id"]: t["label"] for t in topicmod.of_recording(db, rid)}, + entities=_entities_of(db, rid), + ) + b["vector"] = semantic.centre(db, cfg, rid) + if b["vector"] is None: # not indexed (yet): what it's called and its keywords stand for it + b["text"] = " ".join([rec.get("title") or "", *terms]).strip() or None + return b + if kind == "topic": + t = topicmod._topic(db, int(tid)) + if t["space"] not in spaces: + raise KeyError(tid) + labels = [t["label"], *(t.get("alt") or [])] + b.update( + label=t["label"], + text=". ".join(x for x in [", ".join(labels), t.get("definition") or ""] if x)[:TEXT_MAX], + terms=_dedupe(labels)[:TERMS], + topics={t["id"]: t["label"]}, + ) + return b + e = ents._entity(db, int(eid)) + if e["space"] not in spaces: + raise KeyError(eid) + names = _dedupe([e["name"], *ents.aliases(db, [e["id"]]).get(e["id"], [])]) + b.update( + label=e["name"], + text=". ".join(x for x in [e["name"], e.get("description") or ""] if x)[:TEXT_MAX], + terms=names[:TERMS], + entities={e["id"]: e["name"]}, + ) + return b + + +def key_terms(db, text, spaces, n=TERMS): + """The words (and two-word phrases) that say most about `text`: the ones it repeats and the archive says in few + recordings first, as analyze.keywords weighs a recording's. Words the namespaces' recordings never say are left + out, since nothing could match them; when nothing is analysed yet, the ones it repeats and the longer ones.""" + ws = analyze.words(text) + tf, surface = Counter(), {} + for st, sf in ws: + tf[st] += 1 + surface.setdefault(st, sf) + for (a, sa), (b2, sb) in zip(ws, ws[1:]): + tf[a + " " + b2] += 1 + surface.setdefault(a + " " + b2, sa + " " + sb) + if not tf: + return [] + df = ( + { + r["term"]: r["df"] + for r in db.rows( + "SELECT term, count() AS df FROM term WHERE space IN $s AND term IN $t GROUP BY term", s=sorted(spaces), t=list(tf) + ) + } + if spaces + else {} + ) + if df: + n_recs = int((db.one("SELECT count() AS n FROM recording WHERE space IN $s GROUP ALL", s=sorted(spaces)) or {}).get("n") or 1) + scored = [ + (surface[t], (1 + math.log(k)) * (math.log((n_recs + 1) / (df[t] + 1)) + 1) * (1.3 if " " in t else 1)) + for t, k in tf.items() + if t in df + ] + else: + scored = [(surface[t], k * (1 + min(len(t), 12) / 6)) for t, k in tf.items() if " " not in t] + scored.sort(key=lambda x: -x[1]) + out: list[str] = [] + for word, _ in scored: # a word inside a phrase already chosen adds little + if word in out or (" " not in word and any(word in o.split() for o in out)): + continue + out.append(word) + if len(out) == n: + break + return out + + +def _entities_of(db, rid): + """A recording's most mentioned entities, {id: name}: not hidden, and not dates or numbers.""" + counts = db.rows("SELECT entity, count() AS n FROM mentions WHERE recording = $r GROUP BY entity", r=rid) + if not counts: + return {} + n = {c["entity"]: c["n"] for c in counts} + rows = db.rows( + "SELECT record::id(id) AS id, name, type, hidden FROM entity WHERE id IN $ids", ids=[R("entity", e) for e in n if e is not None] + ) + keep = [r for r in rows if not r.get("hidden") and r.get("type") not in ents.QUIET] + keep.sort(key=lambda r: (-n.get(r["id"], 0), r["name"].lower())) + return {r["id"]: r["name"] for r in keep[:ENTITIES_MAX]} + + +def _dedupe(labels): + seen, out = set(), [] + for x in labels: + x = " ".join(str(x or "").split()) + if x and x.lower() not in seen: + seen.add(x.lower()) + out.append(x) + return out + + +def _where(spaces, also, skip, within=None): + """The filter (" AND ...") every table here is read through: the namespaces and recordings that may be read, + less the ones left out. Passages, lines, topic_about and mentions all carry `space` and `recording`.""" + params = {"allowed": sorted(spaces)} + where = " AND (space IN $allowed OR recording IN $also)" if also else " AND space IN $allowed" + if also: + params["also"] = sorted(also) + if within is not None: + where += " AND recording IN $within" + params["within"] = sorted(int(r) for r in within) + if skip: + where += " AND recording NOT IN $skip" + params["skip"] = sorted(skip) + return where, params + + +# ---------- the four lists: [(recording, {cites, words, topics, entities, similarity})], best first ---------- +def _by_meaning(db, cfg, b, where, params, need): + k = min(500, max(int(semantic._section(cfg).get("neighbours") or 40), need * 8)) + if b["vector"] is not None: + found = semantic.nearest_to(db, cfg, b["vector"], where, params, KINDS, k) + else: + found = semantic.nearest(db, cfg, b["text"], where, params, KINDS, k) + marks = [t for t in b["terms"] if " " not in t] + by: dict = {} + for p in found: # most alike first + x = by.setdefault(p["recording"], {"cites": [], "similarity": round(p["similarity"], 3)}) + if len(x["cites"]) < CITES: + x["cites"].append(_passage(p, marks)) + return list(by.items()) + + +def _passage(p, marks): + text = p.get("text") or "" + marked = _mark_plain(text, marks) if marks else text + cite = {"match": "meaning", "similarity": round(p["similarity"], 3), "snippet": _snip(marked)} + if p["kind"] == "described": # a shot's time, or on a document its page (kept in t0) + paged = bool(p.get("paged")) + return {**cite, "source": "described", "idx": None, "t0": None if paged else p.get("t0"), "page": p.get("t0") if paged else None} + return {**cite, "source": p["kind"], "idx": p.get("idx0"), "t0": p.get("t0"), "page": p.get("page")} + + +def _by_words(db, terms, where, params, need): + groups = [{"words": [], "phrases": [t]} if " " in t else {"words": [t], "phrases": []} for t in terms] + fields = "record::id(id) AS id, recording, idx, t0, page, text" + cap = min(2000, max(200, need * 20)) + rows = textindex.rows(db, "segment", groups, fields, where, params, cap, (M0, M1)) + if rows is None: # no word index here (search.engine "surrealdb", or it's down): read the lines + words = {f"w{j}": t.lower() for j, t in enumerate(terms)} + cond = " OR ".join(f"string::contains(string::lowercase(text), ${k})" for k in words) + rows = db.rows(f"SELECT {fields} FROM segment WHERE ({cond}){where} LIMIT {cap}", **params, **words) + for r in rows: + r["h1"] = _mark_plain(r["text"], terms) + r["s1"] = r["h1"].count(M0) + by: dict = {} + for r in rows: + said = {m.lower() for m in re.findall(f"{M0}(.*?){M1}", r.get("h1") or "")} + x = by.setdefault(r["recording"], {"lines": [], "words": set()}) + x["lines"].append(r) + x["words"] |= said + for x in by.values(): + x["lines"].sort(key=lambda r: -(r.get("s1") or 0)) + ranked = sorted(by.items(), key=lambda kv: (-len(kv[1]["words"]), -sum(r.get("s1") or 0 for r in kv[1]["lines"][:CITES]), kv[0])) + return [ + ( + rid, + { + "cites": [ + { + "match": "words", + "similarity": None, + "snippet": snippet(r.get("h1") or r["text"], SNIPPET // 2), + "source": "said" if r.get("page") is None else "page", + "idx": r.get("idx"), + "t0": r.get("t0"), + "page": r.get("page"), + } + for r in x["lines"][:CITES] + ], + "words": sorted(x["words"])[:8], + }, + ) + for rid, x in ranked + ] + + +def _by_topics(db, tids, where, params): + rows = db.rows( + f"SELECT recording, topic, weight FROM topic_about WHERE topic IN $t AND status != 'dismissed'{where}", t=sorted(tids), **params + ) + df = Counter(r["topic"] for r in rows) + by: dict = {} + for r in rows: + x = by.setdefault(r["recording"], {"score": 0.0, "topics": []}) + x["score"] += 1 / math.log2(1 + df[r["topic"]]) + x["topics"].append({"id": r["topic"], "label": tids[r["topic"]]}) + ranked = sorted(by.items(), key=lambda kv: (-kv[1]["score"], kv[0])) + return [(rid, {"topics": sorted(x["topics"], key=lambda t: t["label"].lower())}) for rid, x in ranked] + + +def _by_entities(db, eids, where, params): + rows = db.rows( + f"SELECT entity, recording, count() AS n FROM mentions WHERE entity IN $e{where} GROUP BY entity, recording", + e=sorted(eids), + **params, + ) + df = Counter(r["entity"] for r in rows) + by: dict = {} + for r in rows: + x = by.setdefault(r["recording"], {"score": 0.0, "n": 0, "entities": []}) + x["score"] += 1 / math.log2(1 + df[r["entity"]]) + x["n"] += r["n"] + x["entities"].append({"id": r["entity"], "name": eids[r["entity"]]}) + ranked = sorted(by.items(), key=lambda kv: (-kv[1]["score"], -kv[1]["n"], kv[0])) + return [(rid, {"entities": sorted(x["entities"], key=lambda e: e["name"].lower())}) for rid, x in ranked] + + +# ---------- one list ---------- +def _fuse(lists): + """The lists as one, best first: a resource's score is the sum of 1 / (RRF + its rank) in each list it's in.""" + acc: dict = {} + for name in ("meaning", "words", "topics", "entities"): + for i, (rid, d) in enumerate(lists.get(name) or []): + x = acc.setdefault( + rid, + {"recording": rid, "score": 0.0, "match": [], "similarity": None, "cites": [], "words": [], "topics": [], "entities": []}, + ) + x["score"] += 1 / (RRF + i + 1) + x["match"].append(name) + if d.get("similarity") is not None: + x["similarity"] = d["similarity"] + for c in d.get("cites") or []: # the closest passages first, then the lines with the words; each moment once + if len(x["cites"]) < CITES and not any( + (c["source"], c["idx"], c["page"], c["t0"]) == (o["source"], o["idx"], o["page"], o["t0"]) for o in x["cites"] + ): + x["cites"].append(c) + for k in ("words", "topics", "entities"): + x[k] += d.get(k) or [] + return sorted(acc.values(), key=lambda x: (-x["score"], x["recording"])) + + +def _items(db, page): + if not page: + return [] + recs = { + r["id"]: r + for r in db.rows( + "SELECT record::id(id) AS id, title, recorded_at, space, source, media FROM recording WHERE id IN $ids", + ids=[R("recording", x["recording"]) for x in page], + ) + } + names = store.space_names(db) + out = [] + for x in page: + r = recs.get(x["recording"]) + if not r: # deleted since its passages or lines were read + continue + out.append( + { + "recording_id": x["recording"], + "title": r.get("title"), + "recorded_at": r.get("recorded_at"), + "namespace": names.get(r["space"]), + "media_kind": render.kind(r), + "score": round(x["score"], 4), + "match": x["match"], + "similarity": x["similarity"], + "cites": x["cites"], + "words": x["words"], + "topics": x["topics"], + "entities": x["entities"], + } + ) + return out + + +def _snip(marked): + """A passage as a snippet: around its first marked word, else its start.""" + if M0 in marked: + return snippet(marked, SNIPPET // 2) + return snippet(marked, SNIPPET) diff --git a/fastapi_backend/app/schemas/search.py b/fastapi_backend/app/schemas/search.py index 0aa237ff1..4e5297a12 100644 --- a/fastapi_backend/app/schemas/search.py +++ b/fastapi_backend/app/schemas/search.py @@ -126,3 +126,61 @@ class PhotoHit(ResponseModel): class PhotoResults(ResponseModel): hits: list[PhotoHit] = Field(default_factory=list) + + +class SimilarCite(ResponseModel): + """A moment that shows why a resource is alike: one of its passages closest in meaning, or a line with shared words.""" + + match: Literal["meaning", "words"] = Field(description="found by meaning (a close passage) or by the shared words") + source: Literal["said", "page", "described"] = Field( + description='"said" (the transcript), "page" (text on a page of a document or an image) or "described" (what a ' + "shot or a page shows)" + ) + snippet: str = Field(description="HTML: escaped text with around the key words") + idx: int | None = Field(None, description="the line it starts at (from 0)") + t0: int | None = Field(None, description="ms; None on a page") + page: int | None = Field(None, description="on a document or an image: which page (from 0)") + similarity: float | None = Field(None, description="found by meaning: how alike it is (cosine, up to 1)") + + +class SimilarTopic(ResponseModel): + id: int + label: str + + +class SimilarEntity(ResponseModel): + id: int + name: str + + +class SimilarItem(ResponseModel): + recording_id: int + title: str | None = None + recorded_at: str | None = None + namespace: str | None = None + media_kind: str = Field(description="audio, video, transcript, document or image") + score: float = Field(description="reciprocal-rank fusion of the lists it's in; only for ordering") + match: list[Literal["meaning", "words", "topics", "entities"]] = Field( + description="why it's alike: close in meaning, shares key words, about the same topics, mentions the same entities" + ) + similarity: float | None = Field(None, description="by meaning: how alike its closest passage is (cosine, up to 1)") + cites: list[SimilarCite] = Field(default_factory=list, description="up to 3 moments that show it, to open") + words: list[str] = Field(default_factory=list, description="the key words it says") + topics: list[SimilarTopic] = Field(default_factory=list, description="topics both are about") + entities: list[SimilarEntity] = Field(default_factory=list, description="entities both mention") + + +class SimilarResults(ResponseModel): + basis: Literal["text", "recording", "topic", "entity"] = Field(description="what the results are like") + label: str | None = Field(None, description="the text's start, or the resource's title, the topic's label, the entity's name") + text: str | None = Field( + None, description="the text compared by meaning (a topic's labels and definition, an entity's name and description)" + ) + terms: list[str] = Field(default_factory=list, description="the key words looked for") + mode: Literal["meaning", "words"] = Field( + description='"meaning": an embedding model compared it too; "words": only by key words, topics and entities' + ) + meaning: str | None = Field(None, description="why it wasn't compared by meaning, when search by meaning is set up") + semantic: bool = Field(False, description="search by meaning is set up and indexed") + total: int + items: list[SimilarItem] diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index 6a2b783db..cfee929b3 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -196,6 +196,15 @@ def test_search_and_cite(client, env): assert tool_error(client, h, "search", query="capsid", namespace="calls") == "not found, or not something you can read" assert tool_error(client, h, "search", query="capsid", recording_id=call) == "not found, or not something you can read" + # like a text, citing the moments; never what isn't theirs + like = tool(client, h, "find_similar", text="capsid samples for gene therapy") + assert like["by"] == "words" and "capsid" in like["key_words"] and [r["recording_id"] for r in like["results"]] == [ep1] + m = like["results"][0]["moments"][0] + assert m["found"] == "words" and "**" in m["text"] and m["url"].startswith(f"http://localhost:3000/resources/{ep1}") + assert [r["recording_id"] for r in tool(client, h, "find_similar", recording_id=ep1)["results"]] == [ep2] + assert tool_error(client, h, "find_similar", recording_id=call) == "not found, or not something you can read" + assert tool_error(client, h, "find_similar") == "give one of text, recording_id, topic_id or entity_id" + c = tool(client, h, "cite", recording_id=ep1, line=1, lines=2) assert c["quote"].startswith("Wow, really? Dyno Therapeutics") and c["quote"].endswith("by a wide margin.") assert c["speakers"] == ["Bob", "Alice"] and c["title"] == "ep1" and c["line"] == 1 diff --git a/fastapi_backend/tests/api/test_similar.py b/fastapi_backend/tests/api/test_similar.py new file mode 100644 index 000000000..78c44ad34 --- /dev/null +++ b/fastapi_backend/tests/api/test_similar.py @@ -0,0 +1,180 @@ +"""Similar content: the resources most like a text, a resource, a topic or an entity, by meaning when it's set up and +always by key words, shared topics and shared entities, each citing why.""" + +from __future__ import annotations + +import pytest + +from app.domain import semantic, similar, store, topics +from tests import fake_llm +from tests.helpers import login, make_user, quiet, seed + +R = store.R +MONEY = """Erin|N|I have to be honest, we can't afford the rent this month. +Frank|Sad|Cash is tight for everyone, the bills keep coming. +Erin|N|Maybe we should ask about a smaller budget for the studio. +Frank|N|Anyway, did you see the match last night?""" + + +def _all(db): + return set(store.space_names(db)) + + +def _archive(db, cfg, folder): + from app.domain import analyze, ingest + + ids = seed(db, cfg, folder) + p = folder / "money.txt" + p.write_text(MONEY) + ids.append(ingest.import_transcript(db, cfg, "pods", p, log=quiet)) + analyze.analyze_pending(db, cfg, log=quiet) + return ids + + +@pytest.fixture +def llm(cfg): + srv, url = fake_llm.start() + cfg["llm"].update(base_url=url, model="fake") + fake_llm.Handler.embeddings, fake_llm.Handler.embed_fail = True, False + fake_llm.Handler.embedded.clear() + semantic._QUERIES.clear() + semantic._DOWN.clear() + yield fake_llm.Handler + fake_llm.Handler.embeddings = False + srv.shutdown() + + +def _ids(res): + return [x["recording_id"] for x in res["items"]] + + +def _entity(db, ns, name): + return db.one("SELECT record::id(id) AS id FROM entity WHERE space = $s AND name = $n", s=store.ns_id(db, ns), n=name)["id"] + + +def test_like_a_text_by_its_key_words(db, cfg, folder): + ep1, ep2, call, money = _archive(db, cfg, folder) + res = similar.similar(db, cfg, text="Designing a capsid with machine learning for gene therapy", spaces=_all(db)) + # no embeddings server: by the words alone, the ones the archive says least first + assert res["basis"] == "text" and res["mode"] == "words" and not res["semantic"] and res["meaning"] is None + assert {"capsid", "gene", "therapy"} <= set(res["terms"]) and "with" not in res["terms"] + assert _ids(res) == [ep1, call] # ep1 says more of them than the call; the money talk none + top = res["items"][0] + assert top["match"] == ["words"] and top["namespace"] == "pods" and top["media_kind"] == "transcript" + assert {"capsid", "gene", "therapy"} <= set(top["words"]) and len(top["cites"]) == 3 + # each cites a moment to open, its words marked + c = top["cites"][0] + assert c["match"] == "words" and c["source"] == "said" and isinstance(c["idx"], int) and c["t0"] is not None + assert "" in c["snippet"] + # only the namespaces given, and never the resource it was selected in + assert _ids(similar.similar(db, cfg, text="capsid samples", spaces={store.ns_id(db, "pods")})) == [ep1] + assert call not in _ids(similar.similar(db, cfg, text="capsid samples", spaces=_all(db), exclude=call)) + assert similar.similar(db, cfg, text="capsid", spaces=set())["items"] == [] + # paging + assert _ids(similar.similar(db, cfg, text="capsid gene therapy", spaces=_all(db), limit=1, offset=1)) == [call] + with pytest.raises(ValueError): + similar.similar(db, cfg, spaces=_all(db)) + with pytest.raises(ValueError): + similar.similar(db, cfg, text="capsid", recording=ep1, spaces=_all(db)) + + +def test_like_a_resource_by_words_topics_and_entities(db, cfg, folder): + ep1, ep2, call, money = _archive(db, cfg, folder) + pods = store.ns_id(db, "pods") + tid = topics.create(db, pods, "Gene therapy") + topics.tag(db, tid, [ep1, ep2]) + res = similar.similar(db, cfg, recording=ep1, spaces=_all(db)) + assert res["basis"] == "recording" and res["label"] == "ep1" and res["terms"] + assert ep1 not in _ids(res) # never like itself + by = {x["recording_id"]: x for x in res["items"]} + # ep2 is about the same topic and mentions Dyno Therapeutics too; the call (another namespace) shares words + assert _ids(res)[0] == ep2 + assert {"topics", "entities"} <= set(by[ep2]["match"]) + assert by[ep2]["topics"] == [{"id": tid, "label": "Gene therapy"}] + assert "Dyno Therapeutics" in [e["name"] for e in by[ep2]["entities"]] + assert "words" in by[call]["match"] and by[call]["cites"] + assert money not in by # nothing in common + + +def test_like_a_topic_and_an_entity(db, cfg, folder): + ep1, ep2, call, money = _archive(db, cfg, folder) + pods = store.ns_id(db, "pods") + tid = topics.create(db, pods, "Capsid", alt=["viral capsid"], definition="The protein shell of a virus.") + topics.tag(db, tid, [ep1]) + res = similar.similar(db, cfg, topic=tid, spaces=_all(db)) + assert res["basis"] == "topic" and res["label"] == "Capsid" and "protein shell" in res["text"] + by = {x["recording_id"]: x for x in res["items"]} + assert _ids(res)[0] == ep1 and set(by[ep1]["match"]) == {"words", "topics"} + assert by[call]["match"] == ["words"] # says capsid, but isn't tagged + with pytest.raises(KeyError): # a topic of a namespace not given + similar.similar(db, cfg, topic=tid, spaces={store.ns_id(db, "calls")}) + + dyno = _entity(db, "pods", "Dyno Therapeutics") + res = similar.similar(db, cfg, entity=dyno, spaces=_all(db)) + assert res["basis"] == "entity" and res["label"] == "Dyno Therapeutics" + by = {x["recording_id"]: x for x in res["items"]} + assert {ep1, ep2, call} <= set(by) and money not in by + assert "entities" in by[ep1]["match"] and "entities" not in by[call]["match"] # the call's is another namespace's + + +def test_by_meaning_when_set_up(db, cfg, folder, llm): + ep1, ep2, call, money = _archive(db, cfg, folder) + semantic.index_pending(db, cfg, log=quiet) + res = similar.similar(db, cfg, text="we are worried about our finances", spaces=_all(db)) + assert res["mode"] == "meaning" and res["semantic"] + top = res["items"][0] + assert top["recording_id"] == money and top["match"][0] == "meaning" and top["similarity"] >= 0.5 + assert top["cites"][0]["match"] == "meaning" and "afford the rent" in top["cites"][0]["snippet"] + # a resource: the centre of its own passages + assert semantic.centre(db, cfg, ep1) and semantic.centre(db, cfg, 999) is None + res = similar.similar(db, cfg, recording=ep1, spaces=_all(db)) + by = {x["recording_id"]: x for x in res["items"]} + assert ep1 not in by and by[ep2]["match"][0] == "meaning" and by[ep2]["similarity"] > 0.9 + assert by[ep2]["cites"][0]["match"] == "meaning" and by[ep2]["cites"][0]["idx"] == 0 + assert by[call]["match"] == ["words"] # not alike enough in meaning, but it says capsid and Dyno Therapeutics + # the server goes away: the words still answer, saying why + llm.embed_fail = True + semantic._QUERIES.clear() + res = similar.similar(db, cfg, text="capsid samples from the lab", spaces=_all(db)) + assert res["mode"] == "words" and "unavailable" in res["meaning"] and _ids(res) + + +def test_similar_api(client, db, cfg, folder): + ep1, ep2, call, money = _archive(db, cfg, folder) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + hv = login(client, "vi@x.io", "viewer password 1") + r = client.get("/api/v1/similar", params={"text": "capsid samples"}, headers=hv) + assert r.status_code == 200, r.text + assert [x["recording_id"] for x in r.json()["items"]] == [ep1] # the call is in a namespace they can't read + r = client.get("/api/v1/similar", params={"recording": ep1}, headers=hv).json() + assert ep1 not in [x["recording_id"] for x in r["items"]] and call not in [x["recording_id"] for x in r["items"]] + assert client.get("/api/v1/similar", params={"recording": call}, headers=hv).status_code == 404 + calls_dyno = _entity(db, "calls", "Dyno Therapeutics") + assert client.get("/api/v1/similar", params={"entity": calls_dyno}, headers=hv).status_code == 404 + assert client.get("/api/v1/similar", params={"topic": 999}, headers=hv).status_code == 404 + assert client.get("/api/v1/similar", headers=hv).status_code == 422 + assert client.get("/api/v1/similar", params={"text": " "}, headers=hv).status_code == 422 + assert client.get("/api/v1/similar", params={"text": "x", "recording": ep1}, headers=hv).status_code == 422 + assert client.get("/api/v1/similar", params={"text": "capsid"}).status_code == 401 + + +def test_the_assistant_finds_similar_within_its_scope(db, cfg, folder): + import json + + from app.domain import ai_tools + + ep1, ep2, call, money = _archive(db, cfg, folder) + pods, calls = store.ns_id(db, "pods"), store.ns_id(db, "calls") + box = ai_tools.Toolbox(db, cfg, {"id": 1, "email": "e"}, {pods, calls}, set(), {}, None) + assert "find_similar" in [t["function"]["name"] for t in box.specs()] + out, summary = box.call("find_similar", {"text": "send the capsid samples"}) + got = json.loads(out) + assert {r["recording_id"] for r in got["results"]} == {ep1, call} and summary.startswith("Found 2") + # each moment is numbered to cite, its text plain + m = got["results"][0]["moments"][0] + assert m["ref"] == 1 and "" not in m["text"] and "'" not in json.dumps(got) + assert box.refs[0]["recording_id"] == got["results"][0]["recording_id"] + # a conversation about one namespace finds nothing beyond it + box = ai_tools.Toolbox(db, cfg, {"id": 1, "email": "e"}, {pods, calls}, set(), {"namespaces": ["pods"]}, None) + assert {r["recording_id"] for r in json.loads(box.call("find_similar", {"text": "capsid samples"})[0])["results"]} == {ep1} + assert "isn't available" in json.loads(box.call("find_similar", {"recording_id": call})[0])["error"] diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index 32ab2f424..30d670fde 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -828,6 +828,11 @@ export type { FindGraphPathErrors, FindGraphPathResponse, FindGraphPathResponses, + FindSimilarData, + FindSimilarError, + FindSimilarErrors, + FindSimilarResponse, + FindSimilarResponses, FinishData, FinishError, FinishErrors, @@ -2431,6 +2436,11 @@ export type { SigninLinkUseErrors, SigninLinkUseResponse, SigninLinkUseResponses, + SimilarCite, + SimilarEntity, + SimilarItem, + SimilarResults, + SimilarTopic, SkipTopicCandidateData, SkipTopicCandidateError, SkipTopicCandidateErrors, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index b935fbf9b..bec2fdbd8 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -384,6 +384,9 @@ import type { FindGraphPathData, FindGraphPathErrors, FindGraphPathResponses, + FindSimilarData, + FindSimilarErrors, + FindSimilarResponses, FinishData, FinishErrors, FinishResponses, @@ -5126,6 +5129,23 @@ export class Search { }); } + /** + * Find Similar + * + * The resources most like a text, a resource, a topic or an entity (give one), best first, in the namespaces you + * can read and the collections you were given a role on, each with why: its closest passages by meaning (when + * search by meaning is set up), the lines with the key words it shares, and the topics and entities in common. + * Each cited moment can be opened. A resource is never listed as like itself. + */ + public static findSimilar( + options?: Options, + ): RequestResult { + return (options?.client ?? client).get({ + url: "/api/v1/similar", + ...options, + }); + } + /** * Suggest Terms * diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 17a895634..c24d7e765 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -14225,6 +14225,215 @@ export type SigninLinkToken = { token: string; }; +/** + * SimilarCite + * + * A moment that shows why a resource is alike: one of its passages closest in meaning, or a line with shared words. + */ +export type SimilarCite = { + /** + * Match + * + * found by meaning (a close passage) or by the shared words + */ + match: "meaning" | "words"; + /** + * Source + * + * "said" (the transcript), "page" (text on a page of a document or an image) or "described" (what a shot or a page shows) + */ + source: "said" | "page" | "described"; + /** + * Snippet + * + * HTML: escaped text with around the key words + */ + snippet: string; + /** + * Idx + * + * the line it starts at (from 0) + */ + idx?: number | null; + /** + * T0 + * + * ms; None on a page + */ + t0?: number | null; + /** + * Page + * + * on a document or an image: which page (from 0) + */ + page?: number | null; + /** + * Similarity + * + * found by meaning: how alike it is (cosine, up to 1) + */ + similarity?: number | null; + [key: string]: unknown; +}; + +/** + * SimilarEntity + */ +export type SimilarEntity = { + /** + * Id + */ + id: number; + /** + * Name + */ + name: string; + [key: string]: unknown; +}; + +/** + * SimilarItem + */ +export type SimilarItem = { + /** + * Recording Id + */ + recording_id: number; + /** + * Title + */ + title?: string | null; + /** + * Recorded At + */ + recorded_at?: string | null; + /** + * Namespace + */ + namespace?: string | null; + /** + * Media Kind + * + * audio, video, transcript, document or image + */ + media_kind: string; + /** + * Score + * + * reciprocal-rank fusion of the lists it's in; only for ordering + */ + score: number; + /** + * Match + * + * why it's alike: close in meaning, shares key words, about the same topics, mentions the same entities + */ + match: Array<"meaning" | "words" | "topics" | "entities">; + /** + * Similarity + * + * by meaning: how alike its closest passage is (cosine, up to 1) + */ + similarity?: number | null; + /** + * Cites + * + * up to 3 moments that show it, to open + */ + cites?: Array; + /** + * Words + * + * the key words it says + */ + words?: Array; + /** + * Topics + * + * topics both are about + */ + topics?: Array; + /** + * Entities + * + * entities both mention + */ + entities?: Array; + [key: string]: unknown; +}; + +/** + * SimilarResults + */ +export type SimilarResults = { + /** + * Basis + * + * what the results are like + */ + basis: "text" | "recording" | "topic" | "entity"; + /** + * Label + * + * the text's start, or the resource's title, the topic's label, the entity's name + */ + label?: string | null; + /** + * Text + * + * the text compared by meaning (a topic's labels and definition, an entity's name and description) + */ + text?: string | null; + /** + * Terms + * + * the key words looked for + */ + terms?: Array; + /** + * Mode + * + * "meaning": an embedding model compared it too; "words": only by key words, topics and entities + */ + mode: "meaning" | "words"; + /** + * Meaning + * + * why it wasn't compared by meaning, when search by meaning is set up + */ + meaning?: string | null; + /** + * Semantic + * + * search by meaning is set up and indexed + */ + semantic?: boolean; + /** + * Total + */ + total: number; + /** + * Items + */ + items: Array; + [key: string]: unknown; +}; + +/** + * SimilarTopic + */ +export type SimilarTopic = { + /** + * Id + */ + id: number; + /** + * Label + */ + label: string; + [key: string]: unknown; +}; + /** * Source */ @@ -24063,6 +24272,70 @@ export type SearchPhotosResponses = { export type SearchPhotosResponse = SearchPhotosResponses[keyof SearchPhotosResponses]; +export type FindSimilarData = { + body?: never; + path?: never; + query?: { + /** + * Text + * + * what was selected: a sentence, a passage, a title + */ + text?: string | null; + /** + * Recording + * + * a resource: the others most like it + */ + recording?: number | null; + /** + * Topic + * + * a topic: the resources most like its labels and definition, and about it + */ + topic?: number | null; + /** + * Entity + * + * an entity: the resources most like its name and description, and mentioning it + */ + entity?: number | null; + /** + * Exclude + * + * leave out this resource too (the one the text was selected in) + */ + exclude?: number | null; + /** + * Limit + */ + limit?: number; + /** + * Offset + */ + offset?: number; + }; + url: "/api/v1/similar"; +}; + +export type FindSimilarErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type FindSimilarError = FindSimilarErrors[keyof FindSimilarErrors]; + +export type FindSimilarResponses = { + /** + * Successful Response + */ + 200: SimilarResults; +}; + +export type FindSimilarResponse = FindSimilarResponses[keyof FindSimilarResponses]; + export type SuggestTermsData = { body?: never; path?: never; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 3fbb06cd4..c6abdfaa8 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -10794,6 +10794,154 @@ } } }, + "/api/v1/similar": { + "get": { + "tags": [ + "search" + ], + "summary": "Find Similar", + "description": "The resources most like a text, a resource, a topic or an entity (give one), best first, in the namespaces you\ncan read and the collections you were given a role on, each with why: its closest passages by meaning (when\nsearch by meaning is set up), the lines with the key words it shares, and the topics and entities in common.\nEach cited moment can be opened. A resource is never listed as like itself.", + "operationId": "find_similar", + "parameters": [ + { + "name": "text", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "maxLength": 2000 + }, + { + "type": "null" + } + ], + "description": "what was selected: a sentence, a passage, a title", + "title": "Text" + }, + "description": "what was selected: a sentence, a passage, a title" + }, + { + "name": "recording", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "description": "a resource: the others most like it", + "title": "Recording" + }, + "description": "a resource: the others most like it" + }, + { + "name": "topic", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "description": "a topic: the resources most like its labels and definition, and about it", + "title": "Topic" + }, + "description": "a topic: the resources most like its labels and definition, and about it" + }, + { + "name": "entity", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "description": "an entity: the resources most like its name and description, and mentioning it", + "title": "Entity" + }, + "description": "an entity: the resources most like its name and description, and mentioning it" + }, + { + "name": "exclude", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "description": "leave out this resource too (the one the text was selected in)", + "title": "Exclude" + }, + "description": "leave out this resource too (the one the text was selected in)" + }, + { + "name": "limit", + "in": "query", + "required": false, + "schema": { + "type": "integer", + "maximum": 50, + "minimum": 1, + "default": 10, + "title": "Limit" + } + }, + { + "name": "offset", + "in": "query", + "required": false, + "schema": { + "type": "integer", + "minimum": 0, + "default": 0, + "title": "Offset" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SimilarResults" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/search/terms": { "get": { "tags": [ @@ -48919,6 +49067,341 @@ ], "title": "SigninLinkToken" }, + "SimilarCite": { + "properties": { + "match": { + "type": "string", + "enum": [ + "meaning", + "words" + ], + "title": "Match", + "description": "found by meaning (a close passage) or by the shared words" + }, + "source": { + "type": "string", + "enum": [ + "said", + "page", + "described" + ], + "title": "Source", + "description": "\"said\" (the transcript), \"page\" (text on a page of a document or an image) or \"described\" (what a shot or a page shows)" + }, + "snippet": { + "type": "string", + "title": "Snippet", + "description": "HTML: escaped text with around the key words" + }, + "idx": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Idx", + "description": "the line it starts at (from 0)" + }, + "t0": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "T0", + "description": "ms; None on a page" + }, + "page": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Page", + "description": "on a document or an image: which page (from 0)" + }, + "similarity": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Similarity", + "description": "found by meaning: how alike it is (cosine, up to 1)" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "match", + "source", + "snippet" + ], + "title": "SimilarCite", + "description": "A moment that shows why a resource is alike: one of its passages closest in meaning, or a line with shared words." + }, + "SimilarEntity": { + "properties": { + "id": { + "type": "integer", + "title": "Id" + }, + "name": { + "type": "string", + "title": "Name" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "name" + ], + "title": "SimilarEntity" + }, + "SimilarItem": { + "properties": { + "recording_id": { + "type": "integer", + "title": "Recording Id" + }, + "title": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Title" + }, + "recorded_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Recorded At" + }, + "namespace": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Namespace" + }, + "media_kind": { + "type": "string", + "title": "Media Kind", + "description": "audio, video, transcript, document or image" + }, + "score": { + "type": "number", + "title": "Score", + "description": "reciprocal-rank fusion of the lists it's in; only for ordering" + }, + "match": { + "items": { + "type": "string", + "enum": [ + "meaning", + "words", + "topics", + "entities" + ] + }, + "type": "array", + "title": "Match", + "description": "why it's alike: close in meaning, shares key words, about the same topics, mentions the same entities" + }, + "similarity": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Similarity", + "description": "by meaning: how alike its closest passage is (cosine, up to 1)" + }, + "cites": { + "items": { + "$ref": "#/components/schemas/SimilarCite" + }, + "type": "array", + "title": "Cites", + "description": "up to 3 moments that show it, to open" + }, + "words": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Words", + "description": "the key words it says" + }, + "topics": { + "items": { + "$ref": "#/components/schemas/SimilarTopic" + }, + "type": "array", + "title": "Topics", + "description": "topics both are about" + }, + "entities": { + "items": { + "$ref": "#/components/schemas/SimilarEntity" + }, + "type": "array", + "title": "Entities", + "description": "entities both mention" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "recording_id", + "media_kind", + "score", + "match" + ], + "title": "SimilarItem" + }, + "SimilarResults": { + "properties": { + "basis": { + "type": "string", + "enum": [ + "text", + "recording", + "topic", + "entity" + ], + "title": "Basis", + "description": "what the results are like" + }, + "label": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Label", + "description": "the text's start, or the resource's title, the topic's label, the entity's name" + }, + "text": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Text", + "description": "the text compared by meaning (a topic's labels and definition, an entity's name and description)" + }, + "terms": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Terms", + "description": "the key words looked for" + }, + "mode": { + "type": "string", + "enum": [ + "meaning", + "words" + ], + "title": "Mode", + "description": "\"meaning\": an embedding model compared it too; \"words\": only by key words, topics and entities" + }, + "meaning": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Meaning", + "description": "why it wasn't compared by meaning, when search by meaning is set up" + }, + "semantic": { + "type": "boolean", + "title": "Semantic", + "description": "search by meaning is set up and indexed", + "default": false + }, + "total": { + "type": "integer", + "title": "Total" + }, + "items": { + "items": { + "$ref": "#/components/schemas/SimilarItem" + }, + "type": "array", + "title": "Items" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "basis", + "mode", + "total", + "items" + ], + "title": "SimilarResults" + }, + "SimilarTopic": { + "properties": { + "id": { + "type": "integer", + "title": "Id" + }, + "label": { + "type": "string", + "title": "Label" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "label" + ], + "title": "SimilarTopic" + }, "Source": { "properties": { "id": {