Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

Plans and progress for work in flight. Newest first.

## 2026-10-10 · Similar content

Goal (Adeel): a Similar tab on every resource, and selecting text anywhere (a resource's text or title, notes, topics,
entities, the graph canvas) shows similar content in a popover, with a way to find more.

Model (fastapi_backend/app/domain/similar.py, docs/processing.md#similar-content): one list for a text, a resource, a
topic or an entity, fused by reciprocal rank from four: meaning (passages near the text, or near the centre of a
resource's passages: no model call for a resource), key words (the text's words the `term` table says are rare, or
`analyze.keywords`, OR'ed through the word index), shared topics and shared entities (rarer ones weigh more). Each item
cites up to three moments. `GET /similar`, assistant and MCP `find_similar`. Without embeddings (the default) the
other three answer, with no model call.

PRs: API, assistant and MCP tool; the Similar tab; the selection popover and the full Similar page; topics, entities and
graph nodes.

Refine later:
- key words of a text are single words and two-word phrases; stems beyond plurals (designing/design) don't match;
- a resource not indexed for meaning is compared by its title and keywords (one embedding request), not its text;
- the centre of a long resource's passages blurs what its parts are about; comparing a few of its passages may be
sharper;
- entities in another namespace with the same name count only through the words, not as the same entity;
- no "not similar" feedback yet, and nothing is cached between requests beyond the query vectors.

## 2026-10-10 · More voices for spoken answers

Goal (Adeel): read answers and podcasts aloud with ElevenLabs, macOS `say`, Google Gemini TTS or StyleTTS2, with
Expand Down
11 changes: 11 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -838,6 +838,7 @@ recording later leaves its uploaded file in place, like any other media file.
```
GET /api/v1/search
GET /api/v1/search/terms
GET /api/v1/similar
GET /api/v1/graph
GET /api/v1/mentions
```
Expand Down Expand Up @@ -867,6 +868,16 @@ when search by meaning is set up and the query has no "phrases" or OR, else keyw
matched, `semantic` whether search by meaning is set up, and `meaning` why it wasn't used when asked for. Each hit's
`match` is `words`, `meaning` (with its `similarity`, cosine, and shown at the passage's best line) or `both`.

`GET /similar` lists the resources most like one thing, best first, in the namespaces you can read and the
collections you were given a role on ([similar content](processing.md#similar-content)): a `text` (what was selected,
up to 2,000 characters; `exclude` leaves out the resource it was selected in), a `recording`, a `topic` or an
`entity`; give exactly one (422 otherwise, 404 for one you can't read). `limit` (default 10, at most 50) and `offset`
page through them. Each item says why it's alike in `match` (`meaning`, `words`, `topics`, `entities`), with its
`similarity` by meaning, up to three `cites` (moments to open: `source`, `idx`, `t0` or `page`, and a `snippet` with the
key words marked), the key `words` it says, and the `topics` and `entities` in common. The reply's `mode` is `meaning`
when an embedding model compared them too, else `words`; `terms` are the key words looked for and `meaning` says why
it wasn't by meaning when search by meaning is set up. A resource is never listed as like itself.

Search has no prefix search (`interp*` looks for the word "interp"). `GET /search/terms?prefix=interp` lists whole
words said in the namespaces you can read (`ns` for one) that start with it, the most said first, with how often and
in how many recordings (`limit`, default 8, at most 20); the web app offers them as "Try …".
Expand Down
3 changes: 2 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,8 @@ engine), `POST /voice/transcribe` (a clip as the raw body; nothing is kept), `PO
When the configured model supports function calling, chat becomes an agent.

- **What it can look things up with:** transcripts (including on-screen text), recordings, transcript passages,
outputs, entities and their mentions and timelines, the knowledge graph, and speakers.
outputs, entities and their mentions and timelines, the knowledge graph, speakers, and content similar to a text,
a recording, a topic or an entity (`find_similar`).
- **Limits:** everything is limited to what the person can read and to the conversation's scope, and capped by
`ai.max_steps` and `ai.max_transcript_reads`.
- **Citations:** every moment a tool returns is numbered, so answers cite [n] across everything the assistant read.
Expand Down
1 change: 1 addition & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ front of the web app also needs `TRUST_PROXY_HEADERS=true` ([Authentication](aut
| Tool | |
|---|---|
| `search` | moments where words are said, shown on screen or written, best first: the recording, speaker, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, speaker or recording |
| `find_similar` | the recordings, documents and images most like a text, a recording, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links |
| `list_namespaces` | the namespaces you can read, with their recordings, hours, speakers and your role |
| `list_recordings` | recordings, newest first, by namespace, title words, dates, speaker, tag or kind |
| `list_speakers` | the speakers of your namespaces, the ones who talk most first |
Expand Down
19 changes: 19 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,25 @@ Settings → Search shows how far it has got and can queue more now; `lens embed
embeddings server can't be reached, the step is skipped (the routine tries again later) and searches go by the words,
saying why.

### Similar content

The API (`GET /similar`), the assistant and the MCP server (`find_similar`) list the resources most like a text (a
sentence, a passage, a title), a resource, a topic or an entity.

Four lists are fused by reciprocal rank, so a resource alike in several ways comes first:

* **meaning**, when search by meaning is set up: the passages closest to the text (a topic's labels and definition,
an entity's name and description), or for a resource to the centre of its own passages, so no model call is needed;
* **key words**: the text's words that the archive says in fewest recordings (from the analysis step's term counts),
or a resource's keywords, any of them, through the word index: resources saying more of them first;
* **topics**: resources about the same topics, a topic fewer resources are about counting for more;
* **entities**: resources mentioning the same people, organisations, places and terms, the rarer counting for more.

Each result cites why: its closest passages and the lines with the shared words (each opens at that moment or page),
the words, and the topics and entities in common. Without an embeddings server (the default) the last three still
answer, and no language model is called; with one, a selected text costs one embedding request, logged in the
activity ledger.

## Background work

Imports, pipeline runs and folder scans return at once and run as jobs stored in SurrealDB. The server runs
Expand Down
79 changes: 79 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,85 @@ def search(
return out


@tool(
"find_similar",
"Find similar content",
"The recordings, documents and images most like a text (a sentence, a passage, a title), a recording, a topic or "
"an entity: give exactly one. Best first, each with why it's alike: `match` (meaning: close in meaning, when an "
"embedding model is set up; words: shares key words; topics: about the same topics; entities: mentions the same "
"people, organisations, places...), the moments that show it (text with the key words in **bold**, and a url that "
"opens it there), and the words, topics and entities in common. A recording is never listed as like itself.",
Arg("text", "string", "what to find more like: a sentence, a passage, a title", max_length=2000),
Arg("recording_id", "integer", "a recording: the others most like it"),
Arg("topic_id", "integer", "a topic (from list_topics): what's most like its labels and definition, and about it"),
Arg("entity_id", "integer", "an entity (from list_entities): what's most like its name and description, and mentions it"),
Arg("exclude_recording_id", "integer", "leave this recording out too (the one the text is from)"),
Arg("limit", "integer", "how many", default=10, minimum=1, maximum=50),
Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000),
)
def find_similar(
ctx: Context,
text: str | None,
recording_id: int | None,
topic_id: int | None,
entity_id: int | None,
exclude_recording_id: int | None,
limit: int,
offset: int,
) -> dict[str, Any]:
if sum(x is not None for x in (text, recording_id, topic_id, entity_id)) != 1:
raise ToolError("give one of text, recording_id, topic_id or entity_id")
found = search_routes.find_similar(
ctx.acl,
ctx.user,
ctx.db,
ctx.cfg,
text=text,
recording=recording_id,
topic=topic_id,
entity=entity_id,
exclude=exclude_recording_id,
limit=limit,
offset=offset,
)
results = []
for x in found.items:
moments = []
for c in x.cites:
m: dict[str, Any] = {
"found": c.match,
"at": store.tc(c.t0) if c.t0 is not None and c.page is None else None,
"seconds": _seconds(c.t0) if c.page is None else None,
"line": c.idx,
"page": c.page + 1 if c.page is not None else None,
"similarity": c.similarity,
"text": _plain(c.snippet),
"url": ctx.link(x.recording_id, c.t0 if c.page is None else None) + (f"?page={c.page}" if c.page is not None else ""),
}
moments.append({k: v for k, v in m.items() if v is not None})
item: dict[str, Any] = {
"recording_id": x.recording_id,
"title": x.title,
"namespace": x.namespace,
"recorded_at": x.recorded_at,
"media": x.media_kind,
"match": x.match,
"similarity": x.similarity,
"moments": moments,
"words": x.words or None,
"topics": [{"id": t.id, "label": t.label} for t in x.topics] or None,
"entities": [{"id": e.id, "name": e.name} for e in x.entities] or None,
"url": ctx.link(x.recording_id),
}
results.append({k: v for k, v in item.items() if v is not None})
out: dict[str, Any] = {"like": found.label, "by": found.mode, "key_words": found.terms, "total": found.total, "results": results}
if found.meaning:
out["note"] = found.meaning
if offset + len(results) < found.total:
out["next_offset"] = offset + len(results)
return out


# ---------- browsing ----------
@tool(
"list_namespaces",
Expand Down
47 changes: 45 additions & 2 deletions fastapi_backend/app/api/v1/routes/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
from app.api.deps import Acl, Cfg, CurrentUser, Db
from app.api.media import sign_urls
from app.domain import graph as graphmod
from app.domain import graph_history, hierarchy, photos, render, store
from app.domain import graph_history, hierarchy, photos, render, similar, store
from app.domain import search as searchmod
from app.schemas.search import Graph, Mention, PhotoResults, SearchResults, TermSuggestion
from app.schemas.search import Graph, Mention, PhotoResults, SearchResults, SimilarResults, TermSuggestion

router = APIRouter(tags=["search"])

Expand Down Expand Up @@ -117,6 +117,49 @@ def search_photos(
return PhotoResults.model_validate(sign_urls({"hits": out}, full=True))


@router.get("/similar")
def find_similar(
acl: Acl,
user: CurrentUser,
db: Db,
cfg: Cfg,
text: str | None = Query(None, max_length=similar.TEXT_MAX, description="what was selected: a sentence, a passage, a title"),
recording: int | None = Query(None, description="a resource: the others most like it"),
topic: int | None = Query(None, description="a topic: the resources most like its labels and definition, and about it"),
entity: int | None = Query(None, description="an entity: the resources most like its name and description, and mentioning it"),
exclude: int | None = Query(None, description="leave out this resource too (the one the text was selected in)"),
limit: int = Query(10, ge=1, le=similar.LIMIT_MAX),
offset: int = Query(0, ge=0),
) -> SimilarResults:
"""The resources most like a text, a resource, a topic or an entity (give one), best first, in the namespaces you
can read and the collections you were given a role on, each with why: its closest passages by meaning (when
search by meaning is set up), the lines with the key words it shares, and the topics and entities in common.
Each cited moment can be opened. A resource is never listed as like itself."""
given = [x for x in (text and text.strip(), recording, topic, entity) if x not in (None, "")]
if len(given) != 1:
raise HTTPException(422, "give one of text, recording, topic or entity")
if recording is not None:
acl.recording(recording) # 404 unless they may read it
spaces = set(acl.roles)
try:
res = similar.similar(
db,
cfg,
text=text,
recording=recording,
topic=topic,
entity=entity,
spaces=spaces,
also=acl.partial_recordings(),
exclude=exclude,
limit=limit,
offset=offset,
)
except KeyError:
raise HTTPException(404, "not found") from None
return SimilarResults.model_validate(res)


@router.get("/search/terms")
def suggest_terms(
acl: Acl,
Expand Down
54 changes: 54 additions & 0 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from __future__ import annotations

import html
import json
import logging

Expand All @@ -25,6 +26,7 @@
recsets,
render,
search as searchmod,
similar,
speakers as spk,
store,
templates,
Expand All @@ -43,6 +45,15 @@
["query"],
False,
),
(
"find_similar",
"Find the recordings, documents and images in scope most like a text, a recording, a topic or an entity (give "
"one): close in meaning (when search by meaning is set up), sharing key words, about the same topics, mentioning "
"the same entities; each with the moments that show it.",
{"text": _S, "recording_id": _I, "topic_id": _I, "entity_id": _I, "limit": _I},
[],
False,
),
(
"list_recordings",
"List recordings in scope, newest first, optionally by namespace, dates (YYYY-MM-DD), speaker, entity or media.",
Expand Down Expand Up @@ -609,6 +620,49 @@ def t_search_transcripts(self, query, namespace=None, limit=8):
)
return {"total": res["total"], "results": out}, f'Searched for "{query}": {res["total"]} match(es)'

def t_find_similar(self, text=None, recording_id=None, topic_id=None, entity_id=None, limit=5):
if recording_id is not None:
self._ok(recording_id)
try:
res = similar.similar(
self.db,
self.cfg,
text=text,
recording=recording_id,
topic=topic_id,
entity=entity_id,
spaces=self.readable,
within=self.allowed,
limit=min(int(limit or 5), 20),
)
except KeyError:
raise ValueError("that topic or entity isn't available in this conversation") from None
out = []
for x in res["items"]:
moments = []
for c in x["cites"]:
said = html.unescape(c["snippet"].replace("<mark>", "").replace("</mark>", ""))
n = self.ref(x["recording_id"], c["t0"] or 0, said, title=x["title"], source=c["source"], page=c["page"])
moments.append(
{"ref": n, **({"page": c["page"] + 1} if c["page"] is not None else {"time": store.tc(c["t0"] or 0)}), "text": said}
)
out.append(
store.clean(
{
"recording_id": x["recording_id"],
"title": x["title"],
"namespace": x["namespace"],
"alike_by": x["match"],
"moments": moments,
"words": x["words"] or None,
"topics": [t["label"] for t in x["topics"]] or None,
"entities": [e["name"] for e in x["entities"]] or None,
}
)
)
like = res["label"] or ""
return {"like": like, "by": res["mode"], "total": res["total"], "results": out}, f'Found {res["total"]} like "{like[:60]}"'

def t_list_recordings(self, namespace=None, limit=20, speaker_id=None, entity_id=None, media=None, **dates):
f = recsets.clean_filter(
{
Expand Down
21 changes: 20 additions & 1 deletion fastapi_backend/app/domain/semantic.py
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,13 @@ def nearest(db, cfg, q, where_f, params, kinds, k):
text = query_text(q)
if not text or not kinds:
return []
vec = query_vector(cfg, text)
return nearest_to(db, cfg, query_vector(cfg, text), where_f, params, kinds, k)


def nearest_to(db, cfg, vec, where_f, params, kinds, k):
"""nearest() for a vector: a query's, or the centre of a recording's passages (`centre`)."""
if not kinds:
return []
dim = (state(db) or {}).get("dimension")
if dim and len(vec) != dim:
raise EmbedError(f"the model's vectors now have {len(vec)} dimensions, the index {dim}: index again (lens embed --force)")
Expand Down Expand Up @@ -469,3 +475,16 @@ def nearest(db, cfg, q, where_f, params, kinds, k):
rows = [r for r in rows if r.get("similarity") is not None]
least = max([floor(cfg)] + [r["similarity"] - SPREAD for r in rows[:1]])
return [r for r in rows if r["similarity"] >= least]


def centre(db, cfg, rid, kinds=("said", "page", "described")):
"""The mean of a recording's passage vectors (what it is about as a whole), or None when it has none from the
configured model yet."""
if not available(db, cfg):
return None
rows = db.rows("SELECT embedding FROM passage WHERE recording = $r AND kind IN $k AND embedding != NONE", r=int(rid), k=list(kinds))
vecs = [r["embedding"] for r in rows if r.get("embedding")]
if not vecs or len({len(v) for v in vecs}) != 1:
return None
n = len(vecs)
return [sum(col) / n for col in zip(*vecs)]
Loading
Loading