Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Similar content.** Every resource has a **Similar** tab, and selecting text anywhere (a transcript, a page, a
title, a graph label) shows the resources most like it, each citing the passages, words, topics and entities they
share, with **Find more** for the full list. Topics, entities and graph nodes list similar content too, and the
assistant and the MCP server can ask for it (`find_similar`). It works without search by meaning, by key words,
topics and entities ([Processing](docs/processing.md#similar-content)).

## 0.8.1 <small>October 12, 2026</small> {id="0.8.1"}

- **The installer stops waiting when Lens can't start, and shows why.** A microsandbox install that couldn't start Lens
Expand Down
6 changes: 4 additions & 2 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ resource's passages: no model call for a resource), key words (the text's words
cites up to three moments. `GET /similar`, assistant and MCP `find_similar`. Without embeddings (the default) the
other three answer, with no model call.

PRs: API, assistant and MCP tool; the Similar tab; the selection popover and the full Similar page; topics, entities and
graph nodes.
PRs: #296 the API, assistant and MCP tool; then the web app (the Similar tab, the selection popover and the full
Similar page at `/similar`, and similar content for topics, entities and graph nodes).

Refine later:
- key words of a text are single words and two-word phrases; stems beyond plurals (designing/design) don't match;
Expand All @@ -62,6 +62,8 @@ Refine later:
sharper;
- entities in another namespace with the same name count only through the words, not as the same entity;
- no "not similar" feedback yet, and nothing is cached between requests beyond the query vectors.
- selecting text in an editor (notes, the document editor) opens nothing, since selecting there is for editing; a
"Similar" item in the editor's own menu would cover it;

## 2026-10-10 · More voices for spoken answers

Expand Down
15 changes: 13 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,8 @@ saying why.

### Similar content

The API (`GET /similar`), the assistant and the MCP server (`find_similar`) list the resources most like a text (a
sentence, a passage, a title), a resource, a topic or an entity.
The web app, the API (`GET /similar`), the assistant and the MCP server (`find_similar`) list the resources most like
a text (a sentence, a passage, a title), a resource, a topic or an entity.

Four lists are fused by reciprocal rank, so a resource alike in several ways comes first:

Expand All @@ -235,6 +235,17 @@ the words, and the topics and entities in common. Without an embeddings server (
answer, and no language model is called; with one, a selected text costs one embedding request, logged in the
activity ledger.

In the web app:

* every resource has a **Similar** tab, after Entities, listing what is most like it;
* selecting text anywhere (a transcript, a page, a title, a summary, a graph label) opens a popover with the five
resources most like the selection, each citing why, and **Find more** opens the full list at `/similar`, with
paging and a search for the key words. The account menu's **Similar on select** switches between showing the
popover at once and showing a small **Similar** button to click first. Text typed in a field or an editor (notes)
opens nothing;
* a topic's and an entity's drawer, and a resource, topic or entity selected on the graph canvas, list similar
content too.

## Background work

Imports, pipeline runs and folder scans return at once and run as jobs stored in SurrealDB. The server runs
Expand Down
153 changes: 153 additions & 0 deletions nextjs-frontend/__tests__/similar.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { render, screen } from "@testing-library/react";
import "@testing-library/jest-dom";

import { Search } from "@/app/openapi-client";
import type { SimilarResults } from "@/app/openapi-client/types.gen";
import { basisFromParams, basisQuery, citeHref, similarHref, TEXT_MAX } from "@/components/similar/model";
import { nodeBasis } from "@/components/graph/panel";
import { resourceOf, selectedText } from "@/components/similar/selection-popover";
import { SimilarResultsView } from "@/components/similar/similar-list";
import { TooltipProvider } from "@/components/ui/tooltip";

jest.mock("@/app/openapi-client", () => ({ Search: { findSimilar: jest.fn() } }));
jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) }));
const findSimilar = Search.findSimilar as jest.Mock;

const RES: SimilarResults = {
basis: "recording",
label: "ep1",
text: null,
terms: ["capsid", "gene"],
mode: "words",
meaning: null,
semantic: false,
total: 12,
items: [
{
recording_id: 2,
title: "ep2",
namespace: "pods",
recorded_at: "2026-10-01T10:00:00",
media_kind: "document",
score: 0.03,
match: ["words", "topics"],
similarity: null,
cites: [
{ match: "words", source: "page", snippet: "the <mark>capsid</mark> &amp; more", idx: 4, t0: 2000, page: 2 },
{ match: "words", source: "said", snippet: "a <mark>gene</mark>", idx: 7, t0: 65000, page: null },
],
words: ["capsid", "gene"],
topics: [{ id: 9, label: "Gene therapy" }],
entities: [{ id: 5, name: "Dyno Therapeutics" }],
},
],
};

function show(ui: React.ReactNode) {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
return render(
<QueryClientProvider client={qc}>
<TooltipProvider>{ui}</TooltipProvider>
</QueryClientProvider>,
);
}

beforeEach(() => jest.clearAllMocks());

describe("similar content", () => {
it("asks for one basis, and links to the full list and the moments", () => {
expect(basisQuery({ text: "x".repeat(TEXT_MAX + 50), exclude: 3 })).toEqual({
text: "x".repeat(TEXT_MAX),
exclude: 3,
});
expect(basisQuery({ topic: 4 })).toEqual({ topic: 4 });
expect(similarHref({ text: "gene therapy", exclude: 3 })).toBe("/similar?text=gene+therapy&exclude=3");
expect(similarHref({ text: "gene therapy" })).toBe("/similar?text=gene+therapy");
expect(similarHref({ entity: 5 })).toBe("/similar?entity=5");
expect(basisFromParams(new URLSearchParams("text=gene+therapy&exclude=3"))).toEqual({
text: "gene therapy",
exclude: 3,
});
expect(basisFromParams(new URLSearchParams("recording=12"))).toEqual({ recording: 12 });
expect(basisFromParams(new URLSearchParams("topic=x"))).toBeNull();
expect(basisFromParams(new URLSearchParams(""))).toBeNull();
// a page of a document opens at that page; a moment at its second
expect(citeHref(2, { page: 2, t0: 2000 })).toBe("/resources/2?page=2");
expect(citeHref(2, { page: null, t0: 65000 })).toBe("/resources/2?t=65");
});

it("lists what's alike with why, the moments to open, and Find more", async () => {
findSimilar.mockResolvedValue({ data: RES, response: { ok: true, status: 200 } });
show(<SimilarResultsView basis={{ recording: 1 }} />);
expect(await screen.findByRole("link", { name: "ep2" })).toHaveAttribute("href", "/resources/2");
expect(findSimilar.mock.calls[0][0].query).toEqual({ recording: 1, limit: 10 });
expect(screen.getByText("Shares key words")).toBeInTheDocument();
expect(screen.getByText("Same topics")).toBeInTheDocument();
// the snippet is text, its matches marked, its entities decoded
expect(screen.getByText("capsid").tagName).toBe("MARK");
expect(screen.getByText("& more")).toBeInTheDocument();
expect(screen.getByText("p. 3").closest("a")).toHaveAttribute("href", "/resources/2?page=2");
expect(screen.getByText("1:05").closest("a")).toHaveAttribute("href", "/resources/2?t=65");
expect(screen.getByRole("link", { name: /Gene therapy/ })).toHaveAttribute("href", "/topics/9");
expect(screen.getByRole("link", { name: /Dyno Therapeutics/ })).toHaveAttribute("href", "/entities/5");
expect(screen.getByText("By key words, topics and entities.")).toBeInTheDocument();
expect(screen.getByRole("link", { name: /Find more \(12\)/ })).toHaveAttribute("href", "/similar?recording=1");
});

it("says when nothing is alike, and why it wasn't by meaning", async () => {
findSimilar.mockResolvedValue({ data: { ...RES, items: [], total: 0 }, response: { ok: true, status: 200 } });
show(<SimilarResultsView basis={{ text: "nothing like it" }} compact />);
expect(await screen.findByText("Nothing similar found.")).toBeInTheDocument();
});
});

describe("similar content on select", () => {
function select(el: Node) {
const r = document.createRange();
r.selectNodeContents(el);
const sel = window.getSelection()!;
sel.removeAllRanges();
sel.addRange(r);
return sel;
}

it("compares text selected in the page's content, not in fields, menus or elsewhere", () => {
document.body.innerHTML = `
<header><span id="nav">Library</span></header>
<main id="main">
<h1 id="title">Capsid design with
machine learning</h1>
<p id="short">ok</p><p id="digits">12 34</p>
<div contenteditable="true"><p id="editing">being edited now</p></div>
<div data-no-similar><p id="opted">left alone here</p></div>
</main>`;
const main = document.getElementById("main");
const at = (id: string) => selectedText(select(document.getElementById(id)!), main);
expect(at("title")).toBe("Capsid design with machine learning");
expect(at("short")).toBeNull();
expect(at("digits")).toBeNull();
expect(at("editing")).toBeNull();
expect(at("opted")).toBeNull();
expect(at("nav")).toBeNull();
window.getSelection()!.removeAllRanges();
expect(selectedText(window.getSelection(), main)).toBeNull();
});

it("leaves out the resource it was selected in", () => {
expect(resourceOf("/resources/12")).toBe(12);
expect(resourceOf("/recordings/7/edit")).toBe(7);
expect(resourceOf("/topics")).toBeNull();
expect(resourceOf(null)).toBeNull();
});

it("finds what's like a graph node: a recording, topic or entity, or a merged entity's name", () => {
const n = { label: "Dyno Therapeutics", refs: [5] };
expect(nodeBasis({ ...n, kind: "entity" })).toEqual({ entity: 5 });
expect(nodeBasis({ ...n, kind: "entity", refs: [5, 9] })).toEqual({ text: "Dyno Therapeutics" });
expect(nodeBasis({ ...n, kind: "topic" })).toEqual({ topic: 5 });
expect(nodeBasis({ ...n, kind: "recording" })).toEqual({ recording: 5 });
expect(nodeBasis({ ...n, kind: "speaker" })).toBeNull();
expect(nodeBasis({ ...n, kind: "collection" })).toBeNull();
});
});
15 changes: 15 additions & 0 deletions nextjs-frontend/app/(app)/similar/page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import type { Metadata } from "next";
import { Suspense } from "react";

import { SimilarPage } from "@/components/similar/similar-page";

export const metadata: Metadata = { title: "Similar" };

/** Everything like a selected text, a resource, a topic or an entity ("Find more"). */
export default function Page() {
return (
<Suspense>
<SimilarPage />
</Suspense>
);
}
11 changes: 10 additions & 1 deletion nextjs-frontend/components/app-shell/account-menu.tsx
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
"use client";

import { useQuery } from "@tanstack/react-query";
import { KeyRound, Keyboard, LogOut, Plug, Signpost, SunMoon, UserRound, Users } from "lucide-react";
import { KeyRound, Keyboard, LogOut, Plug, Signpost, Sparkles, SunMoon, UserRound, Users } from "lucide-react";
import Link from "next/link";
import { signOut } from "next-auth/react";
import { useState } from "react";

import { Tokens } from "@/app/openapi-client";
import { connectHint, mcpAddress, useAppSignIn } from "@/components/account/connect-assistant";
import { useTheme, type ThemeChoice } from "@/components/app-shell/theme";
import { useSelectMode } from "@/components/similar/prefs";
import { Menu, MenuContent, MenuItem, MenuTrigger } from "@/components/ui/menu";
import { Avatar, Skeleton } from "@/components/ui/states";
import { useToast } from "@/components/ui/toast";
Expand Down Expand Up @@ -42,6 +43,7 @@ export function AccountMenu({ name, email }: { name?: string | null; email: stri
const client = useApiClient();
const [open, setOpen] = useState(false);
const [theme, setTheme] = useTheme();
const [similar, setSimilar] = useSelectMode();
const toast = useToast();
const signIn = useAppSignIn(open);
const tokens = useQuery({
Expand Down Expand Up @@ -110,6 +112,13 @@ export function AccountMenu({ name, email }: { name?: string | null; email: stri
<MenuItem className={ITEM} icon={<SunMoon />} onSelect={() => setTheme(NEXT[theme])}>
Appearance: {theme[0].toUpperCase() + theme.slice(1)}
</MenuItem>
<MenuItem
className={ITEM}
icon={<Sparkles />}
onSelect={() => setSimilar(similar === "show" ? "button" : "show")}
>
Similar on select: {similar === "show" ? "Show" : "Button"}
</MenuItem>
<MenuItem
className={ITEM}
icon={<Keyboard />}
Expand Down
2 changes: 2 additions & 0 deletions nextjs-frontend/components/app-shell/app-shell.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { Shortcuts } from "@/components/app-shell/shortcuts";
import { BudgetMeter } from "@/components/costs/budget-meter";
import { WelcomeTour } from "@/components/home/welcome-tour";
import { PageChat } from "@/components/page-chat/page-chat";
import { SimilarOnSelect } from "@/components/similar/selection-popover";
import { Button, IconButton } from "@/components/ui/button";
import { EmptyState, Skeleton } from "@/components/ui/states";
import { data, useApiClient } from "@/lib/api/browser";
Expand Down Expand Up @@ -195,6 +196,7 @@ export function AppShell({ user, children }: { user: SessionUser; children: Reac
<PageChat />
</Suspense>
<Shortcuts />
<SimilarOnSelect />
<WelcomeTour />
</div>
);
Expand Down
2 changes: 2 additions & 0 deletions nextjs-frontend/components/entities/entity-drawer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { useEffect, useState } from "react";
import { Entities, Topics } from "@/app/openapi-client";
import type { EntityType } from "@/app/openapi-client/types.gen";
import { recordingHref } from "@/components/search/links";
import { SimilarSection } from "@/components/similar/similar-list";
import { Banner } from "@/components/ui/banner";
import { Button } from "@/components/ui/button";
import { Drawer } from "@/components/ui/dialog";
Expand Down Expand Up @@ -316,6 +317,7 @@ export function EntityDrawer({
))}
</ul>
</section>
<SimilarSection basis={{ entity: id }} />
</>
)}
</div>
Expand Down
15 changes: 15 additions & 0 deletions nextjs-frontend/components/graph/panel.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import { useRecordingIndex } from "@/components/search/data";
import { hasMedia, recordingHref } from "@/components/search/links";
import { InlinePlayerBar, useInlinePlayer, type InlinePlayer } from "@/components/search/player";
import { talkTime } from "@/components/speakers/format";
import { SimilarSection } from "@/components/similar/similar-list";
import type { SimilarBasis } from "@/components/similar/model";
import { useSpeakerRecordings } from "@/components/speakers/parts";
import { Banner } from "@/components/ui/banner";
import { Button, IconButton } from "@/components/ui/button";
Expand Down Expand Up @@ -153,6 +155,17 @@ function SpeakerRecordings({ n, player }: { n: GraphNode; player: InlinePlayer }
);
}

/** What a node's similar content is like: a recording, a topic or an entity itself; an entity merged across namespaces
* by its name. Speakers, collections and namespaces have none. */
export function nodeBasis(n: Pick<GraphNode, "kind" | "refs" | "label">): SimilarBasis | null {
const id = n.refs[0];
if (id == null) return null;
if (n.kind === "recording") return { recording: id };
if (n.kind === "topic") return n.refs.length === 1 ? { topic: id } : { text: n.label };
if (n.kind === "entity") return n.refs.length === 1 ? { entity: id } : { text: n.label };
return null;
}

/** The node panel: what it is, where it's mentioned (playable), what it's connected to, and what you can do with it. */
export function NodePanel({
node,
Expand Down Expand Up @@ -212,6 +225,7 @@ function StructurePanel({
<p className="m-0 text-[12.5px] text-fg-secondary">
Right-click it (or long-press, or press M) for its parents, children and paths.
</p>
<SimilarSection basis={nodeBasis(node)} />
{links.length > 0 && (
<>
<h3 className="pt-1 text-[12px] font-bold text-fg-secondary">Connected here</h3>
Expand Down Expand Up @@ -316,6 +330,7 @@ function ThingPanel({
<h3 className="pt-1 text-[12px] font-bold text-fg-secondary">{isSpeaker ? "Recordings" : "Mentions"}</h3>
{isSpeaker ? <SpeakerRecordings n={node} player={player} /> : <EntityMentions n={node} player={player} />}
<InlinePlayerBar player={player} />
<SimilarSection basis={nodeBasis(node)} />
{links.length > 0 && (
<>
<h3 className="pt-1 text-[12px] font-bold text-fg-secondary">Connected</h3>
Expand Down
1 change: 1 addition & 0 deletions nextjs-frontend/components/recording/context.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ export type PanelTab =
| "iiif"
| "details"
| "files"
| "similar"
| "shots"
| "text"
| "people"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -762,6 +762,7 @@ export function DocumentLayout({ compact }: { compact: boolean }) {
{ value: "pages", label: "Text", count: model.segments.length || undefined },
{ value: "summary", label: "Summary" },
{ value: "entities", label: "Entities" },
{ value: "similar", label: "Similar" },
{ value: "chat", label: "Chat" },
{ value: "notes", label: "Notes", count: notes || undefined },
{ value: "comments", label: "Comments", count: open || undefined },
Expand Down
3 changes: 2 additions & 1 deletion nextjs-frontend/components/recording/mobile-layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ export function MobileLayout() {
const [lastTab, setLastTab] = useState(tab);
if (tab !== lastTab) {
setLastTab(tab);
if (["chat", "notes", "comments", "highlights", "history", "details", "files"].includes(tab)) setSheet(true);
if (["chat", "notes", "comments", "highlights", "history", "details", "files", "similar"].includes(tab))
setSheet(true);
}

return (
Expand Down
Loading
Loading