Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[checks](docs/requirements.md#connect-your-agents))
- **Ask, and get cited answers.** A chat assistant on every page, and by voice, searches your archive by words (with
"phrases", OR, and -word to leave lines out) and by meaning, once turned on, reads it, cites it and acts on it with
tools: changes wait for your approval. Ask it about Lens itself ("what's the MCP address?", "is Telegram on?") and it
tools: changes wait for your approval. A search result opens the asset with Find showing the words. Ask it about Lens itself ("what's the MCP address?", "is Telegram on?") and it
answers from Lens's docs and this server, citing both. A Tools chip picks which tools an answer may use, how many
calls it may make and which namespaces it reads. Once an admin turns them on, it also searches the web and reads
pages, cited like the archive, and runs Python and bash in a sandbox of each conversation's own (on a Docker install,
Expand Down
4 changes: 4 additions & 0 deletions changelog.d/search-find.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **A search result opens the asset with its words found.** Results link with `?q=`, the words searched for (not
filters or left-out words), so the asset opens with Find in its text showing them, on the match at the moment the
result was, or on the longest of the words when the text doesn't have them together
([Search](docs/api.md#search)).
4 changes: 4 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1253,6 +1253,10 @@ asset (`facets`: up to 50 values each, most first, and `moments`), and lists the
assets they're in, with how many of those assets each is in (`objects`); past 20,000 moments the counts cover
20,000 of them (`partial`).

In the web app, a result links to the asset with the words searched for (`/resources/<id>?t=<seconds>&q=<words>`,
without filters, left-out words or `OR`): it opens with Find in its text showing them, on the match at that moment,
or on the longest of the words when the text doesn't have them together.

Each hit's `source` says where it was found: `said` (a line of the asset's text or transcript), `screen` (text on screen), `page` (a document's or an
image's text: its `page`, from 0, and its `box` on it), `object` (a kind of object, where it's first seen: its time, or
on a document's pages its `page`, with the `frame` and the `box` it's best seen in), `described` (what a shot or a page
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Questions about Lens itself answered from Lens's docs and this server's facts, both cited, for people and agents over MCP | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_docs_are_found_by_words`, `test_the_assistant_answers_about_lens_itself`, `test_agents_ask_lens_over_mcp`, `test_without_a_question_lens_help_still_gives_the_server` |
| A chat assistant on every page | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_asking_from_a_page`; [page-chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/page-chat.test.tsx); [ask-place.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ask-place.test.tsx) |
| By voice | [assistant-mode.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/assistant-mode.test.tsx) `starts a voice chat from the mic`; [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_what_is_said_comes_back_as_text` |
| A search result opens the asset with Find showing the words searched for, at the moment it linked to | [search-find.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/search-find.test.ts) `carries them on the links from the results`, `finds the words together, else the longest word there, on the match at the linked moment` |
| It searches by words, and by meaning once search by meaning is set up | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_semantic.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_semantic.py) `test_chat_retrieval_by_meaning`, `test_off_or_unconfigured` |
| Word search takes "phrases", OR, and NOT or -word to leave lines out | [test_search_index.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_search_index.py) `test_not_and_minus_leave_lines_out`, `test_the_queries_parts` |
| Answers cite their sources, and only what the person may read | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_chat_cites_only_what_you_can_read`; [chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat.test.tsx) `splits [n], [n, m] and [n][m]` |
Expand Down
37 changes: 37 additions & 0 deletions nextjs-frontend/__tests__/search-find.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { hitHref } from "@/components/recording/files-model";
import { arrivalFind, type Segment } from "@/components/recording/model";
import { recordingHref, withFind } from "@/components/search/links";
import { findText } from "@/components/search/query";

const seg = (idx: number, t0: number, text: string): Segment => ({
idx,
t0,
t1: t0 + 1000,
speaker: null,
text,
emotion: null,
event: null,
});

describe("a search's words open the asset with them found", () => {
it("keeps only the words to find", () => {
expect(findText('budget speaker:"Host B" "next year" -draft NOT memo OR plan*')).toBe("budget next year plan");
expect(findText("ns:pods -only")).toBe("");
});

it("carries them on the links from the results", () => {
expect(withFind(recordingHref(4, 61_000), "budget -draft")).toBe("/resources/4?t=61&q=budget");
expect(withFind(recordingHref(4), "café au lait")).toBe("/resources/4?q=caf%C3%A9%20au%20lait");
const hit = { recording_id: 4, t0: null, file: null, line: null, page: 2 };
expect(withFind(hitHref(hit), "speaker:Mara")).toBe("/resources/4?page=3");
});

it("finds the words together, else the longest word there, on the match at the linked moment", () => {
const segs = [seg(0, 0, "The budget is late."), seg(1, 5000, "Next year's budget."), seg(2, 9000, "Budget again")];
expect(arrivalFind(segs, "budget", 5000)).toEqual({ query: "budget", index: 1 });
expect(arrivalFind(segs, "budget", null)).toEqual({ query: "budget", index: 0 });
expect(arrivalFind(segs, "next year", null)).toEqual({ query: "next year", index: 0 });
expect(arrivalFind(segs, "budget meeting", 9000)).toEqual({ query: "budget", index: 2 });
expect(arrivalFind(segs, "hiring plan", null)).toBeNull();
});
});
7 changes: 5 additions & 2 deletions nextjs-frontend/app/(app)/resources/[id]/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,23 +15,26 @@ type Props = {
file?: string | string[];
line?: string | string[];
page?: string | string[];
q?: string | string[];
}>;
};

/** A resource: a recording (R1–R9, VR1–VR3), a document or an image. `?t=<seconds>` opens it at that moment,
* `?page=<n>` a document on its nth page, `?file=<id>&line=<n>` on one of its files (search links lines of files
* without times that way). /recordings/<id> redirects here. */
* without times that way), `?q=<words>` with those words found (search results link that way). /recordings/<id>
* redirects here. */
export default async function RecordingRoute({ params, searchParams }: Props) {
const { id } = await params;
const rid = Number(id);
if (!Number.isInteger(rid) || rid <= 0) notFound();
const { t, file, line, page } = await searchParams;
const { t, file, line, page, q } = await searchParams;
return (
<RecordingPage
id={rid}
start={parseStart(t)}
focus={parseFileFocus(file, line)}
page={typeof page === "string" ? page : null}
find={typeof q === "string" ? q : null}
/>
);
}
20 changes: 20 additions & 0 deletions nextjs-frontend/components/recording/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -652,6 +652,26 @@ export function findInSegments(segments: Segment[], query: string): FindHit[] {
return out;
}

/** What Find opens with on arriving from a search (?q=): its words together when the text has them, else the longest
* of its words the text has; on the first match at or after `startMs` (the moment the search linked to). Null when
* the text has none of them. */
export function arrivalFind(
segments: Segment[],
q: string,
startMs: number | null,
): { query: string; index: number } | null {
const words = q.split(/\s+/).filter((w) => w.length >= 2);
const tries = [q.trim(), ...[...words].sort((a, b) => b.length - a.length)];
for (const query of tries) {
const hits = findInSegments(segments, query);
if (!hits.length) continue;
const t0 = new Map(segments.map((s) => [s.idx, s.t0]));
const index = startMs == null ? 0 : hits.findIndex((h) => (t0.get(h.seg) ?? 0) >= startMs - 1000);
return { query, index: Math.max(0, index) };
}
return null;
}

/** Lowercase and strip accents without changing the string's length (so offsets map back). */
export function fold(s: string): string {
let out = "";
Expand Down
28 changes: 26 additions & 2 deletions nextjs-frontend/components/recording/recording-page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,14 @@ import { pageAt, pageRef, parsePage } from "@/components/recording/document/mode
import { usePlayer, useRecording, useRecordingJobs } from "@/components/recording/hooks";
import { pageState } from "@/components/recording/jobs";
import { MobileLayout } from "@/components/recording/mobile-layout";
import { adjacentTurnStart, findInSegments, groupTurns, segmentAt, type EntityRef } from "@/components/recording/model";
import {
adjacentTurnStart,
arrivalFind,
findInSegments,
groupTurns,
segmentAt,
type EntityRef,
} from "@/components/recording/model";
import type { NoteDraft } from "@/components/recording/notes-model";
import { RecordingSkeleton } from "@/components/recording/skeleton";
import { RecordingDialogs, type DialogState } from "@/components/recording/dialogs";
Expand All @@ -35,13 +42,16 @@ export function RecordingPage({
start,
focus = null,
page = null,
find = null,
}: {
id: number;
start: number | null;
/** A file to open the Files tab on (?file=&line=). */
focus?: { file: number; line: number | null } | null;
/** A document's page to open on (?page=, from 1). */
page?: string | null;
/** Words to open with found (?q=, from a search). */
find?: string | null;
}) {
const rec = useRecording(id);
const jobs = useRecordingJobs(id);
Expand Down Expand Up @@ -98,6 +108,7 @@ export function RecordingPage({
start={start}
focus={focus}
page={page}
find={find}
rec={rec.data}
model={player.data}
state={state}
Expand Down Expand Up @@ -145,6 +156,7 @@ type InnerProps = {
start: number | null;
focus: RecordingCtx["fileFocus"];
page: string | null;
find: string | null;
rec: RecordingCtx["rec"];
model: RecordingCtx["model"];
state: RecordingCtx["state"];
Expand All @@ -153,7 +165,7 @@ type InnerProps = {
speakers: RecordingCtx["speakers"];
};

function Inner({ id, start, focus, page, rec, model, state, jobs, turns, speakers }: InnerProps) {
function Inner({ id, start, focus, page, find, rec, model, state, jobs, turns, speakers }: InnerProps) {
const api = usePlayerApi();
const { hasMedia } = usePlayerState();
const { roleIn, can, admin } = useArchive();
Expand Down Expand Up @@ -195,6 +207,18 @@ function Inner({ id, start, focus, page, rec, model, state, jobs, turns, speaker
api.seek(start * 1000, { manual: true });
}, [api, start]);

// ?q= (a search's words) opens Find with them found, on the match at or after ?t= when there is one.
const found = useRef(false);
useEffect(() => {
if (found.current || !find) return;
found.current = true;
const at = arrivalFind(model.segments, find, start == null ? null : start * 1000);
if (!at) return;
setQuery(at.query);
setHitIndex(at.index);
setFindOpen(true);
}, [find, start, model.segments]);

// #access (Home's "Needs attention" and access request emails link that way) opens the access settings.
useEffect(() => {
if (window.location.hash === "#access") setDialog({ kind: "access" });
Expand Down
7 changes: 7 additions & 0 deletions nextjs-frontend/components/search/links.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
import type { RecordingSummary, SearchHit } from "@/app/openapi-client/types.gen";
import { findText } from "@/components/search/query";

/** A link to a moment: the recording page, seeking to `ms` (as whole seconds, `?t=`). */
export function recordingHref(id: number, ms?: number | null): string {
return ms && ms > 0 ? `/resources/${id}?t=${Math.floor(ms / 1000)}` : `/resources/${id}`;
}

/** A link from a search to an asset, carrying what was searched for (`?q=`) so the asset opens with it found. */
export function withFind(href: string, q?: string | null): string {
const text = findText(q ?? "");
return text ? `${href}${href.includes("?") ? "&" : "?"}q=${encodeURIComponent(text)}` : href;
}

/** True when a recording has media to play (imported transcripts have none); undefined when unknown. */
export function hasMedia(r: Pick<RecordingSummary, "source" | "media_kind"> | undefined): boolean | undefined {
if (!r) return undefined;
Expand Down
16 changes: 16 additions & 0 deletions nextjs-frontend/components/search/query.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,22 @@ export function tokens(text: string): string[] {
return text.match(/-?"[^"]+"|\S+/g) ?? [];
}

/** The words a search looks for, as text to find in an asset (?q= on its page): without typed filters, left-out
* words (-word, NOT word), OR, quotes or a trailing *. Empty when nothing is left. */
export function findText(input: string): string {
const ts = tokens(parseQuery(input).text);
const out: string[] = [];
for (let i = 0; i < ts.length; i++) {
const up = ts[i].toUpperCase();
if (up === "NOT") i++;
else if (up !== "OR" && up !== "AND" && !ts[i].startsWith("-")) {
const w = ts[i].replace(/^"|"$/g, "").replace(/\*+$/, "").trim();
if (/[\p{L}\p{N}]/u.test(w)) out.push(w);
}
}
return out.join(" ");
}

/** True when there is anything to search for (not just OR or punctuation). */
export function hasTerms(text: string): boolean {
let negate = false;
Expand Down
21 changes: 16 additions & 5 deletions nextjs-frontend/components/search/results.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { forwardRef, type KeyboardEvent } from "react";
import type { SearchHit } from "@/app/openapi-client/types.gen";
import type { HitGroup } from "@/components/search/facets";
import { ROLE_LABEL, hitHref, type FileRole } from "@/components/recording/files-model";
import { foundAs, onPage, recordingHref } from "@/components/search/links";
import { foundAs, onPage, recordingHref, withFind } from "@/components/search/links";
import type { InlinePlayer } from "@/components/search/player";
import { splitSnippet } from "@/components/search/snippet";
import { speakerTone } from "@/components/speakers/format";
Expand Down Expand Up @@ -114,14 +114,16 @@ function HitRow({
first,
player,
audio,
q,
}: {
hit: SearchHit;
first: boolean;
player: InlinePlayer;
audio: boolean | undefined;
q?: string;
}) {
const router = useRouter();
const href = hitHref(hit);
const href = withFind(hitHref(hit), q);
const paged = onPage(hit);
const untimed = hit.t0 == null || paged;
const at = paged ? `Page ${(hit.page ?? 0) + 1}` : hit.t0 == null ? `Line ${(hit.line ?? 0) + 1}` : tc(hit.t0);
Expand Down Expand Up @@ -206,9 +208,11 @@ export const ResultGroups = forwardRef<
groups: HitGroup[];
player: InlinePlayer;
audioOf: (recordingId: number) => boolean | undefined;
/** What was searched for: each link opens the asset with it found. */
q?: string;
className?: string;
}
>(function ResultGroups({ groups, player, audioOf, className }, ref) {
>(function ResultGroups({ groups, player, audioOf, q, className }, ref) {
let first = true;
return (
<div ref={ref} data-hit-list role="list" aria-label="Search results" className={className}>
Expand All @@ -221,7 +225,7 @@ export const ResultGroups = forwardRef<
>
<header className="flex flex-wrap items-baseline gap-x-2.5 gap-y-0.5">
<Link
href={recordingHref(g.recordingId)}
href={withFind(recordingHref(g.recordingId), q)}
className="text-[15px] font-bold leading-snug text-fg hover:text-fg-accent hover:underline"
>
{g.title}
Expand All @@ -235,7 +239,14 @@ export const ResultGroups = forwardRef<
<div className="flex flex-col gap-1">
{g.hits.map((h) => {
const row = (
<HitRow key={String(h.id)} hit={h} first={first} player={player} audio={audioOf(h.recording_id)} />
<HitRow
key={String(h.id)}
hit={h}
first={first}
player={player}
audio={audioOf(h.recording_id)}
q={q}
/>
);
first = false;
return row;
Expand Down
2 changes: 1 addition & 1 deletion nextjs-frontend/components/search/search-page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ export function SearchPage() {
/>
)}
{groups.length > 0 && (
<ResultGroups ref={listRef} groups={groups} player={player} audioOf={audioOf} className="pt-1.5" />
<ResultGroups ref={listRef} groups={groups} player={player} audioOf={audioOf} q={q} className="pt-1.5" />
)}
{results.hasNextPage && (
<div className="flex justify-center py-5">
Expand Down
Loading