diff --git a/README.md b/README.md index 21d1ec24..6663022c 100644 --- a/README.md +++ b/README.md @@ -207,7 +207,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p [evidence PDFs](docs/api.md#evidence-pdfs), [published feeds](docs/feeds.md); [checks](docs/requirements.md#learn-from-it)) - **Automate it.** Pipelines and workflows on a canvas (with Convert steps that make a video's soundtrack an audio - asset and a document's pages image assets, and Read steps that pick their own converter, such as anytopdf), routines on a schedule, batch runs over a collection, and + asset and a document's pages image assets, and Read steps that pick their own converter, such as anytopdf; what their model steps write about + an asset is found by search, under the step's name), routines on a schedule, batch runs over a collection, and notifications to webhooks, Slack, Discord and Matterbridge, including when a routine's run fails or leaves graph changes to review. Everything that happens in Lens (new assets, emails, calendar events, files, feed posts, what people say to Lens, sensor readings, sensors switched on or off, notes, diff --git a/changelog.d/generated-search.md b/changelog.d/generated-search.md new file mode 100644 index 00000000..da59c505 --- /dev/null +++ b/changelog.d/generated-search.md @@ -0,0 +1,7 @@ +- **Search finds what a pipeline's model step wrote.** The words of the output an `llm` step saves on an asset (and of + a workflow's Save output node after an LLM prompt node) are found by search, the API's `GET /search` and MCP's + `search`, as `source: generated` with the step's name in `output_label`; the web app shows "Written by ". + Running the step again replaces them and deleting the asset removes them. A step keeps its output out of search with + *Find what it writes in search* unticked (`searchable: false`); it is on otherwise. The upgrade step + `generated-search` makes outputs a model wrote before searchable too + ([Pipelines and templates](docs/processing.md#pipelines-and-templates)). diff --git a/docs/api.md b/docs/api.md index 732f93df..f8bb02f4 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1300,7 +1300,8 @@ GET /api/v1/mentions ``` `GET /search?q=` finds the moments where the words appear (in an asset's text or transcript, on screen in a video, in a resource's -supplementary transcripts, captions, translations and indexes, or in what a model says its shots and pages show) in the +supplementary transcripts, captions, translations and indexes, in what a model says its shots and pages show, or in +what a pipeline's model step wrote about it) in the namespaces you can read, best first: every word (English stemming), "quoted phrases" as written, `OR` between alternatives, `NOT` or `-` before a word or phrase to leave out the lines with it; `ns`, `speaker`, `emotion`, `event` (a sound heard, such as laughter, applause or music, typed any way; music @@ -1319,8 +1320,11 @@ or on the longest of the words when the text doesn't have them together. Each hit's `source` says where it was found: `said` (a line of the asset's text or transcript), `screen` (text on screen), `page` (a document's or an image's text: its `page`, from 0, and its `box` on it), `object` (a kind of object, where it's first seen: its time, or on a document's pages its `page`, with the `frame` and the `box` it's best seen in), `described` (what a shot or a page -shows, [Descriptions](#descriptions): the shot's time, or the `page`, and its `frame`) or `file` (a line of a -supplementary file: its `file`, `file_role` and `file_label`, and which `line`). A file's lines have no `t0` when the +shows, [Descriptions](#descriptions): the shot's time, or the `page`, and its `frame`), `generated` (what a pipeline's +`llm` step, or a workflow's Save output node after an LLM prompt node, wrote about the asset: the output's +`output_key` and what wrote it, `output_label`, with no time or page; [Pipelines and +templates](processing.md#pipelines-and-templates)) or `file` (a line of a supplementary file: its `file`, `file_role` +and `file_label`, and which `line`). A file's lines have no `t0` when the file doesn't say when they are; the web app opens those in the resource's Files tab. A `speaker`, `emotion` or `event` filter keeps to `said` hits, and each hit says its `emotion` and `event`. diff --git a/docs/database.md b/docs/database.md index 455985ac..3a4d32dd 100644 --- a/docs/database.md +++ b/docs/database.md @@ -123,6 +123,10 @@ The steps so far, in the order they run: without a `head` is read as one line, and saved as one before anything is added to it. * `mcp-sealed-secrets`: the tokens and keys MCP servers kept in plain text in their headers and env are sealed, in every version ([Adding an MCP server](assistant.md#adding-an-mcp-server)); sealed ones are left as they are. +* `generated-search`: outputs a model wrote about an asset before search found them (an `output` whose `origin` names + a `model`, except a spec's note) get their words kept for search in `generated`, under their template's name + ([Pipelines and templates](processing.md#pipelines-and-templates)). Outputs that already have them are left as they + are, so it's safe to run twice. To add a step: diff --git a/docs/processing.md b/docs/processing.md index 48eb06c1..1445f673 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -814,7 +814,11 @@ audio skip themselves for imported text. Steps: - `clean_audio` ([Cleaning up audio](#cleaning-up-audio)), `transcribe`, `diarize`, `clean` ([Cleaning up transcripts](#cleaning-up-transcripts)), `analyze`, `summarize`, and `report` (the built-in report). - `llm`: renders a prompt template, sends it to the configured model, checks the reply against the template's JSON - Schema and saves it as a named output (e.g. `meeting_notes`). + Schema and saves it as a named output (e.g. `meeting_notes`). Search finds the words of that output (its text and + numbers, not its field names), as `source: generated` with the step's name (else the template's) in `output_label` + ([search](api.md#search)); running the step again replaces them, and deleting the asset removes them. Untick *Find + what it writes in search* (`searchable: false`) to keep an output out of search; it is on unless a step says so, + which changes nothing anyone relied on, since search didn't look at outputs before. - `spec`: writes a technical spec from the transcript into a note linked to the asset, with the Spec from transcript template unless the step names another prompt template ([Specs from transcripts](notes.md#specs-from-transcripts)). - `report` with a template: renders an HTML page into the namespace's reports. These pages may not run scripts. @@ -899,7 +903,8 @@ node passes what it makes along its connections. other methods. Asset and graph workflows have it; automations call web addresses with their own action, which asks first ([actions](automations.md#actions-and-intents)). Refine later: credentials passed in through a custom node's parameter aren't sealed, and a sealed header is bound to its host rather than to one workflow. -- **Keeping results:** Save output (`outputs.`), Set field (a custom field, kept in the metadata history), Save +- **Keeping results:** Save output (`outputs.`; when an LLM prompt node is before it, search finds its words + under the node's label, unless its *Find it in search* box is unticked, `searchable: false`), Set field (a custom field, kept in the metadata history), Save entities, and a Classify node set to act on a sure answer (it sets the content type, suggests the topic or adds the tag; a try on the canvas only says what it would do). diff --git a/docs/requirements.md b/docs/requirements.md index fc9ffed2..0ad675a6 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -255,6 +255,7 @@ request, to another check that proves the same thing; loosening a requirement or | Requirement | Checks | |---|---| | Pipelines and workflows on a canvas | [test_workflows.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_workflows.py) `test_workflows_run_in_pipelines_drawn_as_graphs`; [workflow-canvas.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-canvas.test.ts); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `pipeline-create`, `workflow-create` | +| What a pipeline's model step or a workflow's model node writes about an asset is found by search, by the API and MCP, under the step's name; running it again replaces those words, a step can opt out, deleting the asset removes them, and outputs written before are made searchable on upgrade | [test_generated_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_generated_search.py) `test_a_pipelines_model_step_is_found_by_search`, `test_searchable_is_an_llm_steps_setting`, `test_a_workflows_model_output_is_found_by_search`, `test_outputs_a_model_wrote_before_become_searchable`; [search.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/search.test.ts) `says where what wasn't said was found: on screen, on a page, seen, what a shot or page shows, or a model wrote`; [pipelines-templates.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipelines-templates.test.ts) `keeps an llm step's words in search unless it opts out` | | Convert steps make new assets: a video's soundtrack as audio, a document's pages as images, each run through its own pipeline | [test_conversions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversions.py) `test_a_videos_soundtrack_becomes_an_audio_asset`, `test_a_documents_pages_become_image_assets`, `test_settings_are_checked`; [pipeline-convert.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipeline-convert.test.ts) `is added from the library ready to run: to audio unless images are picked` | | A pipeline's Read step picks its own converter (auto, anytopdf or Lens's own) in place of the server's, checked as it's saved | [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_a_read_step_picks_its_own_converter`; [pipeline-convert.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipeline-convert.test.ts) `is kept only on a read step, and said in its summary` | | A Classify node in workflows chooses from labels, content types or topics, and acting on a sure answer keeps something | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_a_workflow_classifies`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `chooses from labels, content types or topics, and acting on a sure answer keeps something` | diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 562285d1..d4ee42a7 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -456,7 +456,8 @@ def _recording(ctx: Context, rid: int) -> dict[str, Any]: "search", "Search the archive", "Full-text search for moments in the archive: the text of every asset — speech in audio and video (transcripts), " - "text shown on screen in videos or written in documents and images, supplementary files, and kinds of object seen. Words are all required " + "text shown on screen in videos or written in documents and images, supplementary files, kinds of object seen, and what " + "pipelines' model steps wrote about assets (found: generated, with the step in output). Words are all required " '(matched after stemming), "quoted phrases" must appear as written, OR separates alternatives, NOT or a leading - leaves out lines with a word or phrase. Results are the best ' "moments first, each with the asset, the person (who said or wrote it), where in it, the matching text (matches in **bold**) " "and a url that opens the asset at that moment. Use get_transcript for the lines around a moment.", @@ -515,6 +516,7 @@ def search( "line": h.idx, "page": h.page + 1 if h.page is not None else None, "file": h.file_label, + "output": h.output_label, "text": _plain(h.snippet), "url": ctx.link(h.recording_id, h.t0), } diff --git a/fastapi_backend/app/api/v1/routes/search.py b/fastapi_backend/app/api/v1/routes/search.py index 6d59181c..85af1bb2 100644 --- a/fastapi_backend/app/api/v1/routes/search.py +++ b/fastapi_backend/app/api/v1/routes/search.py @@ -45,8 +45,8 @@ def search_transcripts( ), ) -> SearchResults: """Moments where the words appear (said in audio or video, shown on screen in a video, written in a document or in - a resource's supplementary transcripts, captions, translations and indexes, or the kinds of object seen in videos, - documents and images), + a resource's supplementary transcripts, captions, translations and indexes, the kinds of object seen in videos, + documents and images, what a model that can see said they show, or what a pipeline's model step wrote about them), best first, in the namespaces you can read and the collections you were given a role on. A person, emotion or sound filter keeps to their lines. By meaning, a moment is a passage about what was asked, though its words may differ.""" if ns: diff --git a/fastapi_backend/app/domain/deletion.py b/fastapi_backend/app/domain/deletion.py index fde47399..4846cc37 100644 --- a/fastapi_backend/app/domain/deletion.py +++ b/fastapi_backend/app/domain/deletion.py @@ -3,7 +3,7 @@ What goes: its transcript and everything derived from it (segments, speakers' appearances, chapters, entities' mentions, terms, edits), video shots, text on screen and face tracks with their frames, the pages drawn of documents -and images, outputs and reports, shares, +and images, outputs (and the words of them search finds) and reports, shares, notes, comments, highlights, supplementary files, permissions, requests for access, and its place in IP groups, fixed collections, chat scopes and batch runs that haven't started it. Speakers and faces that only it had, and that nobody named, go too; named ones stay. Its jobs are cancelled (a job that is running has to stop first), and harvesters hear a Delete when it was public. @@ -41,6 +41,7 @@ "face_track", "object_track", "description", + "generated", "share_link", "share_embed", "note", diff --git a/fastapi_backend/app/domain/migrations.py b/fastapi_backend/app/domain/migrations.py index f814009e..9e90d005 100644 --- a/fastapi_backend/app/domain/migrations.py +++ b/fastapi_backend/app/domain/migrations.py @@ -49,7 +49,7 @@ def steps(): """[(name, fn(db))] in the order they run. Append only.""" - from . import access, auth, chat, entities, hierarchy, jobs, mcp_servers + from . import access, auth, chat, entities, hierarchy, jobs, mcp_servers, pipelines return [ # IIIF-only access levels (public, transcript, signed-in, private) become the access setting @@ -66,6 +66,8 @@ def steps(): ("mcp-sealed-secrets", mcp_servers.migrate_seal), # entities the old rules called terms get the type the rules give them now (people, places, organisations...) ("entity-types", entities.migrate_types), + # what a model step wrote about a recording before then is found by search too + ("generated-search", pipelines.migrate_generated), ] diff --git a/fastapi_backend/app/domain/moving.py b/fastapi_backend/app/domain/moving.py index 7a428cbf..60db9a22 100644 --- a/fastapi_backend/app/domain/moving.py +++ b/fastapi_backend/app/domain/moving.py @@ -49,6 +49,7 @@ "face_track", "object_track", "description", + "generated", "job", "output", "share_link", diff --git a/fastapi_backend/app/domain/pipelines.py b/fastapi_backend/app/domain/pipelines.py index 0c5fc850..e78b53bd 100644 --- a/fastapi_backend/app/domain/pipelines.py +++ b/fastapi_backend/app/domain/pipelines.py @@ -5,7 +5,8 @@ image's pages and their text), convert (new assets made from one: a video's soundtrack as audio, a document's pages as images, conversions.py), diarize, shots, ocr, faces, objects, nsfw (NSFW classification, nsfw.py), describe, clean (a transcript tidied before analysis, cleanup.py), analyze, embed (passages for search by meaning, semantic.py), -summarize, llm (a prompt template whose structured result is saved as a named output), spec (a technical spec from the +summarize, llm (a prompt template whose structured result is saved as a named output, its words found by search unless +the step says `searchable: false`), spec (a technical spec from the transcript, written to a note linked to the recording, specs.py), reply (a cited reply drafted to an email, responder.py), report (built in, or from a report template) and export (a template rendered to a file, optionally copied to a storage source) and workflow (a workflow graph, @@ -73,8 +74,10 @@ "format", "pipeline", "converter", + "searchable", } -ONLY = {"to": "convert", "format": "convert", "pipeline": "convert", "converter": "read"} # settings only one type of step has +# settings only one type of step has +ONLY = {"to": "convert", "format": "convert", "pipeline": "convert", "converter": "read", "searchable": "llm"} CONVERTERS = ("auto", "anytopdf", "lens") # what a read step makes PDFs with, as documents.converter says NODE_RX = re.compile(r"^[A-Za-z0-9_-]{1,40}$") WHEN = {"min_minutes": (int, float), "max_minutes": (int, float), "source": str, "languages": list} @@ -127,6 +130,8 @@ def validate_steps(db, steps): raise ValueError(f"{t} step needs a {need} template, not {tpl['kind']}") if t == "llm" and not KEY_RX.match(str(s.get("key") or "")): raise ValueError("llm step: name the output with lowercase letters, digits and _ (e.g. meeting_notes)") + if "searchable" in s and not isinstance(s["searchable"], bool): + raise ValueError("llm step: searchable is true or false") if t == "export": if not str(s.get("filename") or "").strip(): raise ValueError("export step: give a file name, e.g. {{ recording.title }}.md") @@ -347,22 +352,96 @@ def condition_ok(db, rid, when): ) -def save_output(db, rid, key, value, origin=None): +GENERATED_MAX = 100_000 # characters of an output kept for search + + +def output_text(value): + """The words of an output, for search: its text and numbers, one per line, in order (not its field names).""" + out: list[str] = [] + + def walk(v): + if isinstance(v, str): + if v.strip(): + out.append(" ".join(v.split())) + elif isinstance(v, (int, float)) and not isinstance(v, bool): + out.append(str(v)) + elif isinstance(v, dict): + for x in v.values(): + walk(x) + elif isinstance(v, list): + for x in v: + walk(x) + + walk(value) + return "\n".join(out)[:GENERATED_MAX] + + +def save_output(db, rid, key, value, origin=None, found_as=None): + """Keep a named output of a recording. With `found_as` (who wrote it: the step's or node's name), its words are + kept for search too (`generated`, search.py), in place of what this output had before; without it, nothing of this + output is searchable.""" db.q( "UPSERT $o CONTENT $d", o=R("output", f"{rid}-{key}"), d=store.clean({"recording": rid, "key": key, "value": value, "origin": origin, "created_at": store.now()}), ) + keep_words(db, rid, key, value, found_as) + + +def keep_words(db, rid, key, value, found_as): + """Keep an output's words for search under `found_as`, or with None (or no words), drop what was kept.""" + text = output_text(value) if found_as else "" + g = R("generated", f"{rid}-{key}") + if not text: + db.q("DELETE $g", g=g) + return + db.q( + "UPSERT $g CONTENT $d", + g=g, + d={"recording": rid, "space": _space_of(db, rid), "key": key, "label": found_as, "text": text, "created_at": store.now()}, + ) def run_llm(db, cfg, rid, spec, say): t = templates.get(db, int(spec["template"]), spec.get("version")) prompt = templates.render_body(t["body"], templates.context(db, cfg, rid), "prompt") value = llm.json_out(cfg, t.get("system") or templates.DEFAULT_SYSTEM, prompt, t.get("schema") or {"type": "object"}, spec.get("model")) - save_output( - db, rid, spec["key"], value, {"template": t["id"], "version": t["version"], "model": spec.get("model") or cfg["llm"]["model"]} - ) - say(f"saved output {spec['key']}") + origin = {"template": t["id"], "version": t["version"], "model": spec.get("model") or cfg["llm"]["model"]} + found_as = (spec.get("name") or t["name"]) if spec.get("searchable", True) else None + save_output(db, rid, spec["key"], value, origin, found_as) + say(f"saved output {spec['key']}" + ("" if found_as else " (not searchable)")) + + +def migrate_generated(db): + """Upgrade step: what a model wrote about a recording before outputs were searchable is found by search too, under + the name of the template that asked for it. Outputs a model made that aren't its words (a spec's note) stay out. + Safe to run twice: outputs already searchable are left as they are.""" + from . import specs + + names: dict = {} + last = None + while True: + rows = db.rows( + "SELECT id, record::id(id) AS oid, recording, key, value, origin FROM output WHERE origin.model != NONE" + + (" AND id > $last" if last is not None else "") + + " ORDER BY id LIMIT 500", + last=last, + ) + for o in rows: + if o.get("key") == specs.KEY or db.one("SELECT id FROM $g", g=R("generated", o["oid"])): + continue + if not db.one("SELECT id FROM $r", r=R("recording", o["recording"])): + continue + tid = (o.get("origin") or {}).get("template") + if tid not in names: + try: + names[tid] = templates.get(db, int(tid))["name"] + except (KeyError, TypeError, ValueError): + names[tid] = None + keep_words(db, o["recording"], o["key"], o.get("value"), names[tid] or o["key"]) + if len(rows) < 500: + break + last = rows[-1]["id"] def _space_name(db, rid): diff --git a/fastapi_backend/app/domain/search.py b/fastapi_backend/app/domain/search.py index 5019f1bd..1e0ccb36 100644 --- a/fastapi_backend/app/domain/search.py +++ b/fastapi_backend/app/domain/search.py @@ -113,7 +113,8 @@ def search( ): """Transcript lines (and, unless screen is false, text on screen in videos; with files, the lines of supplementary transcripts, captions, translations and indexes; with objects, the kinds of object seen in videos, documents and - images; with described, what a model that can see said their pages and shots show) matching q. spaces limits the search to namespaces someone may read, and `also` adds recordings they may + images; with described, what a model that can see said their pages and shots show, and what a pipeline's model step + wrote about them) matching q. spaces limits the search to namespaces someone may read, and `also` adds recordings they may read beyond those (in collections they were given a role on); recordings limits it to a set of recordings (such as the transcripts a visitor may read), and obj to those a kind of object is seen in. event keeps to lines with that sound (Laughter, Applause, BGM …). With facets, also how many of all the matching moments (up to FACET_CAP) are in @@ -206,6 +207,7 @@ def search( hits += _objects(db, groups, space_filter(ns, spaces, recording, params), cap, params) if described and alone: # nor what pages and shots show hits += _described(db, groups, space_filter(ns, spaces, recording, params), cap, params) + hits += _generated(db, groups, space_filter(ns, spaces, recording, params), cap, params) hits.sort(key=lambda r: (-r["_score"], r["recording"], r.get("idx") or 0)) meant = [] if used != "keyword": @@ -282,6 +284,7 @@ def search( **({"t0": None, "t1": None, "page": h["t0"]} if h["source"] in ("object", "described") and h.get("paged") else {}), **({"page": h["page"], "box": h.get("box")} if h["source"] == "page" else {}), **(_in_file(in_files.get(h["file"]) or {"id": h["file"]}, h.get("line")) if h["source"] == "file" else {}), + **({"output_key": h["key"], "output_label": h.get("label") or h["key"]} if h["source"] == "generated" else {}), "match": h.get("match") or "words", "similarity": round(h["similarity"], 3) if h.get("similarity") is not None else None, } @@ -511,7 +514,11 @@ def _facets(db, groups, where_f, base, screen, ns, spaces, recording, files=Fals else [] ) shown = ( - _matches(db, groups, "description", "recording, space", space_filter(ns, spaces, recording, base), base) + [ + r + for t in ("description", "generated") + for r in _matches(db, groups, t, "recording, space", space_filter(ns, spaces, recording, base), base) + ] if described and words else [] ) @@ -596,6 +603,11 @@ def _described(db, groups, where_f, cap, base): return _layer(db, groups, where_f, cap, base, "description", "frame, paged", "described") +def _generated(db, groups, where_f, cap, base): + """Hits in what a pipeline's model step (or a workflow's model node) wrote about a recording: one per output.""" + return _layer(db, groups, where_f, cap, base, "generated", "key, label", "generated") + + def _file_lines(db, groups, where_f, cap, base): """Hits in the lines of supplementary files, ranked alongside the transcript.""" return _layer(db, groups, where_f, cap, base, "file_line", "file, idx AS line", "file") diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index a5cc36ab..db550def 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -1743,6 +1743,11 @@ def close(self): # what a model that can see images says a page or a shot shows (descriptions.py) "DEFINE TABLE IF NOT EXISTS description SCHEMALESS", "DEFINE INDEX IF NOT EXISTS description_rec ON description FIELDS recording", + # the words a pipeline's model step (or a workflow's model node) wrote about a recording, for search: one row per + # output, generated:⟨-⟩ (pipelines.keep_words) + "DEFINE TABLE IF NOT EXISTS generated SCHEMALESS", + "DEFINE INDEX IF NOT EXISTS generated_rec ON generated FIELDS recording", + "DEFINE INDEX IF NOT EXISTS generated_space ON generated FIELDS space", "DEFINE TABLE IF NOT EXISTS face_merge SCHEMALESS", # passages embedded for search by meaning (app/domain/semantic.py), and which model's vectors they hold; their # HNSW index is defined when the first vector is stored, since its dimension is the model's @@ -1839,13 +1844,15 @@ def _analyzer(cfg): return f"DEFINE ANALYZER IF NOT EXISTS archive_text TOKENIZERS class FILTERS lowercase, ascii{stem}" -# the full-text indexes, each on the `text` of its table: transcripts, text on screen, and lines of supplementary files +# the full-text indexes, each on the `text` of its table: transcripts, text on screen, lines of supplementary files, the +# kinds of object seen, what pages and shots show, and what a model step wrote TEXT_INDEXES = ( ("segment_text", "segment"), ("ocr_text", "ocr_span"), ("file_text", "file_line"), ("object_text", "object_track"), ("description_text", "description"), + ("generated_text", "generated"), ) diff --git a/fastapi_backend/app/domain/workflows.py b/fastapi_backend/app/domain/workflows.py index e3a05604..dcf98e26 100644 --- a/fastapi_backend/app/domain/workflows.py +++ b/fastapi_backend/app/domain/workflows.py @@ -12,7 +12,8 @@ ("Name|TYPE") and your regular expressions. Passes on a list of entities, each with the line it was found on. - `extract_llm`: asks the model for the entities in the transcript, of the types you name; entities passed in are given to it as what was found so far, to keep, correct or add to. -- `output`: saves the value as a named output of the recording. +- `output`: saves the value as a named output of the recording. When it came from an `llm` node, search finds its + words too, under the node's label (else the prompt template's name), unless `searchable` is false. - `field`: sets a custom field of the recording to the value, checked like an edit (and kept in its history). - `save_entities`: makes the entities passed in the recording's entities (in place of what analyze found), with people's corrections kept, then redoes keywords and sections as analyze does. @@ -62,7 +63,7 @@ "llm": {"template", "version", "model"}, "extract_rules": {"builtin", "terms", "patterns", "types"}, "extract_llm": {"types", "instructions", "model"}, - "output": {"key"}, + "output": {"key", "searchable"}, "field": {"field"}, "save_entities": set(), "classify": {"labels", "vocabulary", "question", "of", "apply"}, @@ -143,6 +144,8 @@ def _check_config(db, nid, t, cfg): elif t == "output": if not pipelines.KEY_RX.match(str(cfg.get("key") or "")): raise ValueError(f"output node {nid}: name it with lowercase letters, digits and _ (e.g. meeting_notes)") + if "searchable" in cfg and not isinstance(cfg["searchable"], bool): + raise ValueError(f"output node {nid}: searchable is true or false") elif t == "field": try: fieldmod.get(db, int(cfg.get("field") or 0)) @@ -394,7 +397,12 @@ def _run_node(r, f, node, value, vals): if r.dry: r.say(f"would save output {c['key']}") else: - pipelines.save_output(db, rid, c["key"], value, {**r.origin, **f.upstream(nid, f.made)}) + made = f.upstream(nid, f.made) + found_as = None + if made.get("model") and c.get("searchable", True): # what a model wrote: search finds it + tpl = db.one("SELECT name FROM $t", t=R("template", made["template"])) or {} + found_as = node.get("label") or tpl.get("name") or c["key"] + pipelines.save_output(db, rid, c["key"], value, {**r.origin, **made}, found_as) r.say(f"saved output {c['key']}") elif t == "field": fd = fieldmod.get(db, int(c["field"])) diff --git a/fastapi_backend/app/schemas/find.py b/fastapi_backend/app/schemas/find.py index 0f75cfae..5da07a89 100644 --- a/fastapi_backend/app/schemas/find.py +++ b/fastapi_backend/app/schemas/find.py @@ -63,7 +63,7 @@ class FindMoment(ResponseModel): last_page: int | None = Field(None, description="a face over several pages: the last of them") text: str | None = Field(None, description="the words there, when it has any") speaker: str | None = None - source: str | None = Field(None, description="words: said, screen, file, page, object or described (as search has it)") + source: str | None = Field(None, description="words: said, screen, file, page, object, described or generated (as search has it)") class FindReason(ResponseModel): diff --git a/fastapi_backend/app/schemas/search.py b/fastapi_backend/app/schemas/search.py index fa829775..60bf0267 100644 --- a/fastapi_backend/app/schemas/search.py +++ b/fastapi_backend/app/schemas/search.py @@ -21,12 +21,13 @@ class SearchHit(ResponseModel): recorded_at: str | None = None namespace: str | None = None snippet: str = Field(description="HTML: escaped text with around matches") - source: Literal["said", "screen", "file", "page", "object", "described"] = Field( + source: Literal["said", "screen", "file", "page", "object", "described", "generated"] = Field( description='"said" (a line of a transcript or other text), "screen" (text on screen in a video), "file" (a line of a ' "supplementary file), " '"page" (text on a page of a document or an image, whose times are only a reading pace), "object" (a kind of ' - "object seen in a video, a document or an image, where it's first seen; its snippet is the kind) or " - '"described" (what a model that can see said a shot of a video, or a page, shows)' + "object seen in a video, a document or an image, where it's first seen; its snippet is the kind), " + '"described" (what a model that can see said a shot of a video, or a page, shows) or "generated" (what a ' + "pipeline's model step, or a workflow's model node, wrote about the asset: one of its outputs, with no time or page)" ) frame: str | None = Field(None, description="screen, object and described hits: signed link to the video frame or the page") box: Any = Field(None, description="screen, page and object hits: where it is on the frame or the page ([x, y, w, h] fractions)") @@ -35,6 +36,8 @@ class SearchHit(ResponseModel): file_role: str | None = Field(None, description="file hits: its role (transcript, captions, translation or index)") file_label: str | None = Field(None, description="file hits: its label, or its name") line: int | None = Field(None, description="file hits: which of its lines (from 0)") + output_key: str | None = Field(None, description="generated hits: the output it is (outputs. on the asset)") + output_label: str | None = Field(None, description="generated hits: what wrote it: the step's or node's name, else its template's") match: Literal["words", "meaning", "both"] = Field( "words", description='how it was found: by the query\'s "words" (BM25), by "meaning" (a passage an embedding model finds ' diff --git a/fastapi_backend/tests/api/test_generated_search.py b/fastapi_backend/tests/api/test_generated_search.py new file mode 100644 index 00000000..dab39f6e --- /dev/null +++ b/fastapi_backend/tests/api/test_generated_search.py @@ -0,0 +1,146 @@ +"""What a pipeline's model step (or a workflow's model node) writes about an asset is found by search, as "generated" +with the step's name; running it again replaces those words, opting out or deleting the asset takes them away, and +outputs a model wrote before this are made searchable by an upgrade step.""" + +from __future__ import annotations + +import pytest + +from app.domain import llm as llmmod +from app.domain import migrations, pipelines, store, workflows +from tests import fake_llm +from tests.api.test_mcp import rpc +from tests.api.test_oauth import _bearer, _grant +from tests.helpers import drain, login, make_user + +R = store.R +CALL = "Alice: Samples are packed.\nDave: Courier at noon.\nAlice: Great." # no "capsid": only the model's notes say it + + +@pytest.fixture +def llm(cfg): + fake_llm.Handler.tool_script, fake_llm.Handler.reject_tools = [], False + srv, url = fake_llm.start() + cfg["llm"].update(base_url=url, model="fake") + cfg["ai"]["tools"] = False + yield fake_llm.Handler + srv.shutdown() + + +@pytest.fixture +def app(cfg, db, llm): + from app.main import create_app + + return create_app(cfg, db, background=False) + + +@pytest.fixture +def h(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +def _found(client, h, q): + hits = client.get("/api/v1/search", params={"q": q, "mode": "keyword"}, headers=h).json()["hits"] + return [x for x in hits if x["source"] == "generated"] + + +def test_a_pipelines_model_step_is_found_by_search(client, db, cfg, h, monkeypatch): + tl = {t["name"]: t["id"] for t in client.get("/api/v1/templates", headers=h).json()} + step = {"type": "llm", "name": "Call notes", "template": tl["Meeting notes"], "key": "meeting_notes"} + pid = client.post("/api/v1/pipelines", headers=h, json={"name": "Notes", "steps": ["analyze", step]}).json()["id"] + assert client.patch("/api/v1/namespaces/calls", headers=h, json={"pipeline": pid}).status_code == 200 + r = client.post("/api/v1/import", headers=h, json={"namespace": "calls", "title": "Courier call", "text": CALL}).json() + rid = r["id"] + drain(db, cfg) + assert client.get(f"/api/v1/jobs/{r['job']}", headers=h).json()["status"] == "succeeded" + + (hit,) = _found(client, h, "capsid") # the fake model's notes: "Capsid samples ship Friday." + assert (hit["recording_id"], hit["output_key"], hit["output_label"]) == (rid, "meeting_notes", "Call notes") + assert "Capsid" in hit["snippet"] and hit["t0"] is None and hit["page"] is None + assert hit["title"] == "Courier call" and hit["namespace"] == "calls" + facets = client.get("/api/v1/search", params={"q": "capsid", "mode": "keyword", "facets": True}, headers=h).json()["facets"] + assert facets["recordings"] == [{"id": rid, "title": "Courier call", "count": 1}] + + # agents: the MCP search tool lists it like the others, saying what wrote it + _, tokens = _grant(client, h, scope="read") + res = rpc(client, _bearer(tokens), "tools/call", {"name": "search", "arguments": {"query": "capsid"}})["result"] + (m,) = [x for x in res["structuredContent"]["results"] if x["found"] == "generated"] + assert (m["recording_id"], m["output"]) == (rid, "Call notes") and "Capsid" in m["text"] + + # running the step again replaces what it wrote + monkeypatch.setattr(llmmod, "json_out", lambda *a, **k: {"tldr": "The vaccine trial moves to March.", "decisions": []}) + pipelines.run_llm(db, cfg, rid, step, print) + assert _found(client, h, "capsid") == [] + assert [x["output_label"] for x in _found(client, h, "vaccine")] == ["Call notes"] + + # a step that says searchable: false keeps its output off search + pipelines.run_llm(db, cfg, rid, {**step, "searchable": False}, print) + assert _found(client, h, "vaccine") == [] + assert client.get(f"/api/v1/recordings/{rid}/outputs", headers=h).json()[0]["value"]["tldr"].startswith("The vaccine") + pipelines.run_llm(db, cfg, rid, step, print) + assert len(_found(client, h, "vaccine")) == 1 + + # deleting the asset takes its words out of search + assert client.delete(f"/api/v1/recordings/{rid}", headers=h).status_code == 200 + assert _found(client, h, "vaccine") == [] + assert db.rows("SELECT * FROM generated") == [] + + +def test_searchable_is_an_llm_steps_setting(client, db, h): + tl = {t["name"]: t["id"] for t in client.get("/api/v1/templates", headers=h).json()} + ok = {"type": "llm", "template": tl["Meeting notes"], "key": "notes", "searchable": False} + assert pipelines.validate_steps(db, [ok])[0]["searchable"] is False + with pytest.raises(ValueError, match="no setting searchable"): + pipelines.validate_steps(db, [{"type": "analyze", "searchable": True}]) + with pytest.raises(ValueError, match="true or false"): + pipelines.validate_steps(db, [{**ok, "searchable": "yes"}]) + + +def test_a_workflows_model_output_is_found_by_search(client, db, cfg, h): + tl = {t["name"]: t["id"] for t in client.get("/api/v1/templates", headers=h).json()} + graph = { + "nodes": [ + {"id": "in", "type": "input", "config": {}}, + {"id": "ask", "type": "llm", "config": {"template": tl["Meeting notes"]}}, + {"id": "keep", "type": "output", "config": {"key": "notes"}, "label": "Notes node"}, + {"id": "quiet", "type": "output", "config": {"key": "quiet", "searchable": False}}, + {"id": "title", "type": "pick", "config": {"path": "tldr"}}, + ], + "edges": [ + {"source": "in", "target": "ask"}, + {"source": "ask", "target": "keep"}, + {"source": "ask", "target": "quiet"}, + {"source": "in", "target": "title"}, + ], + } + graph["nodes"].append({"id": "plain", "type": "output", "config": {"key": "plain"}}) + graph["edges"].append({"source": "title", "target": "plain"}) # not from a model: not searchable + wid = workflows.create(db, "Notes", graph) + rid = client.post("/api/v1/import", headers=h, json={"namespace": "calls", "title": "Courier call", "text": CALL}).json()["id"] + drain(db, cfg) + workflows.run(db, cfg, rid, wid, say=print) + assert {o["key"] for o in client.get(f"/api/v1/recordings/{rid}/outputs", headers=h).json()} >= {"notes", "quiet"} + assert [(x["output_key"], x["output_label"]) for x in _found(client, h, "capsid")] == [("notes", "Notes node")] + assert db.values("SELECT VALUE key FROM generated") == ["notes"] + + +def test_outputs_a_model_wrote_before_become_searchable(client, db, cfg, h): + tl = {t["name"]: t["id"] for t in client.get("/api/v1/templates", headers=h).json()} + rid = client.post("/api/v1/import", headers=h, json={"namespace": "calls", "title": "Old call", "text": CALL}).json()["id"] + drain(db, cfg) + old = {"template": tl["Meeting notes"], "version": 1, "model": "fake"} + rows = [ # as an older Lens left them: outputs, nothing kept for search + ("meeting_notes", {"tldr": "Capsid samples ship Friday."}, old), + ("spec", {"note": 3, "title": "Capsid spec"}, old), # a spec's note: not the model's words + ("report_brief", {"url": "/reports/calls/capsid.html"}, {"template": tl["One-page brief"], "version": 1}), + ] + for key, value, origin in rows: + db.q("UPSERT $o CONTENT $d", o=R("output", f"{rid}-{key}"), d={"recording": rid, "key": key, "value": value, "origin": origin}) + assert _found(client, h, "capsid") == [] + step = dict(migrations.steps())["generated-search"] + step(db) + once = db.rows("SELECT * FROM generated") + step(db) # safe to run twice + assert db.rows("SELECT * FROM generated") == once + assert [(x["output_key"], x["output_label"]) for x in _found(client, h, "capsid")] == [("meeting_notes", "Meeting notes")] diff --git a/fastapi_backend/tests/domain/test_migrations.py b/fastapi_backend/tests/domain/test_migrations.py index bbeb00c5..d236e49f 100644 --- a/fastapi_backend/tests/domain/test_migrations.py +++ b/fastapi_backend/tests/domain/test_migrations.py @@ -22,6 +22,7 @@ "chat-tree", "mcp-sealed-secrets", "entity-types", + "generated-search", ] diff --git a/nextjs-frontend/__tests__/pipelines-templates.test.ts b/nextjs-frontend/__tests__/pipelines-templates.test.ts index 1f085cef..7f2c02f1 100644 --- a/nextjs-frontend/__tests__/pipelines-templates.test.ts +++ b/nextjs-frontend/__tests__/pipelines-templates.test.ts @@ -80,6 +80,23 @@ describe("pipeline specs", () => { }); expect(sameSteps(["analyze"], [{ type: "analyze" }])).toBe(true); }); + it("keeps an llm step's words in search unless it opts out", () => { + expect(cleanSpec({ type: "llm", template: 1, key: "notes", searchable: true })).toEqual({ + type: "llm", + template: 1, + key: "notes", + }); + expect(cleanSpec({ type: "llm", template: 1, key: "notes", searchable: false })).toEqual({ + type: "llm", + template: 1, + key: "notes", + searchable: false, + }); + expect(cleanSpec({ type: "analyze", searchable: false })).toBe("analyze"); + expect(stepSummary({ type: "llm", template: 1, key: "notes", searchable: false }, () => "Meeting notes")).toBe( + "Meeting notes · outputs.notes · not in search", + ); + }); it("describes settings and conditions", () => { expect(whenText({ min_minutes: 5 })).toBe("only if > 5 min"); expect( diff --git a/nextjs-frontend/__tests__/search.test.ts b/nextjs-frontend/__tests__/search.test.ts index 094b16fd..cfa3319a 100644 --- a/nextjs-frontend/__tests__/search.test.ts +++ b/nextjs-frontend/__tests__/search.test.ts @@ -228,7 +228,7 @@ describe("facets and groups", () => { expect(recordingHref(12, 0)).toBe("/resources/12"); }); - it("says where what wasn't said was found: on screen, on a page, seen, or what a shot or page shows", () => { + it("says where what wasn't said was found: on screen, on a page, seen, what a shot or page shows, or a model wrote", () => { expect(foundAs({ source: "said", page: null })).toBeNull(); expect(foundAs({ source: "screen", page: null })).toBe("On screen"); expect(foundAs({ source: "page", page: 2 })).toBe("On the page"); @@ -237,6 +237,8 @@ describe("facets and groups", () => { expect(foundAs({ source: "described", page: null })).toBe("What the shot shows"); expect(foundAs({ source: "described", page: 1 })).toBe("What the page shows"); expect(foundAs({ source: "file", page: null })).toBeNull(); + expect(foundAs({ source: "generated", page: null, output_label: "Call notes" })).toBe("Written by Call notes"); + expect(foundAs({ source: "generated", page: null })).toBe("Written by a model step"); expect([onPage({ source: "described", page: 1 }), onPage({ source: "described", page: null })]).toEqual([ true, false, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index db343b12..9e5c04f9 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -7871,8 +7871,8 @@ export class Search { * Search Transcripts * * Moments where the words appear (said in audio or video, shown on screen in a video, written in a document or in - * a resource's supplementary transcripts, captions, translations and indexes, or the kinds of object seen in videos, - * documents and images), + * a resource's supplementary transcripts, captions, translations and indexes, the kinds of object seen in videos, + * documents and images, what a model that can see said they show, or what a pipeline's model step wrote about them), * best first, in the namespaces you can read and the collections you were given a role on. A person, emotion or * sound filter keeps to their lines. By meaning, a moment is a passage about what was asked, though its words may differ. */ diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 83343054..2e2fc641 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -9467,7 +9467,7 @@ export type FindMoment = { /** * Source * - * words: said, screen, file, page, object or described (as search has it) + * words: said, screen, file, page, object, described or generated (as search has it) */ source?: string | null; [key: string]: unknown; @@ -24201,9 +24201,9 @@ export type SearchHit = { /** * Source * - * "said" (a line of a transcript or other text), "screen" (text on screen in a video), "file" (a line of a supplementary file), "page" (text on a page of a document or an image, whose times are only a reading pace), "object" (a kind of object seen in a video, a document or an image, where it's first seen; its snippet is the kind) or "described" (what a model that can see said a shot of a video, or a page, shows) + * "said" (a line of a transcript or other text), "screen" (text on screen in a video), "file" (a line of a supplementary file), "page" (text on a page of a document or an image, whose times are only a reading pace), "object" (a kind of object seen in a video, a document or an image, where it's first seen; its snippet is the kind), "described" (what a model that can see said a shot of a video, or a page, shows) or "generated" (what a pipeline's model step, or a workflow's model node, wrote about the asset: one of its outputs, with no time or page) */ - source: "said" | "screen" | "file" | "page" | "object" | "described"; + source: "said" | "screen" | "file" | "page" | "object" | "described" | "generated"; /** * Frame * @@ -24246,6 +24246,18 @@ export type SearchHit = { * file hits: which of its lines (from 0) */ line?: number | null; + /** + * Output Key + * + * generated hits: the output it is (outputs. on the asset) + */ + output_key?: string | null; + /** + * Output Label + * + * generated hits: what wrote it: the step's or node's name, else its template's + */ + output_label?: string | null; /** * Match * diff --git a/nextjs-frontend/components/pipelines/pipeline-model.ts b/nextjs-frontend/components/pipelines/pipeline-model.ts index d78b4dfe..78d5b099 100644 --- a/nextjs-frontend/components/pipelines/pipeline-model.ts +++ b/nextjs-frontend/components/pipelines/pipeline-model.ts @@ -21,6 +21,8 @@ export type StepSpec = { destination?: { source: number; path?: string }; model?: string; force?: boolean; + /** An llm step: false keeps what it writes out of search (searchable unless it says so). */ + searchable?: boolean; workflow?: number; /** A convert step: what the new assets are, their audio format, and the pipeline they run ("none": kept as they are). */ to?: "audio" | "images"; @@ -90,6 +92,7 @@ export function cleanSpec(s: StepSpec): StepSpec | string { }; if (s.model?.trim()) out.model = s.model.trim(); if (s.force) out.force = true; + if (s.type === "llm" && s.searchable === false) out.searchable = false; if (s.type === "convert") { if (s.to) out.to = s.to; if (s.to === "audio" && s.format) out.format = s.format; @@ -240,6 +243,7 @@ export function stepSummary(s: StepSpec, templateName?: (id: number) => string | } if (s.model) parts.push(s.model); if (s.force) parts.push("forced"); + if (s.type === "llm" && s.searchable === false) parts.push("not in search"); const w = whenText(s.when); if (w) parts.push(w); return parts.join(" · "); diff --git a/nextjs-frontend/components/pipelines/step-settings.tsx b/nextjs-frontend/components/pipelines/step-settings.tsx index 01d6b61a..d4c05abc 100644 --- a/nextjs-frontend/components/pipelines/step-settings.tsx +++ b/nextjs-frontend/components/pipelines/step-settings.tsx @@ -298,6 +298,12 @@ export function StepSettings({ /> )} + set({ searchable: v ? undefined : false })} + disabled={readOnly} + /> )} {step.type === "export" && ( diff --git a/nextjs-frontend/components/search/links.ts b/nextjs-frontend/components/search/links.ts index 1cc1a037..3148b1c1 100644 --- a/nextjs-frontend/components/search/links.ts +++ b/nextjs-frontend/components/search/links.ts @@ -23,9 +23,11 @@ export function onPage(hit: Pick): boolean { return hit.source === "page" || ((hit.source === "object" || hit.source === "described") && hit.page != null); } -/** Where a hit was found, when it wasn't said: on screen, on the page, seen, or what a shot or page shows. */ -export function foundAs(hit: Pick): string | null { +/** Where a hit was found, when it wasn't said: on screen, on the page, seen, what a shot or page shows, or what a + * pipeline's model step wrote about the asset (named after the step). */ +export function foundAs(hit: Pick): string | null { const paged = onPage(hit); + if (hit.source === "generated") return `Written by ${hit.output_label || "a model step"}`; if (hit.source === "screen") return "On screen"; if (hit.source === "object") return paged ? "Seen on the page" : "Seen on screen"; if (hit.source === "described") return paged ? "What the page shows" : "What the shot shows"; diff --git a/nextjs-frontend/components/search/results.tsx b/nextjs-frontend/components/search/results.tsx index b642c6dd..22ba2702 100644 --- a/nextjs-frontend/components/search/results.tsx +++ b/nextjs-frontend/components/search/results.tsx @@ -52,14 +52,16 @@ function PlayButton({ hit, player, audio }: { hit: SearchHit; player: InlinePlay const playing = player.isPlaying(key); const untimed = hit.t0 == null; const paged = onPage(hit); + const written = hit.source === "generated"; const none = untimed || paged || audio === false || player.noAudio.has(hit.recording_id); - const label = paged - ? `Nothing to play: ${(foundAs(hit) ?? "on a page").toLowerCase()}` - : untimed - ? "No time: a line of a file without times" - : none - ? "No audio — text only" - : `${playing ? "Pause" : "Play"} from ${tc(hit.t0)}`; + const label = + paged || written + ? `Nothing to play: ${(foundAs(hit) ?? "on a page").toLowerCase()}` + : untimed + ? "No time: a line of a file without times" + : none + ? "No audio — text only" + : `${playing ? "Pause" : "Play"} from ${tc(hit.t0)}`; const btn = (