diff --git a/README.md b/README.md index 1fe97edf..1107eedc 100644 --- a/README.md +++ b/README.md @@ -145,7 +145,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs, Office files, photos, JSON and zip or tar archives (photos and videos dated by when they were taken, from the file itself, and photos placed by their GPS on the Library's map, by API and MCP too), and transcripts made elsewhere - (SRT, WebVTT, text, Word, PDF), each line credited to the speaker the file names; and, with an Apache Tika server (an + (SRT, WebVTT, text, Word, PDF), each line credited to the speaker the file names, and one that came without times + lined up with its audio when you ask; and, with an Apache Tika server (an opt-in extra), documents nothing else here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off by default), taking a video's own captions as its transcript when people @@ -156,7 +157,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p files, a link list, a source's files) is one run: the toast after it, and the Import screen's list of recent runs, undo it whole. ([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika), - [import webhooks](docs/api.md#import-webhooks), [playable copies](docs/video.md#playable-copies), + [import webhooks](docs/api.md#import-webhooks), [lining up a transcript](docs/processing.md#aligning), + [playable copies](docs/video.md#playable-copies), [where an asset was taken](docs/processing.md#where-an-asset-was-taken), [import runs](docs/api.md#import-runs); [checks](docs/requirements.md#bring-in-everything)) diff --git a/changelog.d/align-transcript.md b/changelog.d/align-transcript.md new file mode 100644 index 00000000..8522c34e --- /dev/null +++ b/changelog.d/align-transcript.md @@ -0,0 +1,9 @@ +- **Line up a transcript with its audio.** A transcript imported without times (plain text, Word, PDF, Markdown) now + says its times are estimated, and once its audio is attached its page offers **Line up with the audio**: one click + transcribes the audio in the background with the engine transcription already uses (activity ledger, budgets and + quiet processing hours apply), matches the transcript's words to the words heard, and gives each line their times; + lines none of whose words were heard sit between their neighbours. The text and speakers stay exactly as they were. + Nothing is lined up unless someone asks. Agents use `POST /api/v1/resources/{id}/align` and the MCP tool + `align_transcript`; `GET /api/v1/resources/{id}` says `times` (estimated or aligned). Upgrading marks transcripts + imported before with estimated times (the `estimated-times` step) + ([Lining up a transcript with its audio](docs/processing.md#aligning)). diff --git a/docs/api.md b/docs/api.md index e1597a02..effb0d05 100644 --- a/docs/api.md +++ b/docs/api.md @@ -639,6 +639,7 @@ GET /api/v1/resources/{rid}/embed-link GET /api/v1/resources/{rid}/audio GET /api/v1/resources/{rid}/wordcloud.svg POST /api/v1/resources/{rid}/reprocess +POST /api/v1/resources/{rid}/align POST /api/v1/resources/{rid}/share DELETE /api/v1/resources/{rid}/share GET /api/v1/resources/{rid}/shares @@ -807,6 +808,19 @@ manifest and Dublin Core record give the tag). The MCP tool `set_language` does in the web app shows the language by name with its tag, which editors change there. The Language filter counts `en-GB` apart from `en`; a pipeline step's `languages` condition of `en` takes `en-GB` and `en-US` too. +### Lining up a transcript with its audio + +A transcript imported without times (plain text, Word, PDF, Markdown, `Name: text` lines without times) has times +estimated from the text, and `GET /resources/{rid}` says so: `times` is `estimated`. Attaching its audio keeps them. +`POST /resources/{rid}/align` (the process permission) lines the transcript up with its audio in the background: +202 with the job, which transcribes the audio as the transcribe step does (the same engine, activity ledger and +budgets; a heavy step, so quiet processing hours apply), gives each line the times of its words, then runs analyze, +embed and report again. The text and speakers stay as they are. Afterwards `times` is `aligned` and `aligned` is +`{at, matched, engine}`: when, the share of the transcript's words heard (0 to 1) and the engine that heard them. +Asking while it runs gives the same job; 400 when the resource has no audio, or its lines' times came with it or from +transcription (`times` is null). Audited as `recording.align`; the MCP tool `align_transcript` does the same +([Lining up a transcript with its audio](processing.md#aligning)). + ### Documents and images A document (a PDF; a Word, PowerPoint or spreadsheet file, OpenDocument or RTF; text or Markdown; a saved web page; an diff --git a/docs/architecture.md b/docs/architecture.md index 5d090e62..3eec7d5f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -82,8 +82,8 @@ jobs), drain it (it also hands its job back after the step it's on) and resume i record, so it holds across restarts. `GET /api/v1/events` streams job progress as server-sent events. **Quiet processing** (Settings › Workers, both off by default) keeps the heavy steps (`workers.heavy_steps`: by -default transcribe, read, diarize, shots, ocr, faces, objects, nsfw and blur_faces, the ones that work through media -on this machine) to when it suits a small machine at home: +default transcribe, read, diarize, shots, ocr, faces, objects, nsfw, blur_faces and align, the ones that work through +media on this machine) to when it suits a small machine at home: - `workers.heavy_hours`, a window like `01:00-06:00` in `workers.timezone` (UTC by default; it may pass midnight). Outside it no job starts a heavy step: a job that reaches one goes back on the queue, its log saying why, and waits diff --git a/docs/database.md b/docs/database.md index f081714b..5149500d 100644 --- a/docs/database.md +++ b/docs/database.md @@ -133,6 +133,10 @@ The steps so far, in the order they run: merges carried them over (the `not_same` table) get a pair naming the person they were merged into, following merges of merges; a pair that would name one person twice is dropped. The old rows stay, so undoing such a merge brings them back, and a pair already there isn't made again, so it's safe to run twice. +* `estimated-times`: transcripts imported before an asset said its lines' times were estimated (`times`, + [Lining up a transcript with its audio](processing.md#aligning)) say so now, where their lines are spaced as the + estimate spaces them (the first at 0, each 250 ms after the one before ends). Only assets without `times` are + looked at, so it's safe to run twice. To add a step: diff --git a/docs/mcp.md b/docs/mcp.md index ed0a05fe..83c16952 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -134,6 +134,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `get_face_blur` | a video's faces (track ids and names, to pick from) and its [copy with faces blurred](video.md#blurring-faces-for-sharing): the file and where to download it, whose faces it blurs and whether visitors play it; and the latest job making one, with how far it got, why it waits or why it failed. Members of its namespace | | `blur_faces` | make a [copy of a video with faces blurred](video.md#blurring-faces-for-sharing) for sharing: all faces, the unnamed ones or the tracks given; it runs in the background and is kept with the video's files, the original unchanged. Needs a write-scope token and the process permission | | `set_face_blur_for_visitors` | have visitors (public pages, embeds, share links, IIIF, feeds) play a video's copy with faces blurred instead of the original, or not. Needs a write-scope token and owner access | +| `align_transcript` | [line an imported transcript up with its audio](processing.md#aligning) when its times are estimated from the text (`get_recording` says `times`): the audio is transcribed in the background and each line takes the times of its words, the text and speakers unchanged. Needs a write-scope token and editor access (the process permission) | | `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there | | `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token | | `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files | @@ -220,7 +221,8 @@ queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [p `write_spec`, which queues a spec, `translate_transcript`, which queues a translation, `draft_reply` and `edit_reply`, which draft and change [email replies](email-replies.md) (nothing is sent), `send_reply`, which asks to send one (it waits in Approvals), `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and editors), `set_language`, which sets an asset's language (editors), `enhance`, which enhances an image and reads it again (the process permission), `blur_faces` and `set_face_blur_for_visitors`, which make a video's copy with faces blurred and choose whether visitors play it (the -process permission, and owners), +process permission, and owners), `align_transcript`, which lines an imported transcript up with its audio (the process +permission), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `install_plugin`, which installs a plugin from a marketplace as your own extension, `add_mcp_server`, which adds an MCP server (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), `set_mail_routing`, which changes where an email account's messages go (admins), `add_mesh_node`, diff --git a/docs/processing.md b/docs/processing.md index e4e9054e..93dd3494 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -226,7 +226,8 @@ on a text-only asset's page. The file is uploaded and becomes the asset's media; speakers stay. Then these steps run, after anything the asset's job still had to do: transcribe (which keeps the transcript and draws the waveform), diarize (speakers by voice, unless the transcript named them), shots, text on screen and faces (video only), analyze and report. The asset takes the file's fingerprint, so scans and uploads of the -same file find it, unless another asset in the namespace has it already. +same file find it, unless another asset in the namespace has it already. A transcript that came without times keeps +its estimated ones until you [line it up with the audio](#aligning). Speakers are recognised from `Name: text`, `[12:30] Name: text`, `Name (12:30): text`, Otter/Zoom/Teams exports (a `Name 12:30` line, then what they said), `speaker|emotion|text` lines, and the speakers in SRT/VTT and JSON. In @@ -238,6 +239,22 @@ first: front matter or the first heading becomes the title; imports, exports, JS imported as a transcript needs a text layer; import a scan as a document instead, and its pages are read by OCR. Named speakers are matched to the namespace's people by name; generic labels (SPEAKER_00, S1, CH0) become new people. +### Lining up a transcript with its audio {#aligning} + +A transcript imported without times keeps its estimated times when its audio is attached, and its page says "Times +estimated". **Line up with the audio** on that page (editors; the API's `POST /resources/{rid}/align`, or the MCP tool `align_transcript`) gives its lines real times, in the +background: the audio is transcribed by the same engine the transcribe step uses (its fallbacks, your vocabulary, +the activity ledger and budgets apply, and it is a heavy step, so quiet processing hours apply), the transcript's words +are matched to the words heard (rare words first, then the words between them), and each line runs from its first +matched word to its last, keeping those words' times so the player follows along word by word. A line none of whose +words were heard shares the time between the lines either side by how many words each has. The text and the +speakers stay exactly as they are; only the times change. Then analyze, embed and report run again, so chapters, +search passages and the report follow the new times; a summary's key points keep their times until it's summarised +again. The page then says "Times from the audio" and how many of the words were heard. When none were (the wrong file, +or another language), the step fails, saying so, and the estimated times stay. Nothing is lined up unless someone +asks: it costs a transcription. Forcing the transcribe step to run again replaces the imported lines with the +engine's own. + ## People and namespaces Person ids belong to one namespace. Per recording, `diarize.engine: auto` works like this: diff --git a/docs/requirements.md b/docs/requirements.md index 358cde46..85d583aa 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -170,6 +170,7 @@ request, to another check that proves the same thing; loosening a requirement or | Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` | | Uploads of audio, PDFs, Office files, photos, JSON and archives | [test_uploads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_uploads.py) `test_uploading_audio_in_chunks`, `test_json_uploaded_is_read_as_records`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_a_pdf_becomes_a_document_with_its_pages_and_text`, `test_scans_and_images_are_read_by_ocr`; [test_conversion.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversion.py) `test_office_and_text_files_become_documents`; [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_an_uploaded_zip_keeps_its_files`; [test_archives.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_archives.py) `test_reading_an_archive`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `import-file`, `import-paste` | | Transcripts made elsewhere (text, Markdown, Word, PDF, SRT, WebVTT, JSON) imported with each line credited to the speaker the file names, WebVTT voice spans (``, ``) included | [test_importing.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_importing.py) `test_formats_and_speakers_by_name`, `test_webvtt_voice_spans_name_each_line` | +| A transcript imported without times says its times are estimated, and is lined up with its audio only when asked (API, MCP and one button): its words matched to the words the speech engine hears, each line taking their times and the rest placed between their neighbours, the text and speakers unchanged | [test_align.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_align.py) `test_lines_take_the_times_of_their_words_and_the_rest_fill_in_between`, `test_a_long_transcript_is_matched_as_well_as_word_by_word_and_quickly`, `test_imports_say_their_times_are_estimated_and_older_ones_are_recognised`, `test_lining_up_a_transcript_with_its_audio`, `test_agents_line_up_a_transcript_through_mcp`; [align.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/align.test.tsx) `is offered only for estimated times with audio, and not while a job is at it`, `lines it up in one click, then gets out of the way` | | Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` | | Photos and videos dated by when they were taken, from the file itself (a photo's EXIF date and GPS fix as `place`, a video's creation time) when no name or source dates them; a date already set is kept and none is invented | [test_file_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_dates.py) `test_a_photo_is_dated_and_placed_by_its_exif`, `test_a_date_already_set_is_kept`, `test_a_video_is_dated_by_its_creation_time`, `test_a_scanned_video_is_dated_by_its_creation_time` | | Assets placed by their file's GPS fix (a photo's EXIF, a video's ISO 6709 location) on the Library's map, listed by API (`located`, `bbox`) and MCP to members only, on tiles from a setting | [test_photo_map.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photo_map.py) `test_located_assets_are_listed_inside_a_box`, `test_an_agent_finds_what_was_taken_in_a_place`, `test_a_video_is_placed_by_its_location`, `test_the_map_tiles_are_a_setting`; [library-map.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/library-map.test.tsx) `placed assets are dots on the tiles from Settings, framed to show them all` | diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index b94baded..7260ba42 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -1030,6 +1030,7 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]: "pages": raw.get("pages"), "language": raw.get("language"), "language_set": bool(raw.get("language_set")) or None, # a person set it; transcription leaves it + "times": rec.times, # an imported transcript's line times: estimated from the text, or aligned with its audio "status": rec.status, "tags": rec.tags or None, "your_role": rec.role, @@ -3176,6 +3177,27 @@ def set_face_blur_for_visitors(ctx: Context, recording_id: int, on: bool) -> dic return _face_blur_out(ctx, recording_id) +# ---------- an imported transcript lined up with its audio (docs/processing.md#aligning) ---------- +@tool( + "align_transcript", + "Line up a transcript with its audio", + "Give an imported transcript's lines real times from its audio: a transcript imported without times (plain text, " + "Word, PDF, Markdown) has times estimated from the text (get_recording says times: estimated), and attaching its " + "audio keeps them. This transcribes the audio in the background, matches the transcript's words to the words " + "heard, and gives each line the times of its words; the text and speakers stay as they are, and times becomes " + "aligned. It costs a transcription (a cloud speech engine's call goes in the activity ledger). Check on it with " + "runs. Needs a token with the write scope and the right to process the recording; refused when it has no audio, " + "or its lines' times came with it or from transcription.", + Arg("recording_id", "integer", "the asset", required=True), + writes=True, +) +def align_transcript(ctx: Context, recording_id: int) -> dict[str, Any]: + if not ctx.user.can_write: + raise ToolError("this token is read-only: ask for a token with the write scope to line up transcripts") + out = recording_routes.align_transcript(recording_id, ctx.acl, ctx.user, ctx.db) + return {"job": out.job, "status": "aligning", "recording_url": ctx.link(recording_id)} + + # ---------- a namespace's own ways in for other apps (integrations.py) ---------- def _integrations(ctx: Context, namespace: str, sid: int) -> dict[str, Any]: return { diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py index 4f31915f..f9a0a151 100644 --- a/fastapi_backend/app/api/v1/routes/recordings.py +++ b/fastapi_backend/app/api/v1/routes/recordings.py @@ -19,6 +19,7 @@ from app.api.streaming import LOCKED, file_response, range_response from app.domain import access as acc from app.domain import ( + align, analyze, auth, deletion, @@ -751,6 +752,22 @@ def enhance_recording(rid: int, acl: Acl, user: Writer, db: Db) -> JobQueued: return JobQueued(job=job) +@router.post("/{rid}/align", status_code=202) +def align_transcript(rid: int, acl: Acl, user: Writer, db: Db) -> JobQueued: + """Line an imported transcript up with its audio (docs/processing.md#aligning): the audio is transcribed in the + background as the transcribe step does (the same engine, ledger and budgets; a heavy step, so quiet processing + hours apply), the transcript's words are matched to the words heard, and each line takes the times of its words; + then analyze, embed and report run again. The text and speakers stay as they are; `times` becomes aligned. Asking + while it runs gives the same job. Needs the right to process the asset; 400 when it has no audio, or its lines' + times came with it or from transcription (`times` is neither estimated nor aligned). Audited as + `recording.align`.""" + acl.recording(rid, "process") + with domain_errors(): + jid = align.queue(db, rid, by=user.email) + auth.audit(db, user.as_audit(), "recording.align", f"recording:{rid}", {"job": jid}) + return JobQueued(job=jid) + + @router.post("/{rid}/share") def create_share(rid: int, acl: Acl, user: Writer, db: Db, body: ShareCreate | None = None) -> ShareLink: """A read-only link to this one recording, for people without an account, with a short ``/s/`` address too. Its diff --git a/fastapi_backend/app/domain/align.py b/fastapi_backend/app/domain/align.py new file mode 100644 index 00000000..108f84dc --- /dev/null +++ b/fastapi_backend/app/domain/align.py @@ -0,0 +1,195 @@ +"""Lining up an imported transcript with its audio (docs/processing.md, Lining up a transcript with its audio). + +A transcript imported without times (plain text, Word, PDF, Markdown, `Name: text` lines without times) gets a +speaking-rate estimate for each line, and the asset says so (`times` "estimated"). Attaching its audio keeps those +times. When a person asks (the API, MCP or the asset page), an `align` step runs the audio through the speech engine +transcription uses (ingest.hear: the same engine, fallbacks, vocabulary, ledger and budgets), lines the imported words +up with the words it heard (difflib, rare words first, then the gaps between them), and gives each line the start of +its first matched word and the end of its last; lines that matched nothing share the time between their neighbours by +how many words each has. The text and speakers stay exactly as they are; only the lines' times change, and each line +keeps the times of its matched words (`words`), as transcription's lines do. The asset then says `times` "aligned", +with how many of the words were heard (`aligned`). + +It is a heavy step (quiet.py), queued after what the asset's job still has to do and followed by analyze, embed and +report, so chapters, passages and the report follow the new times. Nothing runs on its own: imports keep their +estimated times until someone asks. + +Refine later: an undo that puts the estimate back; lining up only the part of a long recording a transcript covers; +a summary's key points and action items taking the new times without summarising again; JSON transcripts without +times (their estimate leaves no gap between lines, so the upgrade step doesn't recognise them). +""" + +from __future__ import annotations + +import difflib +import json +import re + +from . import ingest, store, transcript + +R = store.R +STEP = "align" +AFTER = ["analyze", "embed", "report"] # what reads the lines' times +TIMES = ("estimated", "aligned") # lines whose times may be lined up again; any other's came with them or from transcription +GAP = 250 # ms the speaking-rate estimate leaves between lines (ingest._timed) +WORD_MS = 385 # ms a word takes at the estimate's speaking rate +TOKEN = re.compile(rf"{transcript.CJK.pattern}|[^\s{transcript.CJK.pattern[1:-1]}]+") + + +def _words(text): + """A line's words as written, leaving out what has no letters or digits; a CJK character is a word of its own.""" + return [w for w in TOKEN.findall(text or "") if transcript._norm(w)] + + +def heard_words(segs): + """[(word, t0, t1)] the engine heard, normalised (lower case, letters and digits), in order: each line's own + timed words, or its words spread evenly over it when the engine gave none (SenseVoice).""" + out = [] + for s in sorted(segs, key=lambda s: s["t0"]): + timed = [(w[0], w[1], w[2]) for w in transcript.words(s)] or [(s.get("text") or "", int(s["t0"]), int(s["t1"]))] + for text, t0, t1 in timed: + ws = _words(text) + step = (t1 - t0) / max(1, len(ws)) + out += [(transcript._norm(w), int(t0 + k * step), int(t0 + (k + 1) * step)) for k, w in enumerate(ws)] + return out + + +def matches(a, b): + """[(i, j)]: which word of `a` is which of `b`, in order. Rare words anchor first (difflib's autojunk sets the + common ones aside in long texts), then the gaps between anchors are matched word for word.""" + out = [] + + def match(a0, a1, b0, b1, junk): + sm = difflib.SequenceMatcher(None, a[a0:a1], b[b0:b1], autojunk=junk) + i = j = 0 + for blk in sm.get_matching_blocks(): + if junk and blk.a > i and blk.b > j: + match(a0 + i, a0 + blk.a, b0 + j, b0 + blk.b, False) + out.extend((a0 + blk.a + k, b0 + blk.b + k) for k in range(blk.size)) + i, j = blk.a + blk.size, blk.b + blk.size + + match(0, len(a), 0, len(b), True) + return out + + +def line_up(lines, heard, duration_ms=None): + """Each line's (t0, t1, words) from the words heard (heard_words), the number of the lines' words matched, and + how many words they have. A line runs from its first matched word's start to its last one's end, and keeps those + words as [word, t0, t1] in its own spelling; a run of lines that matched nothing shares the time between the + lines either side (to the end of the audio after the last) by how many words each has, and keeps no words.""" + words = [_words(t) for t in lines] + flat = [(n, k) for n, ws in enumerate(words) for k in range(len(ws))] + pairs = matches([transcript._norm(words[n][k]) for n, k in flat], [w for w, _, _ in heard]) + got = [[] for _ in lines] + for i, j in pairs: + n, k = flat[i] + got[n].append([words[n][k], heard[j][1], heard[j][2]]) + out = [(ws[0][1], max(ws[-1][2], ws[0][1] + 1), ws) if ws else None for ws in got] + n = 0 + while n < len(lines): + if out[n] is not None: + n += 1 + continue + k = next((m for m in range(n, len(lines)) if out[m] is not None), len(lines)) + weight = [max(1, len(words[m])) for m in range(n, k)] + lo = out[n - 1][1] if n else 0 + hi = out[k][0] if k < len(lines) else max(duration_ms or 0, lo + WORD_MS * sum(weight)) + span, acc = max(0, hi - lo), 0 + for m, w in zip(range(n, k), weight): + a = lo + span * acc // sum(weight) + acc += w + out[m] = (a, max(a + 1, lo + span * acc // sum(weight)), []) + n = k + return out, len(pairs), len(flat) + + +# ---------- asking for it ---------- +def latest(db, rid): + """The latest job to line the asset up: {id, status, error}, or None.""" + for j in db.rows( + "SELECT record::id(id) AS id, status, steps, error FROM job WHERE recording = $r ORDER BY id DESC LIMIT 20", r=int(rid) + ): + if any(isinstance(s, dict) and s.get("type") == STEP for s in j.get("steps") or []): + return {"id": j["id"], "status": j["status"], "error": j.get("error")} + return None + + +def queue(db, rid, by=None): + """Line the asset's transcript up with its audio, in the background: an `align` step and the steps that read the + times, after what its job still has to do (jobs.add_steps), or a new job; the job already doing it if there is one. + ValueError without audio, or when its lines' times came with it or from transcription; KeyError for no such + asset.""" + from . import jobs + + rec = db.one("SELECT source, times FROM $r", r=R("recording", int(rid))) + if rec is None: + raise KeyError(rid) + if rec.get("source") != "audio": + raise ValueError("Attach its audio first: there's nothing to line the transcript up with.") + if rec.get("times") not in TIMES: + raise ValueError("Its lines already have times: they came with the transcript or from transcription.") + j = latest(db, rid) + if j and j["status"] in jobs.ACTIVE: + return j["id"] + return jobs.add_steps(db, int(rid), [{"type": STEP}, *AFTER], by=by) + + +# ---------- the job step ---------- +def step(db, cfg, rid, say, spec=None): + """The `align` step: the audio heard, the lines given the times of their words, the asset marked aligned.""" + from . import jobs + + rec = db.one(f"SELECT {ingest.HEAR_FIELDS}, source, times, duration_ms FROM $r", r=R("recording", int(rid))) or {} + if rec.get("source") != "audio": + raise jobs.Skip("there's no audio to line the transcript up with") + if rec.get("times") not in TIMES: + raise jobs.Skip("its lines' times came with the transcript or from transcription; kept them") + lines = db.rows("SELECT record::id(id) AS id, idx, text FROM segment WHERE recording = $r ORDER BY idx", r=int(rid)) + if not lines: + raise jobs.Skip("there are no lines to line up") + say(f"listening to the audio to line up {len(lines)} line(s)") + segs, engine, _ = ingest.hear(db, cfg, rec, say) + heard = heard_words(segs) + times, matched, total = line_up([s["text"] for s in lines], heard, rec.get("duration_ms")) + if not matched: + raise RuntimeError( + f"none of the transcript's words were heard in its audio ({len(heard)} word(s) heard): is it the right file? " + "The estimated times are kept" + ) + rows = [ + store.clean({"id": R("segment", s["id"]), "t0": t0, "t1": t1, "dur": t1 - t0, "words": json.dumps(ws) if ws else None}) + for s, (t0, t1, ws) in zip(lines, times) + ] + share = round(matched / max(1, total), 3) + db.run( + [ + "FOR $s IN $rows { UPDATE $s.id SET t0 = $s.t0, t1 = $s.t1, dur = $s.dur, words = $s.words; }", + "UPDATE $rec SET times = 'aligned', aligned = $a", + ], + rows=rows, + rec=R("recording", int(rid)), + a={"at": store.now(), "matched": share, "engine": engine.name}, + ) + alone = sum(1 for _, _, ws in times if not ws) + say( + f"lined up {len(lines)} line(s) with the audio: {matched} of {total} word(s) heard ({share:.0%})" + + (f"; {alone} line(s) with none heard placed between their neighbours" if alone else "") + ) + + +# ---------- what imports say, and older ones ---------- +def looks_estimated(lines): + """Whether these lines' times ({t0, t1} in order) are the speaking-rate estimate a text import without times gets + (ingest._timed): the first at 0, and each GAP after the one before ends.""" + return bool(lines) and lines[0]["t0"] == 0 and all(b["t0"] == a["t1"] + GAP for a, b in zip(lines, lines[1:])) + + +def migrate_estimated(db): + """Transcripts imported before the asset said its times were estimated: those whose lines are spaced as the + estimate spaces them say so now (`times` "estimated"). Only assets without `times` are looked at, so it's safe to + run twice.""" + ids = db.values("SELECT VALUE record::id(id) FROM recording WHERE times = NONE AND string::starts_with(engine ?? '', 'import:')") + for rid in ids: + lines = db.rows("SELECT idx, t0, t1 FROM segment WHERE recording = $r ORDER BY idx", r=rid) + if looks_estimated(lines): + db.q("UPDATE $r SET times = 'estimated' WHERE times = NONE", r=R("recording", rid)) diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index 3a9e26f5..b0201fd7 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -575,7 +575,10 @@ def write_transcript(db, rid, nid, segs, patch): rec=store.R("recording", rid), patch=patch, ) - db.q("UPDATE $rec SET error = NONE, diarized_at = NONE, analyzed_at = NONE", rec=store.R("recording", rid)) + # new lines: their times are their own (an import without times says so after, align.py) + db.q( + "UPDATE $rec SET error = NONE, diarized_at = NONE, analyzed_at = NONE, times = NONE, aligned = NONE", rec=store.R("recording", rid) + ) def audio_path(db, cfg, rec, plain=True): @@ -603,9 +606,14 @@ def add_envelope(db, cfg, rid): db.q("UPDATE $r SET envelope = $e", r=store.R("recording", rid), e=envelope(decode(audio_path(db, cfg, rec)))) -def transcribe_one(db, cfg, rid, log=print, engine=None): - r = db.one("SELECT record::id(id) AS id, space, path, title, remote, channels, audio_cleanup FROM $r", r=store.R("recording", rid)) - engine, t = engine or get_engine(cfg, log), time.time() +HEAR_FIELDS = "record::id(id) AS id, space, path, title, remote, channels, audio_cleanup" # what hear() reads of a recording + + +def hear(db, cfg, r, log=print, engine=None): + """What the speech engine hears in a recording's audio (`r`, its HEAR_FIELDS): (segments, the engine, the + waveform). The configured engine or the one it falls back to, the namespace's vocabulary as hints, and the sound + as its clean_audio step asked for it to be heard; a cloud engine's call goes in the activity ledger.""" + engine = engine or get_engine(cfg, log) path = audio_path(db, cfg, r) audio, joins = decode(path), None env = envelope(audio) # the player's waveform is the recording's own @@ -618,6 +626,13 @@ def transcribe_one(db, cfg, rid, log=print, engine=None): del audio if joins: segs = audio_cleanup.remap(segs, joins) + return segs, engine, env + + +def transcribe_one(db, cfg, rid, log=print, engine=None): + r = db.one(f"SELECT {HEAR_FIELDS} FROM $r", r=store.R("recording", rid)) + t = time.time() + segs, engine, env = hear(db, cfg, r, log, engine) write_transcript( db, rid, @@ -1187,6 +1202,8 @@ def _store_import(db, cfg, ns, t, title, fp, src, st, audio, speaker_names, engi } ), ) + if not t.get("timed"): # every line's time is a speaking-rate estimate, until it's lined up with audio (align.py) + db.q("UPDATE $r SET times = 'estimated'", r=store.R("recording", rid)) names = {**(t.get("speakers") or {}), **(speaker_names or {})} locals_ = {s["speaker"] for s in segs if s.get("speaker")} if locals_: diff --git a/fastapi_backend/app/domain/jobs.py b/fastapi_backend/app/domain/jobs.py index 1d7f686a..c96042ad 100644 --- a/fastapi_backend/app/domain/jobs.py +++ b/fastapi_backend/app/domain/jobs.py @@ -40,6 +40,7 @@ "nsfw", "describe", "blur_faces", + "align", } # they take longer the longer the recording FILED = ("audio", "document", "image") # sources with a file of their own for the steps to work on TIMINGS = 25 # recent timings kept per kind of step, for estimates @@ -289,6 +290,13 @@ def _blur_faces(db, cfg, rid, say, spec=None): face_blur.step(db, cfg, rid, say, spec) +def _align(db, cfg, rid, say, spec=None): + """An imported transcript's lines given times from its audio (align.py); only when someone asks.""" + from . import align + + align.step(db, cfg, rid, say, spec) + + def _convert(db, cfg, rid, say, spec=None): """New assets made from this one: a video's soundtrack as audio, a document's pages as images (conversions.py).""" from . import conversions @@ -329,6 +337,7 @@ def _describe(db, cfg, rid, say, spec=None): "reply": _draft_reply, "translate": _translate, "blur_faces": _blur_faces, + "align": _align, } @@ -1184,6 +1193,8 @@ def __init__(self, db, cfg_fn, name=None, steps=None, log=None): self.can.add("clean") if not steps and self.can & {"transcribe", "read"}: # converting needs what they need; older lists run it too self.can.add("convert") + if not steps and "transcribe" in self.can: # lining a transcript up with its audio needs no more; older lists run it + self.can.add("align") if not steps and "faces" in self.can: # blurring faces needs no more than finding them; older lists run it too self.can.add("blur_faces") if not steps and "objects" in self.can: # NSFW classification runs on ONNX Runtime too; older lists run it as well diff --git a/fastapi_backend/app/domain/migrations.py b/fastapi_backend/app/domain/migrations.py index 2a4037ab..a899ea11 100644 --- a/fastapi_backend/app/domain/migrations.py +++ b/fastapi_backend/app/domain/migrations.py @@ -49,7 +49,7 @@ def steps(): """[(name, fn(db))] in the order they run. Append only.""" - from . import access, auth, chat, entities, hierarchy, jobs, mcp_servers, pipelines, speakers + from . import access, align, auth, chat, entities, hierarchy, jobs, mcp_servers, pipelines, speakers return [ # IIIF-only access levels (public, transcript, signed-in, private) become the access setting @@ -70,6 +70,8 @@ def steps(): ("generated-search", pipelines.migrate_generated), # what someone said a person merged away isn't now holds for the one they were merged into ("speaker-not-same-merged", speakers.migrate_not_same), + # transcripts imported without times say their lines' times are estimated, so they can be lined up with audio + ("estimated-times", align.migrate_estimated), ] diff --git a/fastapi_backend/app/domain/quiet.py b/fastapi_backend/app/domain/quiet.py index eb19cce5..82ee2477 100644 --- a/fastapi_backend/app/domain/quiet.py +++ b/fastapi_backend/app/domain/quiet.py @@ -8,7 +8,8 @@ already running finishes; two workers claiming in the same instant can briefly make it one more). The heavy steps are `workers.heavy_steps`: by default the ones that work through media on this machine (speech, -voices, shots, text on screen, faces, objects, NSFW, reading documents, blurring faces). Everything else runs as before. +voices, shots, text on screen, faces, objects, NSFW, reading documents, blurring faces, lining transcripts up with +audio). Everything else runs as before. """ from __future__ import annotations @@ -18,7 +19,7 @@ from . import schedule, store -HEAVY = ["transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "blur_faces"] +HEAVY = ["transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "blur_faces", "align"] WINDOW = re.compile(r"^([01]\d|2[0-3]):([0-5]\d)-([01]\d|2[0-3]):([0-5]\d)$") diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 9a734ac1..6ebb78e1 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -309,7 +309,7 @@ def labels(): # quiet processing (quiet.py): a window like "01:00-06:00" for the heavy steps, and how many run at once; off "heavy_hours": None, "heavy_at_once": None, - "heavy_steps": ["transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "blur_faces"], + "heavy_steps": ["transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "blur_faces", "align"], "timezone": "UTC", "steps": [ "enhance", diff --git a/fastapi_backend/app/schemas/recordings.py b/fastapi_backend/app/schemas/recordings.py index f359c404..38d7385b 100644 --- a/fastapi_backend/app/schemas/recordings.py +++ b/fastapi_backend/app/schemas/recordings.py @@ -356,6 +356,14 @@ class ConvertedFrom(ResponseModel): title: str | None = None +class Aligned(ResponseModel): + """An imported transcript lined up with its audio (docs/processing.md#aligning).""" + + at: str | None = Field(None, description="when") + matched: float | None = Field(None, description="the share of the transcript's words heard in the audio, 0 to 1") + engine: str | None = Field(None, description="the speech engine that heard them") + + class Recording(ResponseModel): """The asset row (less its envelope) plus what the asset page needs.""" @@ -383,6 +391,12 @@ class Recording(ResponseModel): tags: list[str] = Field(default_factory=list) language: str | None = Field(None, description="its language: a BCP 47 tag such as en or en-GB, when known") language_set: bool = Field(False, description="a person set its language; transcription doesn't change it") + times: Literal["estimated", "aligned"] | None = Field( + None, + description="an imported transcript's line times: estimated from the text (it had none), or aligned with its " + "audio (POST /resources/{rid}/align); none when they came with it or from transcription", + ) + aligned: Aligned | None = Field(None, description="with times aligned: when, and how many of its words were heard") email: EmailInfo | None = Field(None, description="an email: its subject, sender, recipients and date") rendition: Rendition | None = Field(None, description="a document that isn't a PDF: how the PDF it's read from was made") awaits_pdf: AwaitsPdf | None = Field(None, description="waiting for the PDF the import webhook that sent it hands back") diff --git a/fastapi_backend/tests/api/test_align.py b/fastapi_backend/tests/api/test_align.py new file mode 100644 index 00000000..7e32c30e --- /dev/null +++ b/fastapi_backend/tests/api/test_align.py @@ -0,0 +1,181 @@ +"""An imported transcript lined up with its audio (align.py, docs/processing.md#aligning): only when someone asks, its +audio is transcribed, the imported words are matched to the words heard, and each line takes their times; the text and +speakers stay.""" + +from __future__ import annotations + +import difflib +import json +import random + +import pytest + +from app.domain import align, ingest, jobs, quiet, store +from tests.api.test_mcp import tool, tool_error +from tests.api.test_oauth import _bearer, _grant +from tests.helpers import drain, login, make_user, write_wav +from tests.helpers import quiet as hush + +R = store.R +TEXT = "Welcome back to the show.\n\nUm.\n\nToday we talk about capsids.\n\nBye." + + +def _heard(): + """What a fake engine hears: one line with word times (Whisper), one without (SenseVoice), a word misheard.""" + return [ + { + "t0": 1000, + "t1": 2200, + "text": "Welcome back to the show.", + "words": json.dumps( + [["Welcome", 1000, 1300], ["back", 1300, 1500], ["to", 1500, 1600], ["the", 1600, 1700], ["show.", 1700, 2200]] + ), + }, + {"t0": 5000, "t1": 8000, "text": "today we talk about capsid", "words": None}, + ] + + +class Engine: + name = "whisper" + + def transcribe(self, audio, terms=()): + return _heard() + + +def test_lines_take_the_times_of_their_words_and_the_rest_fill_in_between(): + lines = ["Welcome back to the show.", "Um.", "Today we talk about capsids.", "Bye."] + heard = align.heard_words(_heard()) + assert heard[5:] == [("today", 5000, 5600), ("we", 5600, 6200), ("talk", 6200, 6800), ("about", 6800, 7400), ("capsid", 7400, 8000)] + out, matched, total = align.line_up(lines, heard, duration_ms=10_000) + assert (matched, total) == (9, 12) # "Um", "capsids" and "Bye" weren't heard + assert [(t0, t1) for t0, t1, _ in out] == [ + (1000, 2200), # its first word's start to its last one's end + (2200, 5000), # nothing heard: between the lines either side + (5000, 7400), # up to "about": "capsids" wasn't heard + (7400, 10_000), # after the last: to the end of the audio + ] + assert out[0][2][-1] == ["show.", 1700, 2200] and out[2][2][0] == ["Today", 5000, 5600] # its own spelling + assert out[1][2] == [] and out[3][2] == [] + + # a CJK line is matched character by character + assert align.line_up(["你好"], [("你", 0, 100), ("好", 100, 200)])[1] == 2 + + +def test_a_long_transcript_is_matched_as_well_as_word_by_word_and_quickly(): + """Rare words anchor first, then the gaps: as many matches as difflib finds word by word (which on 20,000 words + took 7.5 s against 0.25 s), in order.""" + rng = random.Random(1) + said = rng.choices([f"w{i}" for i in range(500)], [1 / (i + 1) for i in range(500)], k=3000) + heard = [w for w in said if rng.random() > 0.08] # some words not heard + heard[::37] = ["zz"] * len(heard[::37]) # and some misheard + got = align.matches(said, heard) + plain = difflib.SequenceMatcher(None, said, heard, autojunk=False).get_matching_blocks() + assert len(got) == sum(b.size for b in plain) > 2500 + assert all(said[i] == heard[j] for i, j in got) and got == sorted(got) and len({j for _, j in got}) == len(got) + + +def test_imports_say_their_times_are_estimated_and_older_ones_are_recognised(db, cfg): + prose = ingest.import_text(db, cfg, "pods", TEXT, title="Prose") + timed = ingest.import_text(db, cfg, "pods", "[00:00] Ann: Hello there.\n[00:02] Ben: Hi.\n[00:04] Ann: Bye.") + times = lambda rid: db.one("SELECT times FROM $r", r=R("recording", rid)).get("times") # noqa: E731 + assert (times(prose), times(timed)) == ("estimated", None) + + # an install from before: no `times`; the upgrade step finds the lines spaced as the estimate spaces them + db.q("UPDATE recording SET times = NONE") + lines = db.rows("SELECT idx, t0, t1 FROM segment WHERE recording = $r ORDER BY idx", r=prose) + assert align.looks_estimated(lines) and lines[0]["t0"] == 0 + for _ in range(2): # safe to run twice + align.migrate_estimated(db) + assert (times(prose), times(timed)) == ("estimated", None) + + +@pytest.fixture +def env(client, db, cfg, folder, monkeypatch): + wav, tr = folder / "show.wav", folder / "show.txt" + write_wav(wav, seconds=10) + tr.write_text(TEXT) + rid = ingest.import_transcript(db, cfg, "pods", tr, audio=wav, log=hush) + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "view@x.io", "viewer password 1", roles={"pods": "viewer"}) + monkeypatch.setattr(ingest, "get_engine", lambda cfg, log=None: Engine()) + return { + "rid": rid, + "wav": wav, + "he": login(client, "ed@x.io", "editor password 1"), + "hv": login(client, "view@x.io", "viewer password 1"), + } + + +def test_lining_up_a_transcript_with_its_audio(client, env, db, cfg, folder): + rid, he, hv = env["rid"], env["he"], env["hv"] + rec = client.get(f"/api/v1/recordings/{rid}", headers=he).json() + assert (rec["times"], rec["aligned"]) == ("estimated", None) + before = db.rows("SELECT idx, text, speaker, t0 FROM segment WHERE recording = $r ORDER BY idx", r=rid) + assert [s["t0"] for s in before] == [0, 2175, 2810, 4985] # the speaking-rate estimate + + # editors only; nothing without audio, nor when the lines came with times + assert client.post(f"/api/v1/recordings/{rid}/align", headers=hv).status_code == 403 + text_only = ingest.import_text(db, cfg, "pods", TEXT, title="No audio") + r = client.post(f"/api/v1/recordings/{text_only}/align", headers=he) + assert (r.status_code, r.json()["detail"]) == (400, "Attach its audio first: there's nothing to line the transcript up with.") + (folder / "timed.txt").write_text("[00:00] Ann: Hello there.\n[00:02] Ben: Hi.\n[00:04] Ann: Bye.") + write_wav(folder / "timed.wav", seconds=3) + timed = ingest.import_transcript(db, cfg, "pods", folder / "timed.txt", audio=folder / "timed.wav", log=hush) + r = client.post(f"/api/v1/recordings/{timed}/align", headers=he) + assert (r.status_code, r.json()["detail"]) == ( + 400, + "Its lines already have times: they came with the transcript or from transcription.", + ) + + # a heavy step in the background, then what reads the times; asking again gives the same job + assert align.STEP in quiet.HEAVY and align.STEP in jobs.STEPS + r = client.post(f"/api/v1/recordings/{rid}/align", headers=he) + assert r.status_code == 202 + jid = r.json()["job"] + assert client.post(f"/api/v1/recordings/{rid}/align", headers=he).json()["job"] == jid + assert [s["type"] for s in db.one("SELECT steps FROM $j", j=R("job", jid))["steps"]] == ["align", "analyze", "embed", "report"] + drain(db, cfg) + job = db.one("SELECT status, log FROM $j", j=R("job", jid)) + assert job["status"] == "succeeded", job["log"] + assert any("9 of 12 word(s) heard (75%)" in line for line in job["log"]) + + # only the times changed + after = db.rows("SELECT idx, text, speaker, t0, t1, dur, words FROM segment WHERE recording = $r ORDER BY idx", r=rid) + assert [(s["idx"], s["text"], s.get("speaker")) for s in after] == [(s["idx"], s["text"], s.get("speaker")) for s in before] + assert [(s["t0"], s["t1"]) for s in after] == [(1000, 2200), (2200, 5000), (5000, 7400), (7400, 10_000)] + assert after[1]["dur"] == 2800 and json.loads(after[0]["words"])[0] == ["Welcome", 1000, 1300] and after[1].get("words") is None + rec = client.get(f"/api/v1/recordings/{rid}", headers=he).json() + assert rec["times"] == "aligned" and (rec["aligned"]["matched"], rec["aligned"]["engine"]) == (0.75, "whisper") + player = client.get(f"/api/v1/recordings/{rid}/player", headers=he).json() + assert player["segments"][0]["t0"] == 1000 + + # transcribing it again makes the lines the engine's own: their times are no longer the import's + jobs.enqueue(db, rid, [{"type": "transcribe", "force": True}]) + drain(db, cfg) + row = db.one("SELECT times, aligned FROM $r", r=R("recording", rid)) + assert (row.get("times"), row.get("aligned")) == (None, None) + + # nothing heard of it: the step fails, saying so, and the times stay + again = ingest.import_text(db, cfg, "pods", "Quite different words entirely.", title="Other") + db.q("UPDATE $r SET source = 'audio', path = $p", r=R("recording", again), p=str(env["wav"])) + jid = client.post(f"/api/v1/recordings/{again}/align", headers=he).json()["job"] + drain(db, cfg) + job = db.one("SELECT status, error FROM $j", j=R("job", jid)) + assert job["status"] == "failed" and "none of the transcript's words were heard" in job["error"] + assert db.one("SELECT times FROM $r", r=R("recording", again))["times"] == "estimated" + + audited = db.values("SELECT VALUE action FROM audit_log WHERE action = 'recording.align'") + assert len(audited) == 3 + + +def test_agents_line_up_a_transcript_through_mcp(client, env, db, cfg): + rid, he = env["rid"], env["he"] + assert tool(client, he, "get_recording", recording_id=rid)["times"] == "estimated" + _, tokens = _grant(client, he, scope="read") + assert "read-only" in tool_error(client, _bearer(tokens), "align_transcript", recording_id=rid) + out = tool(client, he, "align_transcript", recording_id=rid) + assert out["status"] == "aligning" and out["recording_url"].endswith(str(rid)) + drain(db, cfg) + assert db.one("SELECT status FROM $j", j=R("job", out["job"]))["status"] == "succeeded" + assert tool(client, he, "get_recording", recording_id=rid)["times"] == "aligned" + assert tool_error(client, he, "align_transcript", recording_id=999_999) == "not found, or not something you can read" diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index 3c1d6e6b..ee35837d 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -104,6 +104,7 @@ def test_clients_are_sent_to_sign_in(client, app, env): "enhance", "blur_faces", "set_face_blur_for_visitors", + "align_transcript", "set_namespace_integrations", "set_picture_descriptions", "set_file_storage", diff --git a/fastapi_backend/tests/domain/test_migrations.py b/fastapi_backend/tests/domain/test_migrations.py index af5b0cb4..316c6559 100644 --- a/fastapi_backend/tests/domain/test_migrations.py +++ b/fastapi_backend/tests/domain/test_migrations.py @@ -24,6 +24,7 @@ "entity-types", "generated-search", "speaker-not-same-merged", + "estimated-times", ] diff --git a/nextjs-frontend/__tests__/align.test.tsx b/nextjs-frontend/__tests__/align.test.tsx new file mode 100644 index 00000000..6684e78a --- /dev/null +++ b/nextjs-frontend/__tests__/align.test.tsx @@ -0,0 +1,73 @@ +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { fireEvent, render, screen, waitFor } from "@testing-library/react"; +import "@testing-library/jest-dom"; + +import { Resources } from "@/app/openapi-client"; +import { AlignBanner } from "@/components/recording/align"; +import { offerAlign, timesLabel } from "@/components/recording/align-model"; +import type { JobInfo } from "@/components/recording/jobs"; +import { TooltipProvider } from "@/components/ui/tooltip"; + +jest.mock("@/app/openapi-client", () => ({ Resources: { alignTranscript: jest.fn() } })); +jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) })); +const toast = jest.fn(); +jest.mock("@/components/ui/toast", () => ({ useToast: () => toast })); + +const m = (f: unknown) => f as jest.Mock; +const job = (status: string, type = "align") => ({ id: 3, status, steps: [{ type }] }) as unknown as JobInfo; +const estimated = { times: "estimated" as const }; + +function show(props: { editable?: boolean; jobs?: JobInfo[] } = {}) { + const qc = new QueryClient({ defaultOptions: { queries: { retry: false }, mutations: { retry: false } } }); + render( + + + + + , + ); +} + +beforeEach(() => { + jest.clearAllMocks(); + m(Resources.alignTranscript).mockResolvedValue({ data: { ok: true, job: 12 }, response: { ok: true, status: 202 } }); +}); + +describe("lining a transcript up with its audio", () => { + it("is offered only for estimated times with audio, and not while a job is at it", () => { + expect(offerAlign(estimated, true, [])).toBe(true); + expect(offerAlign(estimated, false, [])).toBe(false); // attach the audio first + expect(offerAlign({ times: "aligned" }, true, [])).toBe(false); + expect(offerAlign({ times: null }, true, [])).toBe(false); // they came with it + expect(offerAlign(estimated, true, [job("running")])).toBe(false); + expect(offerAlign(estimated, true, [job("held")])).toBe(false); + expect(offerAlign(estimated, true, [job("failed"), job("running", "analyze")])).toBe(true); + }); + + it("says whether the times are estimated or from the audio", () => { + expect(timesLabel(estimated)).toBe("Times estimated"); + expect(timesLabel({ times: "aligned", aligned: { matched: 0.754 } })).toBe( + "Times from the audio · 75% of words heard", + ); + expect(timesLabel({ times: "aligned" })).toBe("Times from the audio"); + expect(timesLabel({ times: null })).toBeNull(); + }); + + it("lines it up in one click, then gets out of the way", async () => { + show(); + expect(screen.getByText("These times are estimated from the text.")).toBeInTheDocument(); + fireEvent.click(screen.getByRole("button", { name: "Line up with the audio" })); + await waitFor(() => + expect(Resources.alignTranscript).toHaveBeenCalledWith(expect.objectContaining({ path: { rid: 7 } })), + ); + await waitFor(() => + expect(screen.queryByRole("button", { name: "Line up with the audio" })).not.toBeInTheDocument(), + ); + expect(toast).toHaveBeenCalledWith(expect.objectContaining({ title: "Lining up with the audio" })); + }); + + it("is for editors only", () => { + show({ editable: false }); + expect(screen.getByRole("button", { name: "Line up with the audio" })).toBeDisabled(); + }); +}); diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index 1c1d36b0..002a1c72 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -151,6 +151,12 @@ export type { AddVaultPasskeyResponse, AddVaultPasskeyResponses, AgentAnswer, + Aligned, + AlignTranscriptData, + AlignTranscriptError, + AlignTranscriptErrors, + AlignTranscriptResponse, + AlignTranscriptResponses, AllS3KeysData, AllS3KeysResponse, AllS3KeysResponses, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 3120fa1a..bc6f1bc6 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -52,6 +52,9 @@ import type { AddVaultPasskeyData, AddVaultPasskeyErrors, AddVaultPasskeyResponses, + AlignTranscriptData, + AlignTranscriptErrors, + AlignTranscriptResponses, AllS3KeysData, AllS3KeysResponses, AnalyticsStatusData, @@ -6500,6 +6503,26 @@ export class Resources { }); } + /** + * Align Transcript + * + * Line an imported transcript up with its audio (docs/processing.md#aligning): the audio is transcribed in the + * background as the transcribe step does (the same engine, ledger and budgets; a heavy step, so quiet processing + * hours apply), the transcript's words are matched to the words heard, and each line takes the times of its words; + * then analyze, embed and report run again. The text and speakers stay as they are; `times` becomes aligned. Asking + * while it runs gives the same job. Needs the right to process the asset; 400 when it has no audio, or its lines' + * times came with it or from transcription (`times` is neither estimated nor aligned). Audited as + * `recording.align`. + */ + public static alignTranscript( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/resources/{rid}/align", + ...options, + }); + } + /** * Revoke Shares * diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 1e851121..fc82737a 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -638,6 +638,33 @@ export type AgentAnswer = { [key: string]: unknown; }; +/** + * Aligned + * + * An imported transcript lined up with its audio (docs/processing.md#aligning). + */ +export type Aligned = { + /** + * At + * + * when + */ + at?: string | null; + /** + * Matched + * + * the share of the transcript's words heard in the audio, 0 to 1 + */ + matched?: number | null; + /** + * Engine + * + * the speech engine that heard them + */ + engine?: string | null; + [key: string]: unknown; +}; + /** * AnalyticsChurn */ @@ -21703,6 +21730,16 @@ export type Recording = { * a person set its language; transcription doesn't change it */ language_set?: boolean; + /** + * Times + * + * an imported transcript's line times: estimated from the text (it had none), or aligned with its audio (POST /resources/{rid}/align); none when they came with it or from transcription + */ + times?: "estimated" | "aligned" | null; + /** + * with times aligned: when, and how many of its words were heard + */ + aligned?: Aligned | null; /** * an email: its subject, sender, recipients and date */ @@ -39294,6 +39331,36 @@ export type EnhanceRecordingResponses = { export type EnhanceRecordingResponse = EnhanceRecordingResponses[keyof EnhanceRecordingResponses]; +export type AlignTranscriptData = { + body?: never; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/resources/{rid}/align"; +}; + +export type AlignTranscriptErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type AlignTranscriptError = AlignTranscriptErrors[keyof AlignTranscriptErrors]; + +export type AlignTranscriptResponses = { + /** + * Successful Response + */ + 202: JobQueued; +}; + +export type AlignTranscriptResponse = AlignTranscriptResponses[keyof AlignTranscriptResponses]; + export type RevokeSharesData = { body?: never; path: { diff --git a/nextjs-frontend/components/recording/align-model.ts b/nextjs-frontend/components/recording/align-model.ts new file mode 100644 index 00000000..96ef5e7c --- /dev/null +++ b/nextjs-frontend/components/recording/align-model.ts @@ -0,0 +1,27 @@ +/** + * Lining an imported transcript up with its audio (docs/processing.md, Lining up a transcript with its audio): when + * the page offers it, and what it says about the lines' times. Pure functions. + */ +import type { Recording } from "@/app/openapi-client/types.gen"; +import { isActive, type JobInfo } from "@/components/recording/jobs"; + +type Times = Pick; + +/** A job still to line it up (queued, running or held). */ +export function aligning(jobs: readonly JobInfo[]): boolean { + return jobs.some((j) => (isActive(j) || j.status === "held") && j.steps.some((s) => s.type === "align")); +} + +/** Offer it: the lines' times are estimated from the text, the asset has its audio now, and no job is at it already. */ +export function offerAlign(rec: Times, hasAudio: boolean, jobs: readonly JobInfo[]): boolean { + return rec.times === "estimated" && hasAudio && !aligning(jobs); +} + +/** What the lines' times are, for the meta line: estimated, or from the audio and how much of it was heard; null when + * they came with the transcript or from transcription. */ +export function timesLabel(rec: Times): string | null { + if (rec.times === "estimated") return "Times estimated"; + if (rec.times !== "aligned") return null; + const m = rec.aligned?.matched; + return m != null ? `Times from the audio · ${Math.round(m * 100)}% of words heard` : "Times from the audio"; +} diff --git a/nextjs-frontend/components/recording/align.tsx b/nextjs-frontend/components/recording/align.tsx new file mode 100644 index 00000000..e9bfbf89 --- /dev/null +++ b/nextjs-frontend/components/recording/align.tsx @@ -0,0 +1,55 @@ +"use client"; + +import { AudioLines } from "lucide-react"; + +import type { Recording } from "@/app/openapi-client/types.gen"; +import { offerAlign } from "@/components/recording/align-model"; +import { useAlign } from "@/components/recording/hooks"; +import type { JobInfo } from "@/components/recording/jobs"; +import { Banner } from "@/components/ui/banner"; +import { Button } from "@/components/ui/button"; + +/** On an imported transcript whose lines' times are estimated and which has its audio now: one button lines it up with + * the audio, in the background (docs/processing.md, Lining up a transcript with its audio). The job's banner shows + * its progress; when it finishes the page refreshes and this goes. */ +export function AlignBanner({ + rid, + rec, + hasAudio, + jobs, + editable, + reason, + className, +}: { + rid: number; + rec: Pick; + hasAudio: boolean; + jobs: readonly JobInfo[]; + editable: boolean; + reason?: string; + className?: string; +}) { + const align = useAlign(rid); + if (!offerAlign(rec, hasAudio, jobs) || align.isSuccess) return null; + return ( + } + disabled={!editable || align.isPending} + disabledReason={!editable ? reason : undefined} + onClick={() => align.mutate()} + > + Line up with the audio + + } + > + Lining the transcript up with its audio plays each line where it's said. It transcribes the audio in the + background; the text and speakers stay as they are. + + ); +} diff --git a/nextjs-frontend/components/recording/header.tsx b/nextjs-frontend/components/recording/header.tsx index e150c810..6a092f39 100644 --- a/nextjs-frontend/components/recording/header.tsx +++ b/nextjs-frontend/components/recording/header.tsx @@ -41,6 +41,8 @@ import { usePlayerApi } from "@/components/player/media"; import { EvidencePartsDialog } from "@/components/evidence/parts-dialog"; import { NsfwNotice } from "@/components/nsfw/nsfw"; import { CreatePodcastDialog } from "@/components/podcasts/create-dialog"; +import { AlignBanner } from "@/components/recording/align"; +import { timesLabel } from "@/components/recording/align-model"; import { useRec } from "@/components/recording/context"; import { pageAt, pagesSummary } from "@/components/recording/document/model"; import { useEvidence, useExport, usePipelines, useRecordingActions, useSpec } from "@/components/recording/hooks"; @@ -199,6 +201,7 @@ export function RecordingHeader() { {transcriptOnly && Text only} + {!transcriptOnly && timesLabel(rec) && {timesLabel(rec)}} @@ -516,10 +519,20 @@ function MoreMenu({ compact }: { compact?: boolean }) { /** R8 (running, just imported) and R6 (a step failed) banners under the meta line. */ export function Banners({ className }: { className?: string }) { + const { id, rec, model, jobs, canEdit, ns } = useRec(); return ( <> + ); } diff --git a/nextjs-frontend/components/recording/hooks.ts b/nextjs-frontend/components/recording/hooks.ts index 1ad56667..8992ebe6 100644 --- a/nextjs-frontend/components/recording/hooks.ts +++ b/nextjs-frontend/components/recording/hooks.ts @@ -492,6 +492,31 @@ export function useSetLanguage(id: number) { }); } +/** Line an imported transcript up with its audio, in the background (docs/processing.md, Lining up a transcript with + * its audio); the job's banner shows how it goes, and the page refreshes when it finishes. */ +export function useAlign(id: number) { + const client = useApiClient(); + const qc = useQueryClient(); + const toast = useToast(); + return useMutation({ + mutationFn: () => data(Resources.alignTranscript({ client, path: { rid: id } })), + onSuccess: () => { + void qc.invalidateQueries({ queryKey: rk.jobs(id) }); + toast({ + title: "Lining up with the audio", + body: "The lines move to where they’re said when it finishes; the job is in Activity.", + tone: "intent", + }); + }, + onError: (e: unknown) => + toast({ + title: "Couldn’t line up the transcript", + body: e instanceof ApiError ? e.message : "Please try again.", + tone: "red", + }), + }); +} + /** A video's copy with the faces blurred for sharing, and the job making one: asked again every few seconds while it * runs, so its progress shows (docs/video.md, Blurring faces for sharing). */ export function useFaceBlur(id: number, enabled = true) { diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 05736800..1a0c2fc3 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -2988,6 +2988,7 @@ export const FIELDS: FieldSpec[] = [ "objects", "nsfw", "blur_faces", + "align", "describe", "embed", ].map((s) => ({ diff --git a/nextjs-frontend/components/ui/loop.tsx b/nextjs-frontend/components/ui/loop.tsx index 613a65f9..ddd9bd56 100644 --- a/nextjs-frontend/components/ui/loop.tsx +++ b/nextjs-frontend/components/ui/loop.tsx @@ -29,6 +29,7 @@ export const STEP_TONE: Record = { reply: "gate", translate: "gate", blur_faces: "red", + align: "intent", report: "gate", export: "gate", workflow: "green", @@ -56,6 +57,7 @@ export const STEP_LABEL: Record = { reply: "Reply", translate: "Translate", blur_faces: "Blur faces", + align: "Line up", report: "Report", export: "Export", workflow: "Workflow", diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index a7ff665d..2412e303 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -11930,6 +11930,49 @@ } } }, + "/api/v1/resources/{rid}/align": { + "post": { + "tags": [ + "resources" + ], + "summary": "Align Transcript", + "description": "Line an imported transcript up with its audio (docs/processing.md#aligning): the audio is transcribed in the\nbackground as the transcribe step does (the same engine, ledger and budgets; a heavy step, so quiet processing\nhours apply), the transcript's words are matched to the words heard, and each line takes the times of its words;\nthen analyze, embed and report run again. The text and speakers stay as they are; `times` becomes aligned. Asking\nwhile it runs gives the same job. Needs the right to process the asset; 400 when it has no audio, or its lines'\ntimes came with it or from transcription (`times` is neither estimated nor aligned). Audited as\n`recording.align`.", + "operationId": "align_transcript", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "responses": { + "202": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/JobQueued" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/resources/{rid}/share": { "post": { "tags": [ @@ -39406,6 +39449,50 @@ ], "title": "AgentAnswer" }, + "Aligned": { + "properties": { + "at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "At", + "description": "when" + }, + "matched": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Matched", + "description": "the share of the transcript's words heard in the audio, 0 to 1" + }, + "engine": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Engine", + "description": "the speech engine that heard them" + } + }, + "additionalProperties": true, + "type": "object", + "title": "Aligned", + "description": "An imported transcript lined up with its audio (docs/processing.md#aligning)." + }, "AnalyticsChurn": { "properties": { "previous_active": { @@ -75507,6 +75594,33 @@ "description": "a person set its language; transcription doesn't change it", "default": false }, + "times": { + "anyOf": [ + { + "type": "string", + "enum": [ + "estimated", + "aligned" + ] + }, + { + "type": "null" + } + ], + "title": "Times", + "description": "an imported transcript's line times: estimated from the text (it had none), or aligned with its audio (POST /resources/{rid}/align); none when they came with it or from transcription" + }, + "aligned": { + "anyOf": [ + { + "$ref": "#/components/schemas/Aligned" + }, + { + "type": "null" + } + ], + "description": "with times aligned: when, and how many of its words were heard" + }, "email": { "anyOf": [ {