From 086316cef168b893777f1fb4c91ba5fb1e383bac Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Mon, 19 Oct 2026 13:57:51 +0000 Subject: [PATCH] feat(import): credit WebVTT voice spans as each line's speaker An imported WebVTT transcript now reads a cue's voice span, or , closed by or not, as the line's speaker, so the people a file names join the namespace like other named speakers. A cue with two voices becomes two lines sharing its time by word count; cues without a voice are read as before. A YouTube cue starting with a line of one space is no longer dropped. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01QFUNzy1QZy5HhKAyJxTAYJ --- README.md | 3 +- changelog.d/vtt-voices.md | 5 ++ docs/processing.md | 5 +- docs/requirements.md | 1 + fastapi_backend/app/domain/ingest.py | 44 ++++++++++-- .../tests/domain/test_importing.py | 69 +++++++++++++++++++ 6 files changed, 121 insertions(+), 6 deletions(-) create mode 100644 changelog.d/vtt-voices.md diff --git a/README.md b/README.md index ea4ddd244..e789b991d 100644 --- a/README.md +++ b/README.md @@ -144,7 +144,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs, Office files, photos, JSON and zip or tar archives (photos and videos dated by when they were taken, from the file - itself, and photos placed by their GPS), and, with an Apache Tika server (an opt-in extra), documents nothing else + itself, and photos placed by their GPS), and transcripts made elsewhere (SRT, WebVTT, text, Word, PDF), each line + credited to the speaker the file names; and, with an Apache Tika server (an opt-in extra), documents nothing else here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off by default), taking a video's own captions as its transcript when people wrote some (off by default too). Say what a diff --git a/changelog.d/vtt-voices.md b/changelog.d/vtt-voices.md new file mode 100644 index 000000000..63e0bd818 --- /dev/null +++ b/changelog.d/vtt-voices.md @@ -0,0 +1,5 @@ +- **WebVTT voices name the speaker.** A transcript imported from a WebVTT file now credits each line to the speaker its + voice span names, `` or `` with classes, closed by `` or not, and those speakers join the + namespace's people like any other named speaker. A cue with two voices becomes two lines that share its time; the tags + are left out of the text, and a cue without a voice is read as before. A YouTube caption cue that starts with a line + of one space is no longer lost ([Importing text and transcripts](docs/processing.md#importing-transcripts)). diff --git a/docs/processing.md b/docs/processing.md index dbfc38e6a..e5e740607 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -205,7 +205,10 @@ and faces (video only), analyze and report. The asset takes the file's fingerpri same file find it, unless another asset in the namespace has it already. Speakers are recognised from `Name: text`, `[12:30] Name: text`, `Name (12:30): text`, Otter/Zoom/Teams exports (a -`Name 12:30` line, then what they said), `speaker|emotion|text` lines, and the speakers in SRT/VTT and JSON. Anything +`Name 12:30` line, then what they said), `speaker|emotion|text` lines, and the speakers in SRT/VTT and JSON. In +WebVTT, a cue's voice span names its speaker: ``, or `` with classes after the `v`, closed by +`` or not. The tags are left out of the text, and a cue with two voices becomes two lines that share its time by +how many words each says; a cue without a voice takes its speaker from a `Name: text` line, or has none. Anything else becomes paragraphs split into segments of about 40 words with estimated times. Markdown and MDX are reduced to text first: front matter or the first heading becomes the title; imports, exports, JSX and code blocks are dropped. A PDF imported as a transcript needs a text layer; import a scan as a document instead, and its pages are read by OCR. Named speakers are matched to the namespace's people by name; generic labels (SPEAKER_00, S1, CH0) diff --git a/docs/requirements.md b/docs/requirements.md index 527f7ee58..617b3e178 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -169,6 +169,7 @@ request, to another check that proves the same thing; loosening a requirement or | Import webhooks hand back the PDFs of documents a light server can't convert | [test_import_hooks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_hooks.py) `test_a_hook_hands_back_the_pdfs_of_documents_this_server_cant_convert` | | Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` | | Uploads of audio, PDFs, Office files, photos, JSON and archives | [test_uploads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_uploads.py) `test_uploading_audio_in_chunks`, `test_json_uploaded_is_read_as_records`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_a_pdf_becomes_a_document_with_its_pages_and_text`, `test_scans_and_images_are_read_by_ocr`; [test_conversion.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversion.py) `test_office_and_text_files_become_documents`; [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_an_uploaded_zip_keeps_its_files`; [test_archives.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_archives.py) `test_reading_an_archive`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `import-file`, `import-paste` | +| Transcripts made elsewhere (text, Markdown, Word, PDF, SRT, WebVTT, JSON) imported with each line credited to the speaker the file names, WebVTT voice spans (``, ``) included | [test_importing.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_importing.py) `test_formats_and_speakers_by_name`, `test_webvtt_voice_spans_name_each_line` | | Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` | | Photos and videos dated by when they were taken, from the file itself (a photo's EXIF date and GPS fix as `place`, a video's creation time) when no name or source dates them; a date already set is kept and none is invented | [test_file_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_dates.py) `test_a_photo_is_dated_and_placed_by_its_exif`, `test_a_date_already_set_is_kept`, `test_a_video_is_dated_by_its_creation_time`, `test_a_scanned_video_is_dated_by_its_creation_time` | | An uploaded video, end to end | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) | diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index 5e50572ca..b82942d5b 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -733,6 +733,8 @@ def _generic(s): PIPE = re.compile(r"^([^|]{1,40})\|([^|]{0,24})\|(.+)$") SPK = re.compile(r"^(?:\[?((?:\d{1,2}:)?\d{1,2}:\d{2})\]?\s+)?([A-Z][\w .'’-]{0,30}?)\s*:\s+(\S.*)$") +# a WebVTT voice span's start tag: , or with classes after the v +VOICE = re.compile(r"]*)?(?:\s+([^>]*))?>") VTT_TIME = re.compile(r"((?:\d+:)?\d{1,2}:\d{2}[.,]\d{1,3})\s*-->\s*((?:\d+:)?\d{1,2}:\d{2}[.,]\d{1,3})") @@ -743,20 +745,54 @@ def _clock(s): def _cues(raw): segs = [] - for block in re.split(r"\n\s*\n", raw.replace("\r", "")): + raw = re.sub(r"(-->[^\n]*\n)[ \t]+\n", r"\1", raw.replace("\r", "")) # YouTube's cues start with a line of one space + for block in re.split(r"\n\s*\n", raw): lines = [l for l in block.strip().split("\n") if l.strip()] for i, l in enumerate(lines): m = VTT_TIME.search(l) if m: + t0, t1 = _clock(m.group(1)), _clock(m.group(2)) + voices = _voices("\n".join(lines[i + 1 :])) + if voices: + segs += _voiced(t0, t1, voices) + break text = " ".join(lines[i + 1 :]).strip() - sm = SPK.match(text) or re.match(r"^]+)>(.*)$", text) - spk, text = (sm.group(sm.lastindex - 1).strip(), sm.group(sm.lastindex).strip()) if sm else (None, text) + sm = SPK.match(text) + spk, text = (sm.group(2).strip(), sm.group(3).strip()) if sm else (None, text) if text: - segs.append({"t0": _clock(m.group(1)), "t1": _clock(m.group(2)), "text": re.sub(r"<[^>]+>", "", text), "speaker": spk}) + segs.append({"t0": t0, "t1": t1, "text": re.sub(r"<[^>]+>", "", text), "speaker": spk}) break return segs +def _voices(body): + """A cue's WebVTT voice spans (, , closed by or the next one) as [(speaker, text)], one + per change of voice; None when it names no voice.""" + parts = VOICE.split(body) + if len(parts) == 1: + return None + runs = [] + for name, text in [(None, parts[0]), *zip(parts[1::2], parts[2::2])]: + text = " ".join(re.sub(r"<[^>]+>", "", text).split()) + spk = (html.unescape(name).strip() or None) if name else None + if not text: + continue + if runs and runs[-1][0] == spk: + runs[-1] = (spk, runs[-1][1] + " " + text) + else: + runs.append((spk, text)) + return runs or None + + +def _voiced(t0, t1, runs): + """One line per voice in a cue, sharing its time by how many words each says.""" + n, w, out = sum(len(x.split()) for _, x in runs), 0, [] + for spk, text in runs: + start, w = t0 + (t1 - t0) * w // n, w + len(text.split()) + out.append({"t0": start, "t1": t0 + (t1 - t0) * w // n, "text": text, "speaker": spk}) + return out + + HEADER = re.compile(r"^([A-Z][\w .'’-]{0,40}?)\s*[\(\[]?((?:\d{1,2}:)?\d{1,2}:\d{2})(?:\.\d+)?[\)\]]?\s*:?\s*$") PAREN = re.compile(r"^([A-Z][\w .'’-]{0,40}?)\s*[\(\[]((?:\d{1,2}:)?\d{1,2}:\d{2})[\)\]]\s*:\s*(\S.*)$") diff --git a/fastapi_backend/tests/domain/test_importing.py b/fastapi_backend/tests/domain/test_importing.py index 8ca612357..4851e33f4 100644 --- a/fastapi_backend/tests/domain/test_importing.py +++ b/fastapi_backend/tests/domain/test_importing.py @@ -127,6 +127,75 @@ def test_paste_and_chunk_stitching(db, cfg): assert naive * 0.6 < stitched < naive * 0.9 +YOUTUBE_VTT = """WEBVTT +Kind: captions +Language: en + +00:00:00.160 --> 00:00:02.869 align:start position:0% +\x20 +welcome<00:00:00.480> back<00:00:00.880> to<00:00:01.040> the<00:00:01.280> channel + +00:00:02.869 --> 00:00:05.000 align:start position:0% +today<00:00:03.200> we're<00:00:03.440> looking<00:00:03.760> at<00:00:04.000> tides +""" + +MEETING_VTT = """WEBVTT + +1b2c4d5e-2a7f-4b1c-9d3e-0f6a8b7c5d4e/12-0 +00:00:03.120 --> 00:00:06.480 +Can everyone hear me okay? + +1b2c4d5e-2a7f-4b1c-9d3e-0f6a8b7c5d4e/13-0 +00:00:06.900 --> 00:00:09.300 +Yes, loud and clear. + +1b2c4d5e-2a7f-4b1c-9d3e-0f6a8b7c5d4e/14-0 +00:00:09.300 --> 00:00:12.000 +Great, let's start with the +budget review. +""" + +CLASSED_VTT = """WEBVTT + +00:00:00.000 --> 00:00:02.000 +It's a blue apple tree! + +00:00:02.000 --> 00:00:04.000 +No way! +Oh yes! + +00:00:04.000 --> 00:00:05.000 +Nobody named here. +""" + + +def test_webvtt_voice_spans_name_each_line(db, cfg): + yt = ingest.read_text_transcript(YOUTUBE_VTT, "auto")["segments"] # no voices: as before, the inline timings dropped + assert [(s["speaker"], s["text"], s["t0"], s["t1"]) for s in yt] == [ + (None, "welcome back to the channel", 160, 2869), + (None, "today we're looking at tides", 2869, 5000), + ] + meeting = ingest.read_text_transcript(MEETING_VTT, "vtt")["segments"] # Teams and Zoom: ... + assert [(s["speaker"], s["text"]) for s in meeting] == [ + ("Priya Natarajan", "Can everyone hear me okay?"), + ("Tom O'Brien", "Yes, loud and clear."), + ("Priya Natarajan", "Great, let's start with the budget review."), + ] + classed = ingest.read_text_transcript(CLASSED_VTT, "vtt")["segments"] # classes after the v; two voices in one cue + assert [(s["speaker"], s["text"], s["t0"], s["t1"]) for s in classed] == [ + ("Esme", "It's a blue apple tree!", 0, 2000), + ("Mary", "No way!", 2000, 3000), + ("Esme", "Oh yes!", 3000, 4000), + (None, "Nobody named here.", 4000, 5000), + ] + srt = "1\n00:00:01,000 --> 00:00:02,500\nAlice: still read from the line\n" # a cue without a voice: unchanged + assert [(s["speaker"], s["text"]) for s in ingest.read_text_transcript(srt, "srt")["segments"]] == [ + ("Alice", "still read from the line") + ] + ingest.import_text(db, cfg, "pods", MEETING_VTT, title="Standup") # the voices become the namespace's people + assert speaker_names(db, "pods") == {"Priya Natarajan", "Tom O'Brien"} + + def test_an_engine_that_isnt_installed_falls_back_to_one_that_is(monkeypatch): """faster-whisper is there (EXTRAS=whisper), SenseVoice (the default engine) not yet: imports are transcribed anyway.""" made = []