diff --git a/README.md b/README.md index fd6ca836..d647c2c4 100644 --- a/README.md +++ b/README.md @@ -181,15 +181,15 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p (noise, volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server you run), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each - off by default. + off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place. ([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages), [similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md), [topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings), [cleaning up audio](docs/processing.md#cleaning-up-audio), [your own names](docs/processing.md#your-own-names-in-transcription), - [cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates); - [checks](docs/requirements.md#understand-it)) + [cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates), + [unsure words](docs/processing.md#unsure-words); [checks](docs/requirements.md#understand-it)) - **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its source, read aloud in two voices, by Kokoro on this machine or a speech provider. Turn a voice session or a meeting into a technical spec, in a note linked to it, that keeps every decision, every no and every constraint in the speaker's own words, linked to when they said them, diff --git a/changelog.d/word-confidence.md b/changelog.d/word-confidence.md new file mode 100644 index 00000000..9d832ca9 --- /dev/null +++ b/changelog.d/word-confidence.md @@ -0,0 +1,9 @@ +- **See the words transcription wasn't sure of.** Whisper, Deepgram, AssemblyAI and ElevenLabs say how sure they + were of each word, and Lens now keeps it with the word's timing instead of dropping it. In the transcript, a word + below the threshold gets a dotted gold underline and a tooltip ("Transcription wasn't sure of this word (42%)"), so + you can spot it and correct it in place; correcting it takes the mark away. Settings → Transcription → Mark words + it was less sure of than sets the threshold (`transcribe.confidence_highlight`, 0.6; 0 marks none). `GET + …/player` gives each word's confidence as a fifth in `w`, and the threshold as `unsure_below`; the MCP tool + `get_transcript` lists each line's `unsure` words. Transcripts made before, and those from SenseVoice and OpenAI's + own API, which don't say, have no marks and work as before + ([Words transcription wasn't sure of](docs/processing.md#unsure-words)). diff --git a/docs/api.md b/docs/api.md index 41c8457a..21bfdc9e 100644 --- a/docs/api.md +++ b/docs/api.md @@ -716,8 +716,10 @@ An asset's `summary` (the Summarize step's, also in `GET …/player`) has `summa summaries made before they had times hold plain strings. In `GET …/player`, a line whose words have timings from transcription carries them as `w`: `[c0, c1, t0, t1]` for -each word, a character range of its `text` and when it was said (ms). A corrected line keeps the timings of the words -it still has. +each word, a character range of its `text` and when it was said (ms), with a fifth, how sure the engine was of the word +(0 to 1), when it said. A corrected line keeps the timings of the words it still has. `unsure_below` is the threshold +the web app marks words under (`transcribe.confidence_highlight`), or null when marking is off +([Words transcription wasn't sure of](processing.md#unsure-words)). Assets carry `tags`. `PATCH /resources/{rid}` with `{"tags": [...]}` replaces an asset's tags (editors; at most 20, 40 characters each; whitespace is collapsed and repeats are dropped, ignoring case). `POST /resources/tags` with diff --git a/docs/mcp.md b/docs/mcp.md index 72c88f0c..500b00ec 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -78,7 +78,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) | | `list_speakers` | the people of your namespaces, the ones with the most talk time first | | `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW | -| `get_transcript` | an asset's lines in order (who, time, text, link), a page at a time, from a line or a time window | +| `get_transcript` | an asset's lines in order (who, time, text, link, and the words transcription wasn't sure of as `unsure`), a page at a time, from a line or a time window | | `fetch` | an asset's whole text at once, as `[m:ss] Speaker: …` lines | | `cite` | a citation for a moment (a line or a second, and a few lines from there): the words, who said or wrote them, the asset and date, the time, the link, and all of it as Markdown | | `list_entities` | people, organisations, products, places, events, works and terms mentioned, by name, type, namespace or asset | diff --git a/docs/processing.md b/docs/processing.md index c633c7c7..48eb06c1 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -140,6 +140,23 @@ custom pipeline by adding the Clean up step before Analyze (with Force it runs e on an asset with Reprocess → Clean up, `POST /api/v1/recordings/{id}/reprocess` with `{"steps": ["clean", "analyze"]}`, or `lens clean`. Workers whose `workers.steps` list has `analyze` run it too. +### Words transcription wasn't sure of {#unsure-words} + +Whisper (faster-whisper and mlx-whisper), Deepgram, AssemblyAI and ElevenLabs say how sure they were of each word, and +Lens keeps that with the word's timing. In the transcript, a word it was less sure of than 60% gets a dotted gold +underline, and its tooltip says how sure ("Transcription wasn't sure of this word (42%)"), so you can find the words +worth checking and correct them in Edit. Clicking it plays the line, as it does anywhere in the text. A corrected word +loses its mark, and splitting or merging lines keeps the marks on the words they still have. + +Settings → Transcription → **Mark words it was less sure of than** sets the threshold (`transcribe.confidence_highlight`, +0.6 by default, from 0 to 0.99); 0 marks none. SenseVoice, the default engine, and OpenAI's own API don't say how sure +they were, so their transcripts have no marks (an OpenAI-compatible server that gives each word a `probability` is +kept too), and nor do transcripts made before Lens kept it. + +Agents get the same: `GET /api/v1/recordings/{id}/player` gives each word's confidence and the threshold ([API](api.md)), +and the MCP tool `get_transcript` lists each line's `unsure` words with how sure, worth checking before quoting +([MCP](mcp.md)). + ## Importing text and transcripts {#importing-transcripts} Formats: .txt, .md, .markdown, .mdx, .docx, .doc, .pdf, .srt, .vtt, .json (lens, Whisper or a list of segments) and diff --git a/docs/requirements.md b/docs/requirements.md index 7523d1c4..dcd16991 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -199,6 +199,7 @@ request, to another check that proves the same thing; loosening a requirement or | Transcription that expects the names the namespace knows (custom vocabulary, named people, defined entities, names people fixed), off by default | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_the_namespaces_own_names_go_to_the_engine` | | Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) | | One asset's voice split off from the person it was matched to, with Undo | [test_speakers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_speakers.py) `test_detach_one_assets_voice` | +| How sure transcription was of each word kept (Whisper, Deepgram, AssemblyAI, ElevenLabs), words from before still loading, the unsure ones marked in the transcript with how sure, a threshold in Settings (0 for none), and `unsure` words per line over MCP | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_providers_keep_how_sure_they_were_of_each_word`, `test_whisper_keeps_how_sure_it_was_of_each_word`; [test_split_merge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_split_merge.py) `test_words_keep_how_sure_transcription_was`, `test_words_from_before_how_sure_was_kept_still_load`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_get_transcript_says_which_words_transcription_was_unsure_of`; [recording-split-words.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-split-words.test.ts) `keeps how sure transcription was of each word, and words from before it was kept`, `marks the words below the threshold, and none when it's off`; [recording-unsure-words.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-unsure-words.test.tsx) `are underlined with how sure, and a click still plays the line` | | People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` | | An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` | | Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` | diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index f4d614b6..cbf6c99d 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -86,6 +86,7 @@ settings, sources, store, + transcript, untrusted, updates, variables, @@ -437,8 +438,9 @@ def _line(ctx: Context, rid: int, s: dict[str, Any], names: dict[int, str]) -> d return out -def _segments(db: DB, rid: int, where: str = "", limit: int | None = None, **params: Any) -> list[dict[str, Any]]: - sql = f"SELECT idx, t0, t1, speaker, text, page FROM segment WHERE recording = $r{where} ORDER BY idx" +def _segments(db: DB, rid: int, where: str = "", limit: int | None = None, with_words: bool = False, **params: Any) -> list[dict[str, Any]]: + cols = "idx, t0, t1, speaker, text, page" + (", words" if with_words else "") + sql = f"SELECT {cols} FROM segment WHERE recording = $r{where} ORDER BY idx" if limit is not None: sql += f" LIMIT {int(limit)}" return db.rows(sql, r=rid, **params) @@ -970,7 +972,8 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]: "Read an asset's text", "The lines of an asset's text (the transcript of audio or video, or a document's or image's text), in order: who " "said or wrote what, and when, each with " - "a url that opens the asset at that line. Read a long one in pages: from_line (next_line of the last page) or " + "a url that opens the asset at that line, and `unsure`: the words transcription wasn't sure of, each with how sure " + "(0 to 1), worth checking before quoting them. Read a long one in pages: from_line (next_line of the last page) or " "a time window in seconds.", Arg("recording_id", "integer", "the asset", required=True), Arg("from_line", "integer", "start at this line (0 is the first)", minimum=0), @@ -992,15 +995,23 @@ def get_transcript( if end_seconds is not None: where += " AND t0 <= $b" params["b"] = int(end_seconds * 1000) - segs = _segments(ctx.db, recording_id, where, limit + 1, **params) + below = ctx.cfg["transcribe"].get("confidence_highlight") + segs = _segments(ctx.db, recording_id, where, limit + 1, with_words=bool(below), **params) more, segs = len(segs) > limit, segs[:limit] names = _names(ctx.db, [s.get("speaker") for s in segs]) total = ctx.db.values("RETURN array::len((SELECT VALUE id FROM segment WHERE recording = $r))", r=recording_id) + lines = [] + for s in segs: + line = _line(ctx, recording_id, s, names) + unsure = transcript.unsure(s.get("text") or "", transcript.words(s), below) + if unsure: + line["unsure"] = [{"word": w, "sure": p} for w, p in unsure] + lines.append(line) out: dict[str, Any] = { "recording_id": recording_id, "title": rec.get("title"), "lines_in_all": int(total[0]) if total else 0, - "lines": [_line(ctx, recording_id, s, names) for s in segs], + "lines": lines, } if more: out["next_line"] = segs[-1]["idx"] + 1 diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py index e2d4b511..a1f44178 100644 --- a/fastapi_backend/app/api/v1/routes/recordings.py +++ b/fastapi_backend/app/api/v1/routes/recordings.py @@ -621,6 +621,7 @@ def get_player(rid: int, acl: Acl, db: Db, cfg: Cfg, s: str = "") -> Player: rec = acl.recording(rid, share=s) data = render.player_data(db, rid, audio_link(db, cfg, rid, s)) data["nsfw"] = _nsfw(db, cfg, rec, rid) + data["unsure_below"] = cfg["transcribe"].get("confidence_highlight") or None return sign_urls(data, full=acl.member(rec)) diff --git a/fastapi_backend/app/domain/audio_cleanup.py b/fastapi_backend/app/domain/audio_cleanup.py index 8b5b68c3..9a14f31a 100644 --- a/fastapi_backend/app/domain/audio_cleanup.py +++ b/fastapi_backend/app/domain/audio_cleanup.py @@ -247,7 +247,7 @@ def remap(segs, joins): s["t0"], s["t1"] = to_recording(s["t0"], joins), to_recording(s["t1"], joins, end=True) if s.get("words"): words = json.loads(s["words"]) - s["words"] = json.dumps([[w, to_recording(a, joins), to_recording(b, joins, end=True)] for w, a, b in words]) + s["words"] = json.dumps([[w[0], to_recording(w[1], joins), to_recording(w[2], joins, end=True), *w[3:]] for w in words]) return segs diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index 99123826..5e50572c 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -16,7 +16,7 @@ import numpy as np -from . import speech, store, vocabulary +from . import speech, store, transcript, vocabulary TAG_RX = re.compile(r"<\|([^|]*)\|>") SR = 16000 @@ -396,7 +396,8 @@ def transcribe(self, audio, terms=()): # SenseVoice takes no hints (vocabulary. class Whisper: - """faster-whisper (CUDA or CPU) or mlx-whisper (Apple Silicon), with word timestamps.""" + """faster-whisper (CUDA or CPU) or mlx-whisper (Apple Silicon), with word timestamps and how sure it was of each + word.""" def __init__(self, cfg, mlx=False): t = cfg["transcribe"] @@ -425,11 +426,16 @@ def transcribe(self, audio, terms=()): if self.mlx: got = self.m.transcribe(audio, path_or_hf_repo=self.repo, word_timestamps=True, language=self.lang, **hint) heard, segs = got.get("language"), got["segments"] - rows = [(s["start"], s["end"], s["text"], [(w["word"], w["start"], w["end"]) for w in s.get("words") or []]) for s in segs] + rows = [ + (s["start"], s["end"], s["text"], [(w["word"], w["start"], w["end"], w.get("probability")) for w in s.get("words") or []]) + for s in segs + ] else: segs, info = self.m.transcribe(audio, vad_filter=True, word_timestamps=True, language=self.lang, **hint) heard = getattr(info, "language", None) - rows = [(s.start, s.end, s.text, [(w.word, w.start, w.end) for w in (s.words or [])]) for s in segs] + rows = [ + (s.start, s.end, s.text, [(w.word, w.start, w.end, getattr(w, "probability", None)) for w in (s.words or [])]) for s in segs + ] lang = self.lang or (str(heard).lower() if heard else None) return [ { @@ -440,7 +446,7 @@ def transcribe(self, audio, terms=()): "emotion": None, "event": None, "lang": lang, - "words": json.dumps([[w.strip(), int(p * 1000), int(q * 1000)] for w, p, q in ws]), + "words": json.dumps([transcript.stored(w.strip(), int(p * 1000), int(q * 1000), sure) for w, p, q, sure in ws]), } for a, b, x, ws in rows if x.strip() diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index 06b66268..a2033fa7 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -574,6 +574,10 @@ def _check(section, key, value, default): if not (isinstance(value, int) and not isinstance(value, bool) and 1 <= value <= 500): raise ValueError("ai.max_tool_calls is a whole number of tool calls from 1 to 500") return value + if (section, key) == ("transcribe", "confidence_highlight"): + if isinstance(value, bool) or not isinstance(value, (int, float)) or not 0 <= value < 1: + raise ValueError("transcribe.confidence_highlight is a number from 0 (mark no words) to below 1") + return float(value) if (section, key) == ("llm", "describe_max"): if not (isinstance(value, int) and not isinstance(value, bool) and 1 <= value <= 1000): raise ValueError("llm.describe_max is a whole number from 1 to 1000") diff --git a/fastapi_backend/app/domain/speech.py b/fastapi_backend/app/domain/speech.py index f1750734..fb41abf6 100644 --- a/fastapi_backend/app/domain/speech.py +++ b/fastapi_backend/app/domain/speech.py @@ -22,6 +22,10 @@ Each provider takes so much audio in one request ({p}_max_minutes and {p}_max_mb in the speech settings, defaulting to its published limits): a longer recording is cut into pieces at pauses, each piece is transcribed on its own, and their transcripts are joined back on the recording's timeline. + +How sure a provider was of each word (Deepgram's and AssemblyAI's confidence, ElevenLabs' log probability, the +`probability` an OpenAI-compatible server may give) is kept with the word's timing, for the transcript to mark the ones +it wasn't sure of (transcript.py). """ from __future__ import annotations @@ -31,6 +35,7 @@ import io import json import logging +import math import pathlib import re import shutil @@ -46,7 +51,7 @@ import numpy as np -from . import kokoro, llm, store, vocabulary +from . import kokoro, llm, store, transcript, vocabulary log = logging.getLogger("lens") SR = 16000 @@ -251,8 +256,8 @@ def _event(text): def group_words(words, lang=None): - """Timed words [{text, t0, t1, speaker}] (ms) into segments: a new one when the speaker changes, after a pause, or - at the end of a sentence once it's long enough.""" + """Timed words [{text, t0, t1, speaker, p?}] (ms; p: how sure the provider was of it, 0 to 1) into segments: a new + one when the speaker changes, after a pause, or at the end of a sentence once it's long enough.""" out, cur = [], [] def flush(): @@ -268,7 +273,9 @@ def flush(): "emotion": None, "event": next((w["event"] for w in cur if w.get("event")), None), "lang": lang, - "words": json.dumps([[w["text"], w["t0"], w["t1"]] for w in cur if not w.get("event")]), + "words": json.dumps( + [transcript.stored(w["text"], w["t0"], w["t1"], w.get("p")) for w in cur if not w.get("event")] + ), "provider_speaker": cur[0].get("speaker"), } ) @@ -297,6 +304,13 @@ def add_sentiment(segs, spans): return segs +def _from_log(v): + """A log probability as a probability (0 to 1), or None.""" + if isinstance(v, bool) or not isinstance(v, (int, float)) or v > 0: + return None + return math.exp(v) + + def _label(v): return None if v is None else f"S{v}" if isinstance(v, int) or str(v).isdigit() else str(v) @@ -326,7 +340,11 @@ def _openai(cfg, clip, seconds, terms=()): out = [] for s in segs: t0, t1 = int(float(s.get("start") or 0) * 1000), int(float(s.get("end") or 0) * 1000) - ws = [[w["word"].strip(), int(w["start"] * 1000), int(w["end"] * 1000)] for w in words if t0 <= w["start"] * 1000 < t1] + ws = [ + transcript.stored(w["word"].strip(), int(w["start"] * 1000), int(w["end"] * 1000), w.get("probability")) + for w in words + if t0 <= w["start"] * 1000 < t1 + ] text = (s.get("text") or "").strip() if any(ch.isalnum() for ch in text): out.append( @@ -382,7 +400,7 @@ def _elevenlabs(cfg, clip, seconds, terms=()): # Scribe takes no hints (vocabul if w.get("type") == "audio_event" and words and not ev: continue words.append({"text": text, "t0": int(float(w.get("start") or 0) * 1000), "t1": int(float(w.get("end") or 0) * 1000), - "speaker": _label(w.get("speaker_id")), "event": ev}) # fmt: skip + "speaker": _label(w.get("speaker_id")), "event": ev, "p": _from_log(w.get("logprob"))}) # fmt: skip return group_words(words, _short(j.get("language_code"))) @@ -422,7 +440,7 @@ def _assemblyai(cfg, clip, seconds, terms=()): raise ProviderError(f"AssemblyAI: {job.get('error') or 'the transcript failed'}") lang = _short(job.get("language_code")) words = [ - {"text": w["text"], "t0": int(w["start"]), "t1": int(w["end"]), "speaker": _label(w.get("speaker"))} + {"text": w["text"], "t0": int(w["start"]), "t1": int(w["end"]), "speaker": _label(w.get("speaker")), "p": w.get("confidence")} for w in job.get("words") or [] if (w.get("text") or "").strip() ] @@ -458,6 +476,7 @@ def _deepgram(cfg, clip, seconds, terms=()): "t0": int(w["start"] * 1000), "t1": int(w["end"] * 1000), "speaker": _label(w.get("speaker")), + "p": w.get("confidence"), } # fmt: skip for w in alt.get("words") or [] ] @@ -527,7 +546,7 @@ def _shift(segs, ms, piece): for s in segs: s["t0"], s["t1"] = s["t0"] + ms, s["t1"] + ms if ms and s.get("words"): - s["words"] = json.dumps([[w, a + ms, b + ms] for w, a, b in json.loads(s["words"])]) + s["words"] = json.dumps([[w[0], w[1] + ms, w[2] + ms, *w[3:]] for w in json.loads(s["words"])]) if piece is not None and s.get("provider_speaker"): s["provider_speaker"] = f"c{piece}:{s['provider_speaker']}" return segs diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index bbd38311..29cb5a59 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -159,6 +159,8 @@ def labels(): "snr_floor_db": 20, # with only_if_poor, a recording at or above this signal-to-noise is left as it is }, "vocabulary": False, # give each recording's engine the names its namespace knows (vocabulary.py) + # words the engine was less sure of than this (0 to 1) are marked in the transcript to check; 0: none are + "confidence_highlight": 0.6, }, "diarize": { "engine": "auto", diff --git a/fastapi_backend/app/domain/transcript.py b/fastapi_backend/app/domain/transcript.py index 1ae94177..b74109ac 100644 --- a/fastapi_backend/app/domain/transcript.py +++ b/fastapi_backend/app/domain/transcript.py @@ -1,8 +1,9 @@ """Splitting and merging transcript lines, and the timings of their words (docs/api.md#resources). A line keeps its words' timings from transcription when the engine gave them (``words``, a JSON list of -[word, t0, t1] in ms). The player gets them as character ranges of the line's current text (`align`), so a corrected -line keeps the timings of the words it still has. +[word, t0, t1] in ms, with a fourth, how sure the engine was of the word from 0 to 1, when it said). The player gets +them as character ranges of the line's current text (`align`), so a corrected line keeps the timings of the words it +still has. Lines are numbered: a segment's id is its recording's id × SEG + its index. Splitting or merging renumbers the lines after it, and what points at line numbers follows: the edit history, people's entity corrections and the chapters. @@ -24,8 +25,22 @@ TOKEN = re.compile(r"\S+") +def sure(p): + """How sure an engine was of a word, from 0 to 1 (three decimals), or None when it didn't say or it isn't one.""" + if isinstance(p, bool) or not isinstance(p, (int, float)) or not 0 <= p <= 1: + return None + return round(float(p), 3) + + +def stored(text, t0, t1, p=None): + """A timed word as a line keeps it: [word, t0, t1], with how sure the engine was as a fourth when it said.""" + p = sure(p) + return [text, t0, t1] if p is None else [text, t0, t1, p] + + def words(seg): - """The line's timed words, [(word, t0, t1)], or [] when transcription gave none.""" + """The line's timed words, [(word, t0, t1)] or (word, t0, t1, sure) where the engine said how sure it was, or [] + when transcription gave none. Lines transcribed before Lens kept how sure have three.""" raw = seg.get("words") try: ws = json.loads(raw) if isinstance(raw, str) else (raw or []) @@ -37,8 +52,9 @@ def words(seg): text, a, b = str(w[0]).strip(), int(w[1]), int(w[2]) except (TypeError, ValueError, IndexError): continue + p = sure(w[3]) if len(w) > 3 else None if text: - out.append((text, a, max(a, b))) + out.append((text, a, max(a, b)) if p is None else (text, a, max(a, b), p)) return out @@ -52,7 +68,7 @@ def _placed(text, ws): return [] if CJK.search(text): out, at = [], 0 - for i, (w, _, _) in enumerate(ws): + for i, (w, *_) in enumerate(ws): c = text.find(w, at) if c < 0 or c - at > 2 * len(w) + 8: # gone from the text: don't jump ahead to a later one continue @@ -60,7 +76,7 @@ def _placed(text, ws): at = c + len(w) return out tokens = [(m.start(), m.end(), _norm(m.group())) for m in TOKEN.finditer(text)] - sm = difflib.SequenceMatcher(None, [_norm(w) for w, _, _ in ws], [t[2] for t in tokens], autojunk=False) + sm = difflib.SequenceMatcher(None, [_norm(w[0]) for w in ws], [t[2] for t in tokens], autojunk=False) return [ (blk.a + k, tokens[blk.b + k][0], tokens[blk.b + k][1]) for blk in sm.get_matching_blocks() @@ -70,12 +86,21 @@ def _placed(text, ws): def align(text, ws): - """Where each timed word is in the text: [[c0, c1, t0, t1]] in order. Words the text no longer has are left out.""" - return [[c0, c1, ws[i][1], ws[i][2]] for i, c0, c1 in _placed(text, ws)] + """Where each timed word is in the text: [[c0, c1, t0, t1]] in order, with how sure the engine was as a fifth when + it said. Words the text no longer has are left out.""" + return [[c0, c1, *ws[i][1:]] for i, c0, c1 in _placed(text, ws)] + + +def unsure(text, ws, below): + """The words of the text the engine was less sure of than `below` (0 to 1; 0 or None: none), with how sure, in + order: [(word, sure)].""" + if not below: + return [] + return [(text[w[0] : w[1]], w[4]) for w in align(text, ws) if len(w) > 4 and w[4] < below] def _dump(ws): - return json.dumps([[w, a, b] for w, a, b in ws]) if ws else None + return json.dumps([list(w) for w in ws]) if ws else None def _join(a, b): diff --git a/fastapi_backend/app/domain/voice.py b/fastapi_backend/app/domain/voice.py index e2c8eacc..27d20780 100644 --- a/fastapi_backend/app/domain/voice.py +++ b/fastapi_backend/app/domain/voice.py @@ -34,7 +34,9 @@ def _talk(cfg): def _key(cfg): - return json.dumps([cfg["transcribe"], cfg.get("speech")], sort_keys=True, default=str) + """What the loaded engine depends on: the transcription settings, except how the transcript marks unsure words.""" + t = {k: v for k, v in cfg["transcribe"].items() if k != "confidence_highlight"} + return json.dumps([t, cfg.get("speech")], sort_keys=True, default=str) def can_transcribe(cfg): diff --git a/fastapi_backend/app/schemas/recordings.py b/fastapi_backend/app/schemas/recordings.py index abf3a059..85275cde 100644 --- a/fastapi_backend/app/schemas/recordings.py +++ b/fastapi_backend/app/schemas/recordings.py @@ -415,10 +415,16 @@ class Player(ResponseModel): segments: list[dict[str, Any]] = Field( default_factory=list, description="the lines: t0, t1 (ms), s (person key), text, e (emotion), v (event), and w, the timed words as " - "[c0, c1, t0, t1] (a character range of text, ms) when transcription gave them; a document's or an image's " + "[c0, c1, t0, t1] (a character range of text, ms) when transcription gave them, with a fifth, how sure the engine " + "was of the word (0 to 1), when it said; a document's or an image's " "blocks of text have p, their page (from 0), and b, where they are on it ([x, y, w, h] as fractions of the " "page), and times that are only a reading pace", ) + unsure_below: float | None = Field( + None, + description="words the engine was less sure of than this (the fifth of a word in w) are marked to check " + "(transcribe.confidence_highlight); null: none are", + ) sections: list[dict[str, Any]] = Field(default_factory=list) entities: list[dict[str, Any]] = Field(default_factory=list) keywords: list[Any] = Field(default_factory=list) diff --git a/fastapi_backend/archive.example.yaml b/fastapi_backend/archive.example.yaml index ab28b476..8eb63d60 100644 --- a/fastapi_backend/archive.example.yaml +++ b/fastapi_backend/archive.example.yaml @@ -27,6 +27,7 @@ transcribe: sensevoice: {model: iic/SenseVoiceSmall, hub: ms, vad_max_segment_ms: 30000, batch_size: 16} whisper: {model: large-v3-turbo, compute_type: default} mlx_whisper: {model: mlx-community/whisper-large-v3-turbo} + confidence_highlight: 0.6 # mark words the engine was less sure of than this; 0 marks none diarize: engine: auto # auto: channels when a file is genuinely two-channel, else cluster diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index 92bbf6b7..c9c274c6 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -10,7 +10,7 @@ from fastapi.testclient import TestClient from app.api import mcp, mcp_tools -from app.domain import ingest, store, topics, untrusted +from app.domain import ingest, settings, store, topics, untrusted from tests import injections from tests.api.test_oauth import O, _bearer, _grant from tests.helpers import login, make_user, seed @@ -316,6 +316,16 @@ def test_reading_recordings(client, env): assert tool_error(client, h, "fetch", id=str(call)) == "not found, or not something you can read" +def test_get_transcript_says_which_words_transcription_was_unsure_of(client, env, db, cfg): + h, (ep1, _, _) = env["h"], env["ids"] + words = [["Welcome", 0, 300, 0.97], ["back.", 300, 600, 0.44], ["Today", 700, 900]] + db.q("UPDATE $s SET words = $w", s=store.R("segment", ep1 * store.SEG), w=json.dumps(words)) + first, second = tool(client, h, "get_transcript", recording_id=ep1, limit=2)["lines"] + assert first["unsure"] == [{"word": "back.", "sure": 0.44}] and "unsure" not in second + settings.save(db, cfg, "transcribe", {"confidence_highlight": 0}, "t") # off: no word is marked + assert "unsure" not in tool(client, h, "get_transcript", recording_id=ep1, limit=1)["lines"][0] + + def test_entities_and_the_graph(client, env, monkeypatch): h, hb, (ep1, ep2, _) = env["h"], env["hb"], env["ids"] found = tool(client, h, "list_entities", query="dyno") diff --git a/fastapi_backend/tests/api/test_split_merge.py b/fastapi_backend/tests/api/test_split_merge.py index 320935de..f991ff37 100644 --- a/fastapi_backend/tests/api/test_split_merge.py +++ b/fastapi_backend/tests/api/test_split_merge.py @@ -7,7 +7,7 @@ import pytest -from app.domain import ingest, search, store, transcript +from app.domain import ingest, search, settings, store, transcript from tests.helpers import drain, login, make_user, seed R = store.R @@ -184,6 +184,50 @@ def test_words_keep_their_timings(client, env, db, cfg): assert m["text"] == "Hello there, this is Lense speaking." and [x[2] for x in m["w"]] == [0, 500, 1600, 1900, 2700] +def test_words_keep_how_sure_transcription_was(client, env, db, cfg): + """A word's fifth in `w` is how sure the engine was of it; the player says below what to mark it, and a split or a + merge keeps it with the word. A correction of the word itself takes the mark away.""" + rid = ingest.import_text(db, cfg, "pods", json.dumps(TIMED), title="Timed") + he = env["he"] + sure = [["Hello", 0, 400, 0.99], ["there,", 500, 900, 0.95], ["this", 1600, 1800], ["is", 1900, 2000, 0.9], + ["Lens", 2100, 2600, 0.42], ["speaking.", 2700, 3600, 0.8]] # fmt: skip + db.q("UPDATE $s SET words = $w", s=R("segment", rid * store.SEG), w=json.dumps(sure)) + player = client.get(f"/api/v1/recordings/{rid}/player", headers=he).json() + assert player["unsure_below"] == 0.6 + w = player["segments"][0]["w"] + assert w[2] == [13, 17, 1600, 1800] and w[4] == [21, 25, 2100, 2600, 0.42] # "this" was kept without one + r = client.post(f"/api/v1/recordings/{rid}/segments/0/split", headers=he, json={"at": player["segments"][0]["text"].index("this")}) + assert r.status_code == 200, r.text + assert _lines(client, rid, he)[1]["w"] == [ + [0, 4, 1600, 1800], + [5, 7, 1900, 2000, 0.9], + [8, 12, 2100, 2600, 0.42], + [13, 22, 2700, 3600, 0.8], + ] + client.post(f"/api/v1/recordings/{rid}/segments/0/merge", headers=he) + line = _lines(client, rid, he)[0] + assert [x[4] if len(x) > 4 else None for x in line["w"]] == [0.99, 0.95, None, 0.9, 0.42, 0.8] + client.patch(f"/api/v1/recordings/{rid}/segments/0", headers=he, json={"text": line["text"].replace("Lens", "Lense")}) + assert all(x[2] != 2100 for x in _lines(client, rid, he)[0]["w"]) # corrected: no longer the word it wasn't sure of + settings.save(db, cfg, "transcribe", {"confidence_highlight": 0}, "t") + assert client.get(f"/api/v1/recordings/{rid}/player", headers=he).json()["unsure_below"] is None + for bad in (1, -0.1, "0.5", True): + with pytest.raises(ValueError): + settings.save(db, cfg, "transcribe", {"confidence_highlight": bad}, "t") + + +def test_words_from_before_how_sure_was_kept_still_load(): + """Lines transcribed before Lens kept how sure the engine was have three-part words; both shapes load, side by side.""" + raw = json.dumps([["Hello", 0, 400], ["world.", 500, 900, 0.31], ["again", 950, 990, "very"], ["x", 1, 2, 1.5]]) + ws = transcript.words({"words": raw}) + assert ws == [("Hello", 0, 400), ("world.", 500, 900, 0.31), ("again", 950, 990), ("x", 1, 2)] + assert transcript.align("Hello world. again x", ws)[:2] == [[0, 5, 0, 400], [6, 12, 500, 900, 0.31]] + assert transcript.unsure("Hello world. again x", ws, 0.6) == [("world.", 0.31)] + assert transcript.unsure("Hello world.", ws, 0) == [] and transcript.unsure("Hello world.", ws, None) == [] + assert transcript.unsure("Hello world.", ws, 0.3) == [] + assert json.loads(transcript._dump(ws)) == [["Hello", 0, 400], ["world.", 500, 900, 0.31], ["again", 950, 990], ["x", 1, 2]] + + def test_alignment_and_joining(): ws = [("Hello", 0, 400), ("world.", 500, 900)] assert transcript.align("Hello world.", ws) == [[0, 5, 0, 400], [6, 12, 500, 900]] diff --git a/fastapi_backend/tests/domain/test_audio_cleanup.py b/fastapi_backend/tests/domain/test_audio_cleanup.py index c9b02148..29ae8eba 100644 --- a/fastapi_backend/tests/domain/test_audio_cleanup.py +++ b/fastapi_backend/tests/domain/test_audio_cleanup.py @@ -201,6 +201,14 @@ def test_spans_joins_and_times(): assert audio_cleanup.speech_spans(np.zeros(100, np.float32), 1000, 1000) == [] +def test_words_keep_how_sure_they_were_when_put_back_on_the_recording(): + x = np.arange(4 * SR, dtype=np.float32) + _, joins = audio_cleanup.cut(x, [(200, 1800), (3100, 3900)]) + segs = [{"t0": 0, "t1": 2400, "text": "a b", "words": json.dumps([["a", 0, 500, 0.42], ["b", 1600, 2400]])}] + out = audio_cleanup.remap(segs, joins) + assert json.loads(out[0]["words"]) == [["a", 200, 700, 0.42], ["b", 3100, 3900]] + + def test_settings_are_checked(db, cfg): settings.save(db, cfg, "transcribe", {"cleanup": {"noise": "high", "echo": "left", "min_silence_ms": 1500}}) eff = settings.effective(db, cfg) diff --git a/fastapi_backend/tests/domain/test_speech.py b/fastapi_backend/tests/domain/test_speech.py index fca8b064..9b9b8b5f 100644 --- a/fastapi_backend/tests/domain/test_speech.py +++ b/fastapi_backend/tests/domain/test_speech.py @@ -51,8 +51,8 @@ def do_GET(self): "status": "completed", "language_code": "en_us", "words": [ - {"text": "Hello", "start": 0, "end": 400, "speaker": "A"}, - {"text": "there.", "start": 400, "end": 800, "speaker": "A"}, + {"text": "Hello", "start": 0, "end": 400, "speaker": "A", "confidence": 0.99}, + {"text": "there.", "start": 400, "end": 800, "speaker": "A", "confidence": 0.52}, {"text": "Terrible", "start": 900, "end": 1300, "speaker": "B"}, {"text": "news.", "start": 1300, "end": 1700, "speaker": "B"}, ], @@ -116,7 +116,7 @@ def do_POST(self): "language_code": "eng", "text": "Hello. (laughter) Hi.", "words": [ - {"text": "Hello.", "start": 0.0, "end": 0.5, "type": "word", "speaker_id": "speaker_0"}, + {"text": "Hello.", "start": 0.0, "end": 0.5, "type": "word", "speaker_id": "speaker_0", "logprob": -0.5}, {"text": " ", "start": 0.5, "end": 0.6, "type": "spacing", "speaker_id": "speaker_0"}, {"text": "(laughter)", "start": 0.6, "end": 0.9, "type": "audio_event", "speaker_id": "speaker_0"}, {"text": "Hi.", "start": 1.0, "end": 1.4, "type": "word", "speaker_id": "speaker_1"}, @@ -129,9 +129,9 @@ def do_POST(self): return self._send({"id": "t1", "status": "queued"}) if p.startswith("/dg/v1/listen"): w = [ - {"word": "great", "punctuated_word": "Great", "start": 0.0, "end": 0.4, "speaker": 0}, - {"word": "work", "punctuated_word": "work.", "start": 0.4, "end": 0.8, "speaker": 0}, - {"word": "thanks", "punctuated_word": "Thanks.", "start": 1.0, "end": 1.5, "speaker": 1}, + {"word": "great", "punctuated_word": "Great", "start": 0.0, "end": 0.4, "speaker": 0, "confidence": 0.98}, + {"word": "work", "punctuated_word": "work.", "start": 0.4, "end": 0.8, "speaker": 0, "confidence": 0.41}, + {"word": "thanks", "punctuated_word": "Thanks.", "start": 1.0, "end": 1.5, "speaker": 1, "confidence": 0.9}, ] return self._send( { @@ -278,6 +278,54 @@ def test_deepgram_transcribes_with_speakers_and_sentiment(db, cfg, fake): assert headers["Authorization"] == "Token dg-key" and headers["Content-Type"] == "audio/ogg" +def test_providers_keep_how_sure_they_were_of_each_word(db, cfg, fake, monkeypatch): + """Deepgram's and AssemblyAI's confidence and ElevenLabs' log probability stay with each word's timing, as a fourth; + a provider that doesn't say (an OpenAI-compatible server) keeps three, as before.""" + monkeypatch.setattr(speech, "POLL_SECONDS", 0) + eff = provided(db, cfg, fake) + + def stored(p): + return [json.loads(s["words"]) for s in speech.Engine(eff, p).transcribe(clip()) if s.get("words")] + + assert stored("deepgram") == [[["Great", 0, 400, 0.98], ["work.", 400, 800, 0.41]], [["Thanks.", 1000, 1500, 0.9]]] + assert stored("assemblyai")[0] == [["Hello", 0, 400, 0.99], ["there.", 400, 800, 0.52]] + assert stored("elevenlabs")[0] == [["Hello.", 0, 500, 0.607]] # e^-0.5 + assert stored("openai") == [[["Book", 0, 300], ["Friday.", 1200, 1900]]] + # a long recording's pieces move onto its timeline with how sure each word was + assert speech._shift([{"t0": 0, "t1": 9, "words": json.dumps([["a", 0, 4, 0.5], ["b", 5, 9]])}], 1000, None)[0]["words"] == json.dumps( + [["a", 1000, 1004, 0.5], ["b", 1005, 1009]] + ) + + +def test_whisper_keeps_how_sure_it_was_of_each_word(): + """faster-whisper's word probability, and mlx-whisper's, are kept with each word's timing.""" + + class W: + def __init__(self, word, start, end, probability): + self.word, self.start, self.end, self.probability = word, start, end, probability + + class S: + start, end, text = 0.0, 1.0, " Book a call." + words = [W(" Book", 0.0, 0.3, 0.97), W(" a", 0.3, 0.4, 0.88), W(" call.", 0.5, 1.0, 0.345)] + + class Faster: + def transcribe(self, audio, **kw): + assert kw["word_timestamps"] is True + return [S()], None + + class Mlx: + def transcribe(self, audio, **kw): + words = [{"word": " Book", "start": 0.0, "end": 0.3, "probability": 0.5}, {"word": " now.", "start": 0.4, "end": 0.9}] + return {"segments": [{"start": 0.0, "end": 1.0, "text": " Book now.", "words": words}]} + + w = object.__new__(ingest.Whisper) + w.mlx, w.m, w.lang = False, Faster(), None + (seg,) = w.transcribe(clip()) + assert json.loads(seg["words"]) == [["Book", 0, 300, 0.97], ["a", 300, 400, 0.88], ["call.", 500, 1000, 0.345]] + w.mlx, w.m, w.repo = True, Mlx(), "mlx-community/whisper" + assert json.loads(w.transcribe(clip())[0]["words"]) == [["Book", 0, 300, 0.5], ["now.", 400, 900]] + + def test_speaker_separation_uses_the_providers_speakers(db, cfg, folder, fake): eff = provided(db, cfg, fake) eff["speakers"]["embedder"] = "none" @@ -348,7 +396,7 @@ def test_a_long_recording_goes_in_pieces_and_comes_back_whole(db, cfg, fake, mon assert [s["provider_speaker"] for s in segs][:2] == ["c0:S0", "c0:S1"] assert {s["provider_speaker"] for s in segs} == {f"c{k}:S{n}" for k in range(3) for n in range(2)} second = next(s for s in segs if s["provider_speaker"] == "c1:S1") - assert json.loads(second["words"]) == [["Thanks.", starts[1] + 1000, starts[1] + 1500]] + assert json.loads(second["words"]) == [["Thanks.", starts[1] + 1000, starts[1] + 1500, 0.9]] # with how sure, on the timeline assert [s["t0"] for s in segs] == sorted(s["t0"] for s in segs) assert rows == [("speech.transcribe", "deepgram:nova-3", {"seconds": 150.0, "pieces": 3})] diff --git a/nextjs-frontend/__tests__/recording-split-words.test.ts b/nextjs-frontend/__tests__/recording-split-words.test.ts index 8fd5d607..b139a2ed 100644 --- a/nextjs-frontend/__tests__/recording-split-words.test.ts +++ b/nextjs-frontend/__tests__/recording-split-words.test.ts @@ -1,6 +1,16 @@ import type { Player } from "@/app/openapi-client/types.gen"; import { describeEdit, revertPatch } from "@/components/recording/edits"; -import { joinedAt, normalizePlayer, splitPoint, textRange, wordAt, type Word } from "@/components/recording/model"; +import { + joinedAt, + normalizePlayer, + splitPoint, + textRange, + unsureRanges, + unsureThreshold, + unsureTitle, + wordAt, + type Word, +} from "@/components/recording/model"; const name = (id: number | null) => (id == null ? "nobody" : `Speaker ${id}`); @@ -26,6 +36,60 @@ describe("timed words", () => { expect("words" in m.segments[1]).toBe(false); }); + it("keeps how sure transcription was of each word, and words from before it was kept", () => { + const m = normalizePlayer({ + id: 1, + unsure_below: 0.6, + segments: [ + { + t0: 0, + t1: 3000, + s: "s1", + text: "Hello there, Lens.", + w: [ + [0, 5, 0, 400, 0.97], + [6, 12, 500, 900], + [13, 18, 1000, 2000, 7], + [13, 18, 1000, 2000, 0.5, 1], + ], + }, + ], + } as unknown as Player); + expect(m.unsureBelow).toBe(0.6); + expect(m.segments[0].words).toEqual([ + [0, 5, 0, 400, 0.97], + [6, 12, 500, 900], + [13, 18, 1000, 2000], // not 0..1: the word stays, without it + ]); + expect(normalizePlayer({ id: 1 } as unknown as Player).unsureBelow).toBeNull(); + }); + + it("marks the words below the threshold, and none when it's off", () => { + const seg = { + idx: 0, + t0: 0, + t1: 3000, + speaker: null, + text: "Book a call Friday.", + emotion: null, + event: null, + words: [ + [0, 4, 0, 300, 0.95], + [5, 6, 300, 400], + [7, 11, 500, 900, 0.62], + [12, 19, 1000, 1900, 0.123], + ] as Word[], + }; + expect(unsureRanges(seg, 0.6)).toEqual([{ start: 12, end: 19, kind: "unsure:12" }]); + expect(unsureRanges(seg, 0.7).map((r) => seg.text.slice(r.start, r.end))).toEqual(["call", "Friday."]); + expect(unsureRanges(seg, null)).toEqual([]); + expect(unsureRanges({ ...seg, words: undefined }, 0.6)).toEqual([]); + expect(unsureTitle("unsure:62")).toBe("Transcription wasn't sure of this word (62%)"); + expect(unsureTitle("entity")).toBeNull(); + expect(unsureTitle(null)).toBeNull(); + expect([0.6, 0, 1, -0.2, "0.6", null].map(unsureThreshold)).toEqual([0.6, null, null, null, null, null]); + }); + it("finds the word being said", () => { const words: Word[] = [ [0, 5, 0, 400], diff --git a/nextjs-frontend/__tests__/recording-unsure-words.test.tsx b/nextjs-frontend/__tests__/recording-unsure-words.test.tsx new file mode 100644 index 00000000..e346221d --- /dev/null +++ b/nextjs-frontend/__tests__/recording-unsure-words.test.tsx @@ -0,0 +1,60 @@ +import { fireEvent, render, screen } from "@testing-library/react"; + +import type { Segment, Turn, Word } from "@/components/recording/model"; +import { TurnView } from "@/components/recording/turn"; + +jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) })); + +const seg: Segment = { + idx: 0, + t0: 1000, + t1: 4000, + speaker: null, + text: "Book a call Friday.", + emotion: null, + event: null, + words: [ + [0, 4, 1000, 1300, 0.95], + [5, 6, 1300, 1400], + [12, 19, 2000, 2900, 0.62], + ] as Word[], +}; +const turn = { key: 0, speaker: null, t0: 1000, t1: 4000, segs: [0] } as unknown as Turn; + +function view(unsureBelow: number | null, onSeek = jest.fn()) { + return render( + , + ); +} + +describe("words transcription wasn't sure of", () => { + it("are underlined with how sure, and a click still plays the line", () => { + const onSeek = jest.fn(); + view(0.7, onSeek); + const word = screen.getByTitle("Transcription wasn't sure of this word (62%)"); + expect(word.textContent).toBe("Friday."); + expect(word.className).toContain("decoration-dotted"); + fireEvent.click(word); + expect(onSeek).toHaveBeenCalledWith(1000); + }); + + it("are not marked when it's off", () => { + view(null); + expect(screen.queryByTitle(/wasn't sure/)).toBeNull(); + expect(screen.getByText("Book a call Friday.")).toBeTruthy(); + }); +}); diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index d651b1fc..52e09d2c 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -18525,11 +18525,17 @@ export type Player = { /** * Segments * - * the lines: t0, t1 (ms), s (person key), text, e (emotion), v (event), and w, the timed words as [c0, c1, t0, t1] (a character range of text, ms) when transcription gave them; a document's or an image's blocks of text have p, their page (from 0), and b, where they are on it ([x, y, w, h] as fractions of the page), and times that are only a reading pace + * the lines: t0, t1 (ms), s (person key), text, e (emotion), v (event), and w, the timed words as [c0, c1, t0, t1] (a character range of text, ms) when transcription gave them, with a fifth, how sure the engine was of the word (0 to 1), when it said; a document's or an image's blocks of text have p, their page (from 0), and b, where they are on it ([x, y, w, h] as fractions of the page), and times that are only a reading pace */ segments?: Array<{ [key: string]: unknown; }>; + /** + * Unsure Below + * + * words the engine was less sure of than this (the fifth of a word in w) are marked to check (transcribe.confidence_highlight); null: none are + */ + unsure_below?: number | null; /** * Sections */ diff --git a/nextjs-frontend/components/recording/model.ts b/nextjs-frontend/components/recording/model.ts index 85896be6..e6b61d5f 100644 --- a/nextjs-frontend/components/recording/model.ts +++ b/nextjs-frontend/components/recording/model.ts @@ -29,14 +29,17 @@ export type Segment = { text: string; emotion: string | null; event: string | null; - /** Timed words, when transcription gave them: [c0, c1, t0, t1], a character range of `text` and when it was said. */ + /** + * Timed words, when transcription gave them: [c0, c1, t0, t1], a character range of `text` and when it was said, with + * how sure the engine was of the word (0..1) as a fifth when it said. + */ words?: Word[]; /** A document's or an image's block of text: its page (from 0) and where it is on it. */ page?: number; box?: Box | null; }; -export type Word = [c0: number, c1: number, t0: number, t1: number]; +export type Word = [c0: number, c1: number, t0: number, t1: number, sure?: number]; export type Chapter = { idx: number; @@ -167,6 +170,8 @@ export type PlayerModel = { objects: ObjectTrack[]; descriptions: Description[]; poster: string | null; + /** Words transcription was less sure of than this (0..1) are marked to check; null: none are (Settings → Transcription). */ + unsureBelow: number | null; }; const num = (v: unknown, d = 0): number => @@ -383,6 +388,7 @@ export function normalizePlayer(raw: Player): PlayerModel { }) .filter((d) => d.text), poster: str(r.poster), + unsureBelow: unsureThreshold(r.unsure_below), }; } @@ -438,20 +444,55 @@ export function indexAt(items: T[], t: number): number } /** The segment being spoken at `t`, or the last one before it (so a pause keeps the line lit). */ -/** A line's timed words from the API, kept only when they fit its text: `{ words }`, or nothing. */ +/** + * A line's timed words from the API, kept only when they fit its text: `{ words }`, or nothing. A word has how sure + * transcription was of it as a fifth when the engine said (lines transcribed before then have four); one that isn't + * 0..1 is dropped, keeping the word. + */ function wordsOf(raw: unknown, len: number): { words?: Word[] } { - const words = arr(raw).filter( - (w): w is Word => - Array.isArray(w) && - w.length === 4 && - w.every((x) => typeof x === "number" && Number.isFinite(x)) && - w[0] >= 0 && - w[0] < w[1] && - w[1] <= len, - ); + const words = arr(raw) + .filter( + (w): w is number[] => + Array.isArray(w) && + (w.length === 4 || w.length === 5) && + w.slice(0, 4).every((x) => typeof x === "number" && Number.isFinite(x)) && + w[0] >= 0 && + w[0] < w[1] && + w[1] <= len, + ) + .map((w): Word => { + const sure = w[4]; + return typeof sure === "number" && sure >= 0 && sure <= 1 + ? [w[0], w[1], w[2], w[3], sure] + : [w[0], w[1], w[2], w[3]]; + }); return words.length ? { words } : {}; } +/** The threshold for marking words from the player's `unsure_below`: a number above 0 and below 1, else null (off). */ +export function unsureThreshold(v: unknown): number | null { + return typeof v === "number" && v > 0 && v < 1 ? v : null; +} + +const UNSURE = "unsure:"; + +/** + * The words of a line transcription was less sure of than `below`, as ranges of its text for the dotted underline: + * their kind carries how sure, as a whole percentage (`unsure:62`). None when it's off or the line's words don't say. + */ +export function unsureRanges(seg: Segment, below: number | null): { start: number; end: number; kind: string }[] { + if (below == null || !seg.words) return []; + return seg.words + .filter((w) => w[4] != null && w[4] < below) + .map((w) => ({ start: w[0], end: w[1], kind: `${UNSURE}${Math.round((w[4] ?? 0) * 100)}` })); +} + +/** The tooltip of a run marked by `unsureRanges`, or null for any other kind of run. */ +export function unsureTitle(kind: string | null): string | null { + if (!kind?.startsWith(UNSURE)) return null; + return `Transcription wasn't sure of this word (${kind.slice(UNSURE.length)}%)`; +} + /** The word being said at `ms`: the last one to have started (-1 before the first). */ export function wordAt(words: Word[], ms: number): number { let lo = 0; diff --git a/nextjs-frontend/components/recording/transcript.tsx b/nextjs-frontend/components/recording/transcript.tsx index 0f1e0fab..e8b154d6 100644 --- a/nextjs-frontend/components/recording/transcript.tsx +++ b/nextjs-frontend/components/recording/transcript.tsx @@ -298,6 +298,7 @@ function TranscriptInner({ compact, slim, className }: { compact?: boolean; slim } entityNames={entityNames} unsure={t.speaker ? (unsure.get(t.speaker) ?? null) : null} + unsureBelow={model.unsureBelow} onScreen={Boolean(t.speaker && onScreen.has(t.speaker))} notes={noteCounts.get(t.key)} onNotes={openNotes} diff --git a/nextjs-frontend/components/recording/turn.tsx b/nextjs-frontend/components/recording/turn.tsx index b1dbcf8b..d0d706cf 100644 --- a/nextjs-frontend/components/recording/turn.tsx +++ b/nextjs-frontend/components/recording/turn.tsx @@ -11,6 +11,8 @@ import { entityRanges, showEmotion, splitRuns, + unsureRanges, + unsureTitle, type Segment, type SpeakerInfo, type Turn, @@ -24,7 +26,8 @@ export type Unsure = { score: number; maybe: string }; /** * One turn: timestamp (seeks), the speaker's colour bar and name, and its segments as one serif paragraph. The line - * being played gets a soft tint; entity names are dotted-underlined; find matches are highlighted. + * being played gets a soft tint; entity names are dotted-underlined, and words transcription wasn't sure of too, in gold + * with how sure in a tooltip; find matches are highlighted. */ export const TurnView = memo(function TurnView({ turn, @@ -35,6 +38,7 @@ export const TurnView = memo(function TurnView({ currentHit, entityNames, unsure, + unsureBelow = null, onScreen, notes, onNotes, @@ -58,6 +62,8 @@ export const TurnView = memo(function TurnView({ currentHit: { seg: number; start: number } | null; entityNames: Map; unsure: Unsure | null; + /** Words transcription was less sure of than this are marked (null: none are). */ + unsureBelow?: number | null; /** Video: this voice's face is on screen in the recording. */ onScreen?: boolean; /** How many of the notes you see are about this turn's moments; the mark opens the Notes tab. */ @@ -180,6 +186,7 @@ export const TurnView = memo(function TurnView({ hits={hits.get(i)} currentStart={currentHit?.seg === i ? currentHit.start : null} names={entityNames.get(i)} + unsureBelow={unsureBelow} hls={highlights?.get(i)} onHighlight={onHighlight} onClick={click(s)} @@ -225,6 +232,7 @@ function SegmentText({ hits, currentStart, names, + unsureBelow, hls, onHighlight, onClick, @@ -235,6 +243,7 @@ function SegmentText({ hits?: { start: number; end: number }[]; currentStart: number | null; names?: string[]; + unsureBelow: number | null; hls?: HighlightRange[]; onHighlight?: (id: number) => void; onClick: (e: MouseEvent) => void; @@ -246,6 +255,8 @@ function SegmentText({ kind: h.start === currentStart ? "hit-current" : "hit", })), ...(names?.length ? entityRanges(seg.text, names) : []), + // last, so a find match or an entity name over the same word wins + ...unsureRanges(seg, unsureBelow), ]; const runs = hls?.length ? highlightRuns(seg.text, ranges, hls) @@ -278,6 +289,7 @@ function SegmentText({ )} > {runs.map((r, i) => { + const doubt = unsureTitle(r.kind); const run = r.kind === "entity" ? ( {r.text} @@ -293,6 +305,13 @@ function SegmentText({ > {r.text} + ) : doubt ? ( + + {r.text} + ) : ( {r.text} ); diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index bc518cb6..f8d9efa8 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -194,6 +194,7 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) {
({ value, label }))} /> +

@@ -218,6 +219,7 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) { ({ value, label }))} /> +

diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 1fea0b90..7b9c26f2 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -754,6 +754,15 @@ export const FIELDS: FieldSpec[] = [ kind: "switch", hint: "Each recording is transcribed knowing its namespace's named people, defined entities, custom vocabulary and the names you fixed in its transcripts, so the same mistakes stop coming back (Whisper, OpenAI-compatible, Deepgram and AssemblyAI; SenseVoice and ElevenLabs take no hints)", }, + { + section: "transcribe", + key: "confidence_highlight", + label: "Mark words it was less sure of than", + kind: "number", + min: 0, + max: 0.99, + hint: "From 0 to 0.99; 0 marks none. Each gets a dotted gold underline in the transcript, to check and correct", + }, // Speaker separation { section: "diarize", diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 0b3e7e74..5cdf772d 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -69057,7 +69057,19 @@ }, "type": "array", "title": "Segments", - "description": "the lines: t0, t1 (ms), s (person key), text, e (emotion), v (event), and w, the timed words as [c0, c1, t0, t1] (a character range of text, ms) when transcription gave them; a document's or an image's blocks of text have p, their page (from 0), and b, where they are on it ([x, y, w, h] as fractions of the page), and times that are only a reading pace" + "description": "the lines: t0, t1 (ms), s (person key), text, e (emotion), v (event), and w, the timed words as [c0, c1, t0, t1] (a character range of text, ms) when transcription gave them, with a fifth, how sure the engine was of the word (0 to 1), when it said; a document's or an image's blocks of text have p, their page (from 0), and b, where they are on it ([x, y, w, h] as fractions of the page), and times that are only a reading pace" + }, + "unsure_below": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Unsure Below", + "description": "words the engine was less sure of than this (the fifth of a word in w) are marked to check (transcribe.confidence_highlight); null: none are" }, "sections": { "items": {