Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,15 +181,15 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
(noise, volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before
analysis (stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server
you run), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each
off by default.
off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[cleaning up audio](docs/processing.md#cleaning-up-audio),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates),
[unsure words](docs/processing.md#unsure-words); [checks](docs/requirements.md#understand-it))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
source, read aloud in two voices, by Kokoro on this machine or a speech provider. Turn a voice session or a meeting into a technical spec, in a note linked to it,
that keeps every decision, every no and every constraint in the speaker's own words, linked to when they said them,
Expand Down
9 changes: 9 additions & 0 deletions changelog.d/word-confidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
- **See the words transcription wasn't sure of.** Whisper, Deepgram, AssemblyAI and ElevenLabs say how sure they
were of each word, and Lens now keeps it with the word's timing instead of dropping it. In the transcript, a word
below the threshold gets a dotted gold underline and a tooltip ("Transcription wasn't sure of this word (42%)"), so
you can spot it and correct it in place; correcting it takes the mark away. Settings → Transcription → Mark words
it was less sure of than sets the threshold (`transcribe.confidence_highlight`, 0.6; 0 marks none). `GET
…/player` gives each word's confidence as a fifth in `w`, and the threshold as `unsure_below`; the MCP tool
`get_transcript` lists each line's `unsure` words. Transcripts made before, and those from SenseVoice and OpenAI's
own API, which don't say, have no marks and work as before
([Words transcription wasn't sure of](docs/processing.md#unsure-words)).
6 changes: 4 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -716,8 +716,10 @@ An asset's `summary` (the Summarize step's, also in `GET …/player`) has `summa
summaries made before they had times hold plain strings.

In `GET …/player`, a line whose words have timings from transcription carries them as `w`: `[c0, c1, t0, t1]` for
each word, a character range of its `text` and when it was said (ms). A corrected line keeps the timings of the words
it still has.
each word, a character range of its `text` and when it was said (ms), with a fifth, how sure the engine was of the word
(0 to 1), when it said. A corrected line keeps the timings of the words it still has. `unsure_below` is the threshold
the web app marks words under (`transcribe.confidence_highlight`), or null when marking is off
([Words transcription wasn't sure of](processing.md#unsure-words)).

Assets carry `tags`. `PATCH /resources/{rid}` with `{"tags": [...]}` replaces an asset's tags (editors; at most
20, 40 characters each; whitespace is collapsed and repeats are dropped, ignoring case). `POST /resources/tags` with
Expand Down
2 changes: 1 addition & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_speakers` | the people of your namespaces, the ones with the most talk time first |
| `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW |
| `get_transcript` | an asset's lines in order (who, time, text, link), a page at a time, from a line or a time window |
| `get_transcript` | an asset's lines in order (who, time, text, link, and the words transcription wasn't sure of as `unsure`), a page at a time, from a line or a time window |
| `fetch` | an asset's whole text at once, as `[m:ss] Speaker: …` lines |
| `cite` | a citation for a moment (a line or a second, and a few lines from there): the words, who said or wrote them, the asset and date, the time, the link, and all of it as Markdown |
| `list_entities` | people, organisations, products, places, events, works and terms mentioned, by name, type, namespace or asset |
Expand Down
17 changes: 17 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,23 @@ custom pipeline by adding the Clean up step before Analyze (with Force it runs e
on an asset with Reprocess → Clean up, `POST /api/v1/recordings/{id}/reprocess` with `{"steps": ["clean", "analyze"]}`,
or `lens clean`. Workers whose `workers.steps` list has `analyze` run it too.

### Words transcription wasn't sure of {#unsure-words}

Whisper (faster-whisper and mlx-whisper), Deepgram, AssemblyAI and ElevenLabs say how sure they were of each word, and
Lens keeps that with the word's timing. In the transcript, a word it was less sure of than 60% gets a dotted gold
underline, and its tooltip says how sure ("Transcription wasn't sure of this word (42%)"), so you can find the words
worth checking and correct them in Edit. Clicking it plays the line, as it does anywhere in the text. A corrected word
loses its mark, and splitting or merging lines keeps the marks on the words they still have.

Settings → Transcription → **Mark words it was less sure of than** sets the threshold (`transcribe.confidence_highlight`,
0.6 by default, from 0 to 0.99); 0 marks none. SenseVoice, the default engine, and OpenAI's own API don't say how sure
they were, so their transcripts have no marks (an OpenAI-compatible server that gives each word a `probability` is
kept too), and nor do transcripts made before Lens kept it.

Agents get the same: `GET /api/v1/recordings/{id}/player` gives each word's confidence and the threshold ([API](api.md)),
and the MCP tool `get_transcript` lists each line's `unsure` words with how sure, worth checking before quoting
([MCP](mcp.md)).

## Importing text and transcripts {#importing-transcripts}

Formats: .txt, .md, .markdown, .mdx, .docx, .doc, .pdf, .srt, .vtt, .json (lens, Whisper or a list of segments) and
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Transcription that expects the names the namespace knows (custom vocabulary, named people, defined entities, names people fixed), off by default | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_the_namespaces_own_names_go_to_the_engine` |
| Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| One asset's voice split off from the person it was matched to, with Undo | [test_speakers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_speakers.py) `test_detach_one_assets_voice` |
| How sure transcription was of each word kept (Whisper, Deepgram, AssemblyAI, ElevenLabs), words from before still loading, the unsure ones marked in the transcript with how sure, a threshold in Settings (0 for none), and `unsure` words per line over MCP | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_providers_keep_how_sure_they_were_of_each_word`, `test_whisper_keeps_how_sure_it_was_of_each_word`; [test_split_merge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_split_merge.py) `test_words_keep_how_sure_transcription_was`, `test_words_from_before_how_sure_was_kept_still_load`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_get_transcript_says_which_words_transcription_was_unsure_of`; [recording-split-words.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-split-words.test.ts) `keeps how sure transcription was of each word, and words from before it was kept`, `marks the words below the threshold, and none when it's off`; [recording-unsure-words.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-unsure-words.test.tsx) `are underlined with how sure, and a click still plays the line` |
| People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` |
| An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` |
| Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
Expand Down
21 changes: 16 additions & 5 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
settings,
sources,
store,
transcript,
untrusted,
updates,
variables,
Expand Down Expand Up @@ -437,8 +438,9 @@ def _line(ctx: Context, rid: int, s: dict[str, Any], names: dict[int, str]) -> d
return out


def _segments(db: DB, rid: int, where: str = "", limit: int | None = None, **params: Any) -> list[dict[str, Any]]:
sql = f"SELECT idx, t0, t1, speaker, text, page FROM segment WHERE recording = $r{where} ORDER BY idx"
def _segments(db: DB, rid: int, where: str = "", limit: int | None = None, with_words: bool = False, **params: Any) -> list[dict[str, Any]]:
cols = "idx, t0, t1, speaker, text, page" + (", words" if with_words else "")
sql = f"SELECT {cols} FROM segment WHERE recording = $r{where} ORDER BY idx"
if limit is not None:
sql += f" LIMIT {int(limit)}"
return db.rows(sql, r=rid, **params)
Expand Down Expand Up @@ -970,7 +972,8 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]:
"Read an asset's text",
"The lines of an asset's text (the transcript of audio or video, or a document's or image's text), in order: who "
"said or wrote what, and when, each with "
"a url that opens the asset at that line. Read a long one in pages: from_line (next_line of the last page) or "
"a url that opens the asset at that line, and `unsure`: the words transcription wasn't sure of, each with how sure "
"(0 to 1), worth checking before quoting them. Read a long one in pages: from_line (next_line of the last page) or "
"a time window in seconds.",
Arg("recording_id", "integer", "the asset", required=True),
Arg("from_line", "integer", "start at this line (0 is the first)", minimum=0),
Expand All @@ -992,15 +995,23 @@ def get_transcript(
if end_seconds is not None:
where += " AND t0 <= $b"
params["b"] = int(end_seconds * 1000)
segs = _segments(ctx.db, recording_id, where, limit + 1, **params)
below = ctx.cfg["transcribe"].get("confidence_highlight")
segs = _segments(ctx.db, recording_id, where, limit + 1, with_words=bool(below), **params)
more, segs = len(segs) > limit, segs[:limit]
names = _names(ctx.db, [s.get("speaker") for s in segs])
total = ctx.db.values("RETURN array::len((SELECT VALUE id FROM segment WHERE recording = $r))", r=recording_id)
lines = []
for s in segs:
line = _line(ctx, recording_id, s, names)
unsure = transcript.unsure(s.get("text") or "", transcript.words(s), below)
if unsure:
line["unsure"] = [{"word": w, "sure": p} for w, p in unsure]
lines.append(line)
out: dict[str, Any] = {
"recording_id": recording_id,
"title": rec.get("title"),
"lines_in_all": int(total[0]) if total else 0,
"lines": [_line(ctx, recording_id, s, names) for s in segs],
"lines": lines,
}
if more:
out["next_line"] = segs[-1]["idx"] + 1
Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/api/v1/routes/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,7 @@ def get_player(rid: int, acl: Acl, db: Db, cfg: Cfg, s: str = "") -> Player:
rec = acl.recording(rid, share=s)
data = render.player_data(db, rid, audio_link(db, cfg, rid, s))
data["nsfw"] = _nsfw(db, cfg, rec, rid)
data["unsure_below"] = cfg["transcribe"].get("confidence_highlight") or None
return sign_urls(data, full=acl.member(rec))


Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/audio_cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ def remap(segs, joins):
s["t0"], s["t1"] = to_recording(s["t0"], joins), to_recording(s["t1"], joins, end=True)
if s.get("words"):
words = json.loads(s["words"])
s["words"] = json.dumps([[w, to_recording(a, joins), to_recording(b, joins, end=True)] for w, a, b in words])
s["words"] = json.dumps([[w[0], to_recording(w[1], joins), to_recording(w[2], joins, end=True), *w[3:]] for w in words])
return segs


Expand Down
16 changes: 11 additions & 5 deletions fastapi_backend/app/domain/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

import numpy as np

from . import speech, store, vocabulary
from . import speech, store, transcript, vocabulary

TAG_RX = re.compile(r"<\|([^|]*)\|>")
SR = 16000
Expand Down Expand Up @@ -396,7 +396,8 @@ def transcribe(self, audio, terms=()): # SenseVoice takes no hints (vocabulary.


class Whisper:
"""faster-whisper (CUDA or CPU) or mlx-whisper (Apple Silicon), with word timestamps."""
"""faster-whisper (CUDA or CPU) or mlx-whisper (Apple Silicon), with word timestamps and how sure it was of each
word."""

def __init__(self, cfg, mlx=False):
t = cfg["transcribe"]
Expand Down Expand Up @@ -425,11 +426,16 @@ def transcribe(self, audio, terms=()):
if self.mlx:
got = self.m.transcribe(audio, path_or_hf_repo=self.repo, word_timestamps=True, language=self.lang, **hint)
heard, segs = got.get("language"), got["segments"]
rows = [(s["start"], s["end"], s["text"], [(w["word"], w["start"], w["end"]) for w in s.get("words") or []]) for s in segs]
rows = [
(s["start"], s["end"], s["text"], [(w["word"], w["start"], w["end"], w.get("probability")) for w in s.get("words") or []])
for s in segs
]
else:
segs, info = self.m.transcribe(audio, vad_filter=True, word_timestamps=True, language=self.lang, **hint)
heard = getattr(info, "language", None)
rows = [(s.start, s.end, s.text, [(w.word, w.start, w.end) for w in (s.words or [])]) for s in segs]
rows = [
(s.start, s.end, s.text, [(w.word, w.start, w.end, getattr(w, "probability", None)) for w in (s.words or [])]) for s in segs
]
lang = self.lang or (str(heard).lower() if heard else None)
return [
{
Expand All @@ -440,7 +446,7 @@ def transcribe(self, audio, terms=()):
"emotion": None,
"event": None,
"lang": lang,
"words": json.dumps([[w.strip(), int(p * 1000), int(q * 1000)] for w, p, q in ws]),
"words": json.dumps([transcript.stored(w.strip(), int(p * 1000), int(q * 1000), sure) for w, p, q, sure in ws]),
}
for a, b, x, ws in rows
if x.strip()
Expand Down
4 changes: 4 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,10 @@ def _check(section, key, value, default):
if not (isinstance(value, int) and not isinstance(value, bool) and 1 <= value <= 500):
raise ValueError("ai.max_tool_calls is a whole number of tool calls from 1 to 500")
return value
if (section, key) == ("transcribe", "confidence_highlight"):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not 0 <= value < 1:
raise ValueError("transcribe.confidence_highlight is a number from 0 (mark no words) to below 1")
return float(value)
if (section, key) == ("llm", "describe_max"):
if not (isinstance(value, int) and not isinstance(value, bool) and 1 <= value <= 1000):
raise ValueError("llm.describe_max is a whole number from 1 to 1000")
Expand Down
Loading
Loading