Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
assistant, which can also use that browser on the sites you allow (open, read, click, type), asking before anything
that sends, deletes, buys or posts. ([Web clipper](docs/web-clipper.md),
[the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser))
- **Understand it.** Transcription, people recognised by voice (long recordings go to a speech provider in pieces that
fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
- **Understand it.** Transcription, which can expect the names your namespace knows (off by default), people
recognised by voice (long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
server (off by default, per namespace), entities and topics in a knowledge graph you can explore, query with Cypher or
SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
Expand All @@ -137,6 +137,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[long recordings](docs/speech-providers.md#long-recordings),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/vocabulary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Transcription expects your own names.** With Settings → Transcription → Expect your own names on (off by
default), each recording is transcribed knowing up to 60 words its namespace already knows: the custom vocabulary,
its named people, its defined entities and the names people fixed in its transcripts. Whisper and OpenAI-compatible
servers get them as the prompt, Deepgram as key terms or keywords, AssemblyAI as key terms or boosted words; SenseVoice
and ElevenLabs take no hints ([Your own names in transcription](docs/processing.md#your-own-names-in-transcription)).
18 changes: 18 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,24 @@ How assets move through Lens: where they come from, what each step does, and how

Every step takes `--ns`, `--limit` and `--force`; `run` does them all, and a lock stops two runs overlapping.

### Your own names in transcription

Speech-to-text gets wrong the names only you use: family, colleagues, places, projects. With `transcribe.vocabulary`
on (Settings → Transcription → Expect your own names; off by default), each recording is transcribed knowing up to 60
words its namespace already knows, in this order:

- the custom vocabulary in Settings → Analysis (`analysis.gazetteer`), as written there;
- the names of the namespace's named people;
- its defined entities ([Entity setup](#entity-setup));
- the words people put in when they corrected its transcripts, the capitalised ones a line didn't have before (the
text cleanup's own changes don't count), most often first.

Whisper (faster-whisper and mlx-whisper) and OpenAI-compatible servers get them as the prompt, a line of the names (the
diarizing models take none); Deepgram as key terms with Nova-3, or keywords with the models before it; AssemblyAI as
key terms with Universal-3 and Slam-1, or boosted words with the others. SenseVoice and ElevenLabs take no hints, so
they transcribe as before. The words go only to the engine the recording was going to anyway, and a provider's row in
the [activity ledger](activity.md) says how many it was given. Run Transcribe again on an asset to use new names.

### Cleaning up transcripts

Speech comes out of transcription as it was said, and the extractor, chapters, keywords and search all read it. With
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Requirement | Checks |
|---|---|
| Transcription | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_an_openai_compatible_server_transcribes`, `test_deepgram_transcribes_with_speakers_and_sentiment` |
| Transcription that expects the names the namespace knows (custom vocabulary, named people, defined entities, names people fixed), off by default | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_the_namespaces_own_names_go_to_the_engine` |
| Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` |
| Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
Expand Down
15 changes: 9 additions & 6 deletions fastapi_backend/app/domain/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

import numpy as np

from . import speech, store
from . import speech, store, vocabulary

TAG_RX = re.compile(r"<\|([^|]*)\|>")
SR = 16000
Expand Down Expand Up @@ -300,7 +300,7 @@ def _run(self, clips):
except Exception: # noqa: BLE001 - older FunASR builds only take one input at a time
return [self.asr.generate(input=c, cache={}, language=self.lang, use_itn=True)[0] for c in clips]

def transcribe(self, audio):
def transcribe(self, audio, terms=()): # SenseVoice takes no hints (vocabulary.py)
spans = self.vad.generate(input=audio)[0]["value"]
pieces = [(s, min(int(e), s + self.max_ms)) for b, e in spans for s in range(int(b), int(e), self.max_ms)]
out = []
Expand Down Expand Up @@ -350,12 +350,14 @@ def __init__(self, cfg, mlx=False):
name = "mlx-whisper" if mlx else "faster-whisper"
raise EngineMissing(f"{name} isn't installed: uv sync --extra {'mlx' if mlx else 'whisper'}") from e

def transcribe(self, audio):
def transcribe(self, audio, terms=()):
"""The segments; `terms` (vocabulary.py) go in as the prompt, the names Whisper should expect."""
hint = {"initial_prompt": vocabulary.prompt(terms)} if terms else {}
if self.mlx:
segs = self.m.transcribe(audio, path_or_hf_repo=self.repo, word_timestamps=True, language=self.lang)["segments"]
segs = self.m.transcribe(audio, path_or_hf_repo=self.repo, word_timestamps=True, language=self.lang, **hint)["segments"]
rows = [(s["start"], s["end"], s["text"], [(w["word"], w["start"], w["end"]) for w in s.get("words") or []]) for s in segs]
else:
segs, _ = self.m.transcribe(audio, vad_filter=True, word_timestamps=True, language=self.lang)
segs, _ = self.m.transcribe(audio, vad_filter=True, word_timestamps=True, language=self.lang, **hint)
rows = [(s.start, s.end, s.text, [(w.word, w.start, w.end) for w in (s.words or [])]) for s in segs]
return [
{
Expand Down Expand Up @@ -501,7 +503,8 @@ def transcribe_one(db, cfg, rid, log=print, engine=None):
r = db.one("SELECT record::id(id) AS id, space, path, title, remote FROM $r", r=store.R("recording", rid))
engine, t = engine or get_engine(cfg, log), time.time()
audio = decode(audio_path(db, cfg, r))
segs = engine.transcribe(audio)
words = vocabulary.terms(db, cfg, r["space"])
segs = engine.transcribe(audio, terms=words) if words else engine.transcribe(audio) # an engine of a plugin may take no hints
env = envelope(audio)
del audio
write_transcript(
Expand Down
36 changes: 24 additions & 12 deletions fastapi_backend/app/domain/speech.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@

import numpy as np

from . import store
from . import store, vocabulary

log = logging.getLogger("lens")
SR = 16000
Expand Down Expand Up @@ -298,7 +298,7 @@ def _label(v):


# ---------- providers ----------
def _openai(cfg, clip, seconds):
def _openai(cfg, clip, seconds, terms=()):
base, m, key = base_url(cfg, "openai"), model(cfg, "openai"), _key(cfg, "openai")
data, name, ctype = clip
lang = _language(cfg)
Expand All @@ -311,6 +311,8 @@ def _openai(cfg, clip, seconds):
("timestamp_granularities[]", "word")] # fmt: skip
if lang:
fields.append(("language", lang))
if terms and "diarize" not in m: # the diarizing models take no prompt
fields.append(("prompt", vocabulary.prompt(terms)))
body, ctype_ = multipart(fields, [("file", name, data, ctype)])
headers = {"Content-Type": ctype_, **({"Authorization": f"Bearer {key}"} if key else {})}
j = _json(base + "/audio/transcriptions", data=body, headers=headers, timeout=section(cfg)["timeout"])
Expand Down Expand Up @@ -350,7 +352,7 @@ def _short(lang):
return names.get(lang) or lang.split("-")[0].split("_")[0][:3]


def _elevenlabs(cfg, clip, seconds):
def _elevenlabs(cfg, clip, seconds, terms=()): # Scribe takes no hints (vocabulary.py)
data, name, ctype = clip
fields = [("model_id", model(cfg, "elevenlabs")), ("diarize", "true"), ("tag_audio_events", "true"), ("timestamps_granularity", "word")]
if _language(cfg):
Expand Down Expand Up @@ -380,7 +382,7 @@ def _elevenlabs(cfg, clip, seconds):
return group_words(words, _short(j.get("language_code")))


def _assemblyai(cfg, clip, seconds):
def _assemblyai(cfg, clip, seconds, terms=()):
base, key, t = base_url(cfg, "assemblyai"), _key(cfg, "assemblyai"), section(cfg)["timeout"]
data, _, ctype = clip
up = _json(base + "/v2/upload", data=data, headers={"Authorization": key, "Content-Type": "application/octet-stream"}, timeout=t)
Expand All @@ -394,6 +396,11 @@ def _assemblyai(cfg, clip, seconds):
ask["language_detection"] = True
if section(cfg)["sentiment"]:
ask["sentiment_analysis"] = True
if terms: # key terms for the models that take them, else boosted words (at most six words each)
if str(m or "").startswith(("slam", "universal-3")):
ask["keyterms_prompt"] = list(terms)
else:
ask["word_boost"] = [t for t in terms if len(t.split()) <= 6]
d = cfg.get("diarize") or {}
if d.get("min_speakers") or d.get("max_speakers"):
ask["speaker_options"] = {
Expand All @@ -420,17 +427,19 @@ def _assemblyai(cfg, clip, seconds):
return add_sentiment(segs, spans)


def _deepgram(cfg, clip, seconds):
def _deepgram(cfg, clip, seconds, terms=()):
data, _, ctype = clip
q = {"model": model(cfg, "deepgram"), "smart_format": "true", "punctuate": "true", "diarize": "true", "utterances": "true"}
q: dict = {"model": model(cfg, "deepgram"), "smart_format": "true", "punctuate": "true", "diarize": "true", "utterances": "true"}
if _language(cfg):
q["language"] = _language(cfg)
else:
q["detect_language"] = "true"
if section(cfg)["sentiment"]:
q["sentiment"] = "true"
if terms: # Nova-3 takes key terms; the models before it, keywords
q["keyterm" if str(q["model"]).startswith("nova-3") else "keywords"] = list(terms)
j = _json(
base_url(cfg, "deepgram") + "/v1/listen?" + urllib.parse.urlencode(q),
base_url(cfg, "deepgram") + "/v1/listen?" + urllib.parse.urlencode(q, doseq=True),
data=data,
headers={"Authorization": f"Token {_key(cfg, 'deepgram')}", "Content-Type": ctype},
timeout=section(cfg)["timeout"],
Expand Down Expand Up @@ -459,8 +468,8 @@ def _deepgram(cfg, clip, seconds):
return add_sentiment(segs, spans)


# each takes (cfg, an encoded clip as (bytes, file name, media type), its seconds) and returns its segments, timed
# from the clip's start
# each takes (cfg, an encoded clip as (bytes, file name, media type), its seconds, the words to expect) and returns its
# segments, timed from the clip's start
TRANSCRIBE = {"openai": _openai, "elevenlabs": _elevenlabs, "assemblyai": _assemblyai, "deepgram": _deepgram}


Expand Down Expand Up @@ -529,15 +538,18 @@ def __init__(self, cfg, provider):
_need(cfg, provider)
self.cfg, self.provider, self.name = cfg, provider, provider

def transcribe(self, audio):
"""The recording's transcript: in one request when it fits the provider's limits, else piece by piece."""
def transcribe(self, audio, terms=()):
"""The recording's transcript: in one request when it fits the provider's limits, else piece by piece; `terms`
are the words to expect (vocabulary.py)."""
most, most_bytes = limits(self.cfg, self.provider)
detail = {"seconds": round(len(audio) / SR, 2)} # the ledger row reads it when the call ends
if terms:
detail["vocabulary"] = len(terms)
parts = []
with _ledger("speech.transcribe", self.cfg, f"{self.provider}:{model(self.cfg, self.provider)}", detail):
for a, b in pieces(audio, most):
for start, end, clip in clips(audio, a, b, most_bytes):
parts.append((start, TRANSCRIBE[self.provider](self.cfg, clip, round((end - start) / SR, 2))))
parts.append((start, TRANSCRIBE[self.provider](self.cfg, clip, round((end - start) / SR, 2), terms)))
if len(parts) > 1:
detail["pieces"] = len(parts)
log.info("speech: %s s of audio went to %s in %d pieces", detail["seconds"], LABELS[self.provider], len(parts))
Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ def labels():
"sensevoice": {"model": "iic/SenseVoiceSmall", "hub": "ms", "vad_max_segment_ms": 30000, "batch_size": 16},
"whisper": {"model": "large-v3-turbo", "compute_type": "default"},
"mlx_whisper": {"model": "mlx-community/whisper-large-v3-turbo"},
"vocabulary": False, # give each recording's engine the names its namespace knows (vocabulary.py)
},
"diarize": {
"engine": "auto",
Expand Down
93 changes: 93 additions & 0 deletions fastapi_backend/app/domain/vocabulary.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
"""A namespace's own words for transcription (`transcribe.vocabulary`, off by default).

The names speech-to-text gets wrong are the ones only you use: family, colleagues, places, projects. With the setting
on, each recording's transcription is given the words its namespace already knows, so the same mistakes stop coming
back:
- the custom vocabulary (`analysis.gazetteer`, Settings › Analysis), first, as you wrote it there;
- the names of the namespace's named speakers;
- its defined entities (entity_setup.py), however the setup writes them;
- words people put in when they corrected its transcripts (segment_edit, newest first), the ones with a capital
letter that weren't there before: the names they had to fix.

How an engine takes them (`hints`): Whisper (faster-whisper, mlx-whisper) and OpenAI-compatible servers as the
prompt, a line of the names; Deepgram as key terms (Nova-3) or keywords; AssemblyAI as key terms (Universal-3 and
Slam-1) or boosted words. SenseVoice and ElevenLabs take no hints, so they transcribe as before. Nothing is sent
anywhere it wasn't going already: the words go only to the engine the recording goes to.
"""

from __future__ import annotations

import re
from collections import Counter

from . import store

MOST = 60 # terms per recording: Deepgram's key terms and Whisper's prompt both have room for about this many
PROMPT_CHARS = 600 # Whisper keeps the last 224 tokens of a prompt; a little more than that in characters
EDITS = 500 # the corrections read, newest first
WORD = re.compile(r"[^\W\d_][\w'’\-]*[^\W_]|[^\W\d_]", re.UNICODE)


def on(cfg):
return bool((cfg.get("transcribe") or {}).get("vocabulary"))


def _corrected(db, nid):
"""Words people put in when they corrected the namespace's lines, most often first: the capitalised ones a line
didn't have before (a name the engine got wrong), not the text cleanup's."""
from .cleanup import CLEANER

rows = db.rows(
"SELECT before.text AS before, after.text AS after, at FROM segment_edit"
" WHERE recording IN (SELECT VALUE record::id(id) FROM recording WHERE space = $s)"
" AND after.text != NONE AND by != $c ORDER BY at DESC LIMIT $n",
s=int(nid),
c=CLEANER,
n=EDITS,
)
seen: Counter[str] = Counter()
for r in rows:
had = {w.lower() for w in WORD.findall(str(r.get("before") or ""))}
for w in WORD.findall(str(r.get("after") or "")):
if len(w) >= 3 and any(ch.isupper() for ch in w) and w.lower() not in had:
seen[w] += 1
return [w for w, _ in seen.most_common()]


def terms(db, cfg, nid):
"""The words a recording of namespace `nid` is transcribed with, most wanted first; [] with the setting off."""
if not on(cfg) or nid is None:
return []
gaz = [str(line).partition("|")[0] for line in (cfg.get("analysis") or {}).get("gazetteer") or []] # "Name | TYPE"
spoken = db.values("SELECT VALUE name FROM speaker WHERE space = $s AND name != NONE AND name != '' ORDER BY name", s=int(nid))
named = [str(n) for n in spoken if not str(n).strip().lower().startswith("speaker")] # not "Speaker 2"
defined = db.values("SELECT VALUE name FROM entity WHERE space = $s AND defined = true AND hidden != true ORDER BY name", s=int(nid))
out, keys = [], set()
for t in [*gaz, *named, *(str(x) for x in defined), *_corrected(db, nid)]:
t = re.sub(r"\s+", " ", t).strip()
k = t.lower()
if t and k not in keys and len(t) <= 50:
keys.add(k)
out.append(t)
if len(out) >= MOST:
break
return out


def prompt(words):
"""The words as a line for a Whisper-style prompt, cut at a whole word to fit PROMPT_CHARS; '' for none."""
line = ""
for w in words:
nxt = f"{line}, {w}" if line else w
if len(nxt) + 1 > PROMPT_CHARS:
break
line = nxt
return f"{line}." if line else ""


def for_recording(db, cfg, rid):
"""The words for recording `rid` (by its namespace)."""
if not on(cfg):
return []
row = db.one("SELECT space FROM $r", r=store.R("recording", rid))
return terms(db, cfg, (row or {}).get("space"))
Loading
Loading