From 2f87e007332d5b9518939d342497dbe260ec3cc3 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 17 Oct 2026 01:34:48 +0000 Subject: [PATCH] feat(analysis): put punctuation back in transcripts that lack it The clean step can now restore full stops, commas and question marks with a local model, kredor/punctuate-all (MIT, 12 languages), the one the pasted punctuation script used. Settings -> Analysis -> Put punctuation back: off (the default), where a transcript has hardly any (imported captions), or in every transcript. Marks a transcript already has are kept, a mark the model is less than 70% sure of isn't added, and each line is given marks once. The model runs on Transformers' token-classification pipeline directly, since deepmultilingualpunctuation no longer loads with Transformers 5. The model, Transformers and PyTorch's CPU build (about 2 GB) are fetched the first time a clean step needs them (the punctuation extra and component); until then the rules run alone and the job's log says why. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ --- CHANGELOG.md | 5 + docs/components.md | 1 + docs/processing.md | 8 + fastapi_backend/app/domain/cleanup.py | 151 ++++++++++++++++-- fastapi_backend/app/domain/components.py | 38 +++++ fastapi_backend/app/domain/jobs.py | 2 +- fastapi_backend/app/domain/settings.py | 1 + fastapi_backend/app/domain/store.py | 4 +- fastapi_backend/pyproject.toml | 2 + fastapi_backend/tests/domain/test_cleanup.py | 97 ++++++++++- fastapi_backend/uv.lock | 8 +- .../components/settings/bodies.tsx | 7 +- nextjs-frontend/components/settings/model.ts | 12 ++ threat-model.json | 2 +- threat-model.md | 1 + threat-model.yaml | 4 +- 16 files changed, 324 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bb9bc8ffc..e8edce05f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased +- **Punctuation put back in transcripts.** With transcripts cleaned up, Settings → Analysis → Put punctuation back + adds full stops, commas and question marks with a local model (kredor/punctuate-all, MIT, 12 languages), to + transcripts that have hardly any (imported captions) or to every one, keeping the marks they have. The model and + PyTorch (about 2 GB) are fetched the first time it's needed; off by default + ([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)). - **Clean up transcripts before analysis.** With Settings → Analysis → Clean up transcripts before analysis on (off by default), a new Clean up step runs before Analyze: a word said three times in a row is kept once ("I I I looked" → "I looked"), doubled pronouns and articles once, "um" and "uh" go, and spacing, "i" and sentence capitals are diff --git a/docs/components.md b/docs/components.md index dce0c4309..591ef48b7 100644 --- a/docs/components.md +++ b/docs/components.md @@ -21,6 +21,7 @@ never fetch a speech or voice model. | Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` | | NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime | | Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) | +| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build | | DuckDuckGo search | `web.search` is `duckduckgo` | ddgs (MIT, about 30 MB with lxml and primp), the client the assistant's [web search](assistant.md#web-search-and-reading-pages) reads DuckDuckGo's results with | | yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address | | Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) | diff --git a/docs/processing.md b/docs/processing.md index 8d6c0bfeb..8151ef983 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -36,6 +36,14 @@ fetches nothing and sends nothing out: - one space between words, none before a comma or a full stop, one after a comma; - "i" becomes "I", and each sentence starts with a capital. A line that carries on the one before (same speaker, no full stop) keeps its lower case. +- punctuation (`analysis.cleanup_punctuation`, Put punctuation back; off): a local model, kredor/punctuate-all (MIT; + English, German, French, Spanish, Italian, Dutch, Portuguese, Polish, Czech, Slovak, Slovenian and Bulgarian), adds + full stops, commas and question marks after words that have none, where it is at least 70% sure. "Where a + transcript has none" (`missing`) does it only for a transcript with fewer than one full stop in 40 words, such as + imported captions; "In every transcript" (`always`) for every one. It keeps the marks a transcript has, and gives + each line marks once. The model, Transformers and PyTorch (about 2 GB) are fetched the first time a clean step + needs them ([Components](components.md)); until then, and in other languages, the rules run alone and the job's + log says why. These are English rules ("um" is a word in German and Portuguese), so they apply to English lines: the line's or the asset's language when transcription gave one, else when the text reads as English. In other languages only a word diff --git a/fastapi_backend/app/domain/cleanup.py b/fastapi_backend/app/domain/cleanup.py index 1b4657e85..763273ea7 100644 --- a/fastapi_backend/app/domain/cleanup.py +++ b/fastapi_backend/app/domain/cleanup.py @@ -15,6 +15,13 @@ said three times or more in a row kept once, and their sentences capitalised. A document's or an image's text is kept as it was read. +Punctuation (`analysis.cleanup_punctuation`, off by default): a transcript with hardly a full stop in it (imported +captions, some engines) gets full stops, commas and question marks back from a local model, kredor/punctuate-all +(MIT, XLM-RoBERTa, 12 languages), the one the punctuation script this comes from used. "missing" does it for those +transcripts only, "always" for every one; either way a mark the transcript has is kept, and each line is given marks +once. The model and PyTorch (about 2 GB) are fetched the first time a clean step needs them (components.py); until +then, or where it can't run, the rules run alone. + Every line it changes is kept in the asset's edit history, by "text cleanup": Change history lists them and Revert puts a line back. A line someone corrected is never cleaned, since people's corrections outrank the rules, and running the rules again on what they made changes nothing. @@ -139,34 +146,156 @@ def clean_line(text, english_rules=True, fillers=True, starts_sentence=True): return out.strip() +# ---------- punctuation, by a local model ---------- +PUNCTUATION_MODEL = ("kredor/punctuate-all", "0fe37019de3f5e4fbd83289fd94e07fa588e47df") # pinned: MIT, 12 languages +PUNCTUATION_FILES = ["config.json", "pytorch_model.bin", "special_tokens_map.json", "tokenizer.json", "tokenizer_config.json"] +PUNCTUATION_LANGS = {"en", "de", "fr", "es", "bg", "it", "pl", "nl", "cs", "pt", "sk", "sl"} +PUNCTUATED = f"{PUNCTUATION_MODEL[0]}@{PUNCTUATION_MODEL[1][:12]}" # a line the model has given marks to +SURE = 0.7 # a mark the model is less sure of than this isn't added (the script it comes from used 0.7 too) +CHUNK, OVERLAP = 150, 5 # words the model reads at once (well inside its 512 tokens), and how many chunks share +OWN_MARK = re.compile(r"[.,;:!?…]['\"”’)]*$") +SENTENCE_END = re.compile(r"[.!?…]['\"”’)]*$") +NUMBER_MARK = re.compile(r"(?= len(words) else len(batch) - OVERLAP + ends, at = [], 0 + for w in batch: + at += len(w) + ends.append(at) + at += 1 + marks, j = ["0"] * len(batch), 0 + for piece in pipe(" ".join(batch)): + while j < len(ends) and piece["end"] > ends[j]: + j += 1 + if j < len(ends): + marks[j] = piece["entity"] if float(piece["score"]) >= SURE else "0" + out += marks[:keep] + k += keep + return out + + +def punctuate(texts): + """The texts with the full stops, commas and question marks the model puts back after words that have no mark; + the marks they have are kept.""" + pipe = _pipe() + words = [t.split() for t in texts] + flat = [NUMBER_MARK.sub("", w) or w for ws in words for w in ws] + labels = _labels(pipe, flat) if flat else [] + out, k = [], 0 + for ws in words: + line = [w + m if m in (".", ",", "?") and not OWN_MARK.search(w) else w for w, m in zip(ws, labels[k : k + len(ws)])] + out.append(" ".join(line)) + k += len(ws) + return out + + +def lacks_punctuation(texts): + """Hardly a sentence's end in it: fewer than one in 40 words.""" + ws = [w for t in texts for w in t.split()] + return len(ws) >= 20 and sum(1 for w in ws if SENTENCE_END.search(w)) * 40 < len(ws) + + +def _punctuation(cfg, rec, segs): + """(the lines' texts with marks put back, the ids of the lines given marks, a note for the job's log).""" + mode = (cfg.get("analysis") or {}).get("cleanup_punctuation") or "off" + texts = [s.get("text") or "" for s in segs] + todo = [k for k, s in enumerate(segs) if s.get("punctuated") != PUNCTUATED and texts[k].strip()] + lang = str(rec.get("language") or "").lower()[:2] + known = lang in PUNCTUATION_LANGS or (not lang and english(None, texts[:400])) + if mode == "off" or not todo or not known or (mode == "missing" and not lacks_punctuation(texts)): + return texts, [], None + try: + marked = punctuate(texts) + except Unavailable as e: + return texts, [], f"punctuation not restored: {e}" + for k in todo: + texts[k] = marked[k] + return texts, [segs[k]["id"] for k in todo], f"punctuation restored by {PUNCTUATION_MODEL[0]}" + + +# ---------- an asset's lines ---------- def plan(db, cfg, rid): - """[(segment id, idx, text, cleaned)] for the lines the rules change; [] for a document or an image.""" + """What cleaning an asset does: {changes: [(segment id, idx, text, cleaned)], punctuated: [segment ids], note}. + Nothing for a document or an image.""" rec = db.one("SELECT source, language FROM $r", r=R("recording", rid)) or {} if rec.get("source") not in SPEECH: - return [] - segs = db.rows("SELECT record::id(id) AS id, idx, text, lang, speaker FROM segment WHERE recording = $r ORDER BY idx", r=rid) + return {"changes": [], "punctuated": [], "note": None} + segs = db.rows( + "SELECT record::id(id) AS id, idx, text, lang, speaker, punctuated FROM segment WHERE recording = $r ORDER BY idx", r=rid + ) people = set(db.values("SELECT VALUE idx FROM segment_edit WHERE recording = $r AND by != $c AND after.text != NONE", r=rid, c=CLEANER)) fillers = bool((cfg.get("analysis") or {}).get("cleanup_fillers", True)) whole = english(rec.get("language"), [s.get("text") or "" for s in segs[:400]]) + marked, punctuated, note = _punctuation(cfg, rec, segs) out, prev = [], None - for s in segs: - text = s.get("text") or "" - ended = prev is None or prev["speaker"] != s.get("speaker") or re.search(r"[.!?…]['\"”’)]*$", prev["text"]) is not None + for s, text in zip(segs, marked): + ended = prev is None or prev["speaker"] != s.get("speaker") or SENTENCE_END.search(prev["text"]) is not None prev = {"speaker": s.get("speaker"), "text": text} if s["idx"] in people or not text.strip(): continue lang = s.get("lang") en = english(lang, [text]) if lang and lang not in ("auto", "nospeech") else whole new = clean_line(text, en, fillers, ended) - if new and new != text: - out.append((s["id"], s["idx"], text, new)) + if new and new != (s.get("text") or ""): + out.append((s["id"], s["idx"], s.get("text") or "", new)) prev["text"] = new - return out + kept = {s["id"] for s in segs if s["idx"] in people} + return {"changes": out, "punctuated": [i for i in punctuated if i not in kept], "note": note} -def clean_recording(db, cfg, rid): +def clean_recording(db, cfg, rid, say=None): """Clean an asset's transcript lines; how many changed. Each change goes into its edit history.""" - changes = plan(db, cfg, rid) + p = plan(db, cfg, rid) + if p["note"] and say: + say(p["note"]) + changes = p["changes"] + if p["punctuated"]: + db.q("UPDATE segment SET punctuated = $m WHERE id IN $ids", m=PUNCTUATED, ids=[R("segment", i) for i in p["punctuated"]]) if not changes: return 0 last = int(db.values("UPSERT $r SET n += $k RETURN VALUE n", r=R("seq", "segment_edit"), k=len(changes))[0]) diff --git a/fastapi_backend/app/domain/components.py b/fastapi_backend/app/domain/components.py index df3b6015e..ae249d80b 100644 --- a/fastapi_backend/app/domain/components.py +++ b/fastapi_backend/app/domain/components.py @@ -496,6 +496,36 @@ def fetch(self, cfg, m, say): fetch_file(cfg, n, say) +class Punctuation(Component): + """PyTorch, Transformers and the punctuation model (cleanup.py), when Settings → Analysis cleans up transcripts and + restores their punctuation.""" + + kind = "model" + + def needed(self, cfg, m): + a = cfg.get("analysis") or {} + return bool(a.get("cleanup")) and (a.get("cleanup_punctuation") or "off") != "off" + + def present(self, cfg): + from . import cleanup + + return importable("transformers", "torch") and cleanup.punctuation_ready() + + def fetch(self, cfg, m, say): + if not importable("transformers", "torch"): + pip_install(cfg, "punctuation", say) + from . import cleanup + + if say: + say(f"downloading the punctuation model {cleanup.PUNCTUATION_MODEL[0]}") + cleanup.fetch_punctuation() + + def detail(self, cfg): + from . import cleanup + + return cleanup.PUNCTUATION_MODEL[0] + + class Anytopdf(Component): """The anytopdf program (anytopdf.py), fetched when Settings → Documents asks for it to run here, or Settings → Analysis asks it for places and dates (enrich.py), or Settings → Video for objects (objects.py) or faces.""" @@ -771,6 +801,14 @@ def detail(self, cfg): license="Apache-2.0", ), LAYA, + Punctuation( + "punctuation", + "Punctuation model", + "puts full stops, commas and question marks back in transcripts that lack them (analysis.cleanup_punctuation)", + steps={"clean"}, + size_mb=1300, + license="MIT", + ), InjectionModel( "injection-model", "Prompt-injection classifier", diff --git a/fastapi_backend/app/domain/jobs.py b/fastapi_backend/app/domain/jobs.py index ccd74c61c..7d2ca549c 100644 --- a/fastapi_backend/app/domain/jobs.py +++ b/fastapi_backend/app/domain/jobs.py @@ -88,7 +88,7 @@ def _clean(db, cfg, rid, say, spec=None): raise Skip("cleaning up transcripts is off (Settings → Analysis)") if _source(db, rid) not in cleanup.SPEECH: raise Skip("a document's or an image's text is kept as it was read") - n = cleanup.clean_recording(db, cfg, rid) + n = cleanup.clean_recording(db, cfg, rid, say) say(f"cleaned up {n} line(s); Change history lists them" if n else "nothing to clean up") diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index a22913065..e707b6c8e 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -158,6 +158,7 @@ ("speakers", "embedder"): {"speechbrain", "none"}, ("speakers", "cross_namespace"): {"suggest", "off"}, ("analysis", "entities"): {"rules", "spacy"}, + ("analysis", "cleanup_punctuation"): {"off", "missing", "always"}, ("search", "stemming"): {"english", "none"}, ("search", "engine"): set(store.SEARCH_ENGINES), ("reports", "audio"): {"link", "embed", "none"}, diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index e40e26b1d..2298c1e03 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -154,7 +154,8 @@ def labels(): "cross_namespace": "suggest", }, # anytopdf: the places and dates anytopdf (run here) finds, as hints for the extractor (enrich.py); cleanup: the - # clean step tidies transcripts before analysis (cleanup.py), cleanup_fillers taking out "um" and "uh" too + # clean step tidies transcripts before analysis (cleanup.py), cleanup_fillers taking out "um" and "uh" too, and + # cleanup_punctuation (off, missing, always) putting marks back with a local model "analysis": { "entities": "rules", "spacy_model": "en_core_web_sm", @@ -162,6 +163,7 @@ def labels(): "anytopdf": False, "cleanup": False, "cleanup_fillers": True, + "cleanup_punctuation": "off", }, # chat_models: the models people may pick in Chat; empty: whatever the model server lists (docs/configuration.md) "llm": { diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml index 2eed5fc03..0f1b99cc9 100644 --- a/fastapi_backend/pyproject.toml +++ b/fastapi_backend/pyproject.toml @@ -45,6 +45,8 @@ insightface = ["insightface", "onnxruntime"] # Objects: YOLOX on ONNX Runtime (Apache-2.0; the model is a YOLOX .onnx file). Ultralytics YOLO also works if # installed by hand (pip install ultralytics): it's AGPL-3.0, so it isn't an extra here. objects = ["onnxruntime"] +# Punctuation put back in transcripts (analysis.cleanup_punctuation): kredor/punctuate-all on Transformers and PyTorch +punctuation = ["transformers", "torch"] # A local prompt-injection classifier (ai.injection_check "classifier"): ProtectAI's DeBERTa model on ONNX Runtime injection = ["onnxruntime", "tokenizers"] # Checking IIIF manifests against the official schema diff --git a/fastapi_backend/tests/domain/test_cleanup.py b/fastapi_backend/tests/domain/test_cleanup.py index a1bec589e..ad2b474a9 100644 --- a/fastapi_backend/tests/domain/test_cleanup.py +++ b/fastapi_backend/tests/domain/test_cleanup.py @@ -117,7 +117,7 @@ def test_documents_and_fillers(db, cfg, folder): drain(db, cfg) assert _lines(db, rid)[0] == "Yeah. Sorry, sorry about that. Okay, um, so now it's recording." db.q("UPDATE $r SET source = 'document'", r=R("recording", rid)) - assert cleanup.plan(db, cfg, rid) == [] + assert cleanup.plan(db, cfg, rid)["changes"] == [] assert cleanup.clean_pending(db, cfg, log=quiet) == 0 # the command line's clean: transcripts only @@ -133,3 +133,98 @@ def test_workers_with_an_older_steps_list_run_it(db, cfg): assert "clean" in w.can cfg["workers"]["steps"] = ["analyze", "report"] assert "clean" in jobs.Worker(db, lambda: cfg, name="older").can + + +# ---------- punctuation ---------- +BARE = """Ann|N|yeah sorry about that okay so now it's recording so if I go back in network +Ben|N|is it like it should be in the chunks it's only every 30 seconds so it might take a while +Ann|N|I looked into different methods as well like trying to do that even ran local API""" +ENDS = {"recording": ".", "network": ".", "chunks": "?", "while": ".", "API": ".", "yeah": ","} + + +class FakePipe: + """Transformers' token-classification pipeline as the punctuation model answers: a piece per word (two for long + words, the mark on the last), the mark after the words in ENDS.""" + + def __init__(self): + self.calls = 0 + + def __call__(self, text): + self.calls += 1 + out, at = [], 0 + for w in text.split(" "): + if len(w) > 6: + out.append({"entity": "0", "score": 0.99, "start": at, "end": at + 3}) + out.append({"entity": ENDS.get(w, "0"), "score": 0.9, "start": at, "end": at + len(w)}) + at += len(w) + 1 + return out + + +@pytest.fixture +def model(monkeypatch): + pipe = FakePipe() + monkeypatch.setitem(cleanup._PIPE, "pipe", pipe) + return pipe + + +def test_labels_read_long_text_in_overlapping_chunks(model): + words = ("so we went to the market and then we bought apples while " * 40).split() + labels = cleanup._labels(model, words) + assert len(labels) == len(words) and model.calls == 4 + assert [w for w, m in zip(words, labels, strict=True) if m == "."] == ["while"] * 40 + + +def test_punctuation_put_back_where_it_is_missing(db, cfg, model): + cfg["analysis"].update(cleanup=True, cleanup_punctuation="missing") + rid = ingest.import_text(db, cfg, "pods", BARE, "Bare", "auto") + jid = jobs.enqueue(db, rid, ["clean"]) + drain(db, cfg) + assert _lines(db, rid) == [ + "Yeah, sorry about that okay so now it's recording. So if I go back in network.", + "Is it like it should be in the chunks? It's only every 30 seconds so it might take a while.", + "I looked into different methods as well like trying to do that even ran local API.", + ] + log = db.one("SELECT log FROM $j", j=R("job", jid))["log"] + assert any("punctuation restored by kredor/punctuate-all" in line for line in log) + assert set(db.values("SELECT VALUE punctuated FROM segment WHERE recording = $r", r=rid)) == {cleanup.PUNCTUATED} + + # a line is given marks once, and a transcript with them isn't read again + calls = model.calls + cfg["analysis"]["cleanup_punctuation"] = "always" + jobs.enqueue(db, rid, ["clean"]) + drain(db, cfg) + assert model.calls == calls + assert len(db.rows("SELECT * FROM segment_edit WHERE recording = $r", r=rid)) == 3 + + +def test_a_transcript_with_punctuation_keeps_it(db, cfg, model): + cfg["analysis"].update(cleanup=True, cleanup_punctuation="missing") + rid = _talk(db, cfg) + jobs.enqueue(db, rid, ["clean"]) + drain(db, cfg) + assert model.calls == 0 # it has its full stops: only the rules ran + assert _lines(db, rid)[0] == "Yeah. Sorry, sorry about that. Okay, so now it's recording." + # "always" reads it too, but keeps every mark it has + assert cleanup.punctuate(["Sorry, I mean it. yeah"]) == ["Sorry, I mean it. yeah,"] + + +def test_without_the_model_the_rules_run_alone(db, cfg): + cfg["analysis"].update(cleanup=True, cleanup_punctuation="always") + rid = ingest.import_text(db, cfg, "pods", BARE, "Bare", "auto") + jid = jobs.enqueue(db, rid, ["clean"]) + drain(db, cfg) + log = db.one("SELECT log FROM $j", j=R("job", jid))["log"] + assert any("punctuation not restored: the punctuation model isn't ready here" in line for line in log) + assert _lines(db, rid)[0] == "Yeah sorry about that okay so now it's recording so if I go back in network" + assert not db.values("SELECT VALUE punctuated FROM segment WHERE recording = $r AND punctuated != NONE", r=rid) + + +def test_the_model_is_fetched_only_when_it_is_asked_for(cfg): + from app.domain import components, machine + + c, m = components.BY_ID["punctuation"], machine.probe(cfg["data_dir"]) + assert not c.needed(cfg, m) + cfg["analysis"]["cleanup_punctuation"] = "missing" + assert not c.needed(cfg, m) # cleanup itself is off + cfg["analysis"]["cleanup"] = True + assert c.needed(cfg, m) and c.serves(cfg, m) == {"clean"} diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock index 411ba82f2..fee4729f7 100644 --- a/fastapi_backend/uv.lock +++ b/fastapi_backend/uv.lock @@ -1871,6 +1871,10 @@ msg = [ objects = [ { name = "onnxruntime" }, ] +punctuation = [ + { name = "torch" }, + { name = "transformers" }, +] pyannote = [ { name = "pyannote-audio" }, ] @@ -1971,13 +1975,15 @@ requires-dist = [ { name = "starlette", specifier = ">=1.3.1" }, { name = "surrealdb", specifier = ">=2.0,<3" }, { name = "tokenizers", marker = "extra == 'injection'" }, + { name = "torch", marker = "extra == 'punctuation'" }, { name = "torch", marker = "extra == 'sensevoice'" }, { name = "torch", marker = "extra == 'voices'" }, { name = "torchaudio", marker = "extra == 'sensevoice'" }, { name = "torchaudio", marker = "extra == 'voices'" }, + { name = "transformers", marker = "extra == 'punctuation'" }, { name = "webauthn", specifier = ">=3.0.1,<4" }, ] -provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "injection", "validate", "smb", "sandbox", "web", "msg"] +provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "injection", "validate", "smb", "sandbox", "web", "msg"] [package.metadata.requires-dev] dev = [ diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index 702eab9b5..3c26a7cf4 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -243,7 +243,12 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) { return ( <> - {raw("analysis.cleanup") === true && } + {raw("analysis.cleanup") === true && ( +
+ + +
+ )} {raw("analysis.entities") === "spacy" && } diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index f620f00ec..3de5d870d 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -762,6 +762,18 @@ export const FIELDS: FieldSpec[] = [ label: "Take out “um” and “uh”", kind: "switch", }, + { + section: "analysis", + key: "cleanup_punctuation", + label: "Put punctuation back", + kind: "select", + options: [ + { value: "off", label: "Off" }, + { value: "missing", label: "Where a transcript has none" }, + { value: "always", label: "In every transcript" }, + ], + hint: "A local model adds full stops, commas and question marks, keeping the ones there; about 2 GB, fetched the first time it’s needed", + }, { section: "analysis", key: "gazetteer", diff --git a/threat-model.json b/threat-model.json index 76f12f39a..f3f237519 100644 --- a/threat-model.json +++ b/threat-model.json @@ -6,7 +6,7 @@ "scope_subpath": null, "description": "Lens is a self-hosted archive for documents, recordings, photos and messages: it reads, transcribes, indexes and publishes them, and people search and chat over them. One operator (the instance admin) runs an instance for their own people: a FastAPI API, a Next.js web app that proxies the API and holds the sign-in session, and background workers, around one SurrealDB database, with content split into namespaces and per-namespace roles. Exposing an instance to the internet behind HTTPS (reverse proxy or Cloudflare tunnel) is supported. Only the instance admin (equivalent to host code execution) and the host operator are trusted; everyone else is trusted only as far as their role.", "confidence": { - "documented": 247, + "documented": 248, "inferred": 99 }, "components": [ diff --git a/threat-model.md b/threat-model.md index e727ab00b..d62618ed7 100644 --- a/threat-model.md +++ b/threat-model.md @@ -459,6 +459,7 @@ third-party source; dependencies come from `uv.lock`, the pnpm lockfile and cont | cloudflared, llama.cpp, `hf:` models, pip packages | integrity only as far as HTTPS and the publisher | upstream; tampered downloads are disclaimed (N-DOWNLOADS) | *(maintainer, 2026-10)* | | Catalog GGUF models | pinned revision and SHA-256 | in model (P-MODEL-CHECKSUM) | *(documented, docs/local-models.md "checked against their SHA-256")* | | Prompt-injection classifier (ProtectAI deberta-v3-base-prompt-injection-v2, `injection` extra, opt-in) | a judgement on whether a tool's result reads like an injection; the model is fetched at a pinned revision and checked against its SHA-256 | upstream for the model's judgement: a miss or a false flag is N-PROMPT; a model file that passes unchecked is in model | *(documented, docs/assistant.md "fetched when you pick it")* | +| Punctuation model (kredor/punctuate-all, `punctuation` extra, opt-in: `analysis.cleanup_punctuation`) | full stops, commas and question marks for a transcript's lines; the model is fetched from Hugging Face at a pinned revision, and Transformers loads it there with no remote code | upstream for the model's judgement (a missed or wrong mark, which the edit history can revert); a model file from another revision is in model | *(documented, docs/components.md "at a pinned revision")* | | Optional ML extras (whisper, pyannote, speechbrain, insightface, doctr) | model inference; some fetch weights on their own | upstream | *(inferred, Q19)* | | StyleTTS2 service (`styletts2/`, opt-in Compose profile) and macOS `say` | text to speech; StyleTTS2 fetches its weights from Hugging Face and GitHub on first use and listens only on the Compose network; `say` gets the text on stdin, never as arguments | upstream for the model and synthesis; the argument handling in model | *(maintainer, 2026-10)* | | MCP runner (`mcp-runner/`, opt-in Compose profile) and the stdio MCP servers admins add (usually `npx` or `uvx` packages) | running those servers away from Lens: a container with none of Lens's data, keys or database, a non-root user, every capability dropped, `no-new-privileges`, a read-only image, memory and process limits, a network shared only with the backend (plus the internet to fetch packages), an empty home and only its extension's environment per server, and a bearer token on a volume only the two share | upstream for the servers and the packages they fetch (an admin's choice, trusted input, §1.3); the runner's isolation and token check in model | *(documented, docs/assistant.md "The MCP runner")* | diff --git a/threat-model.yaml b/threat-model.yaml index fb32efc5e..c1e30cf6c 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,10 +2,10 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:45d097d8ee9f65b716fd67657f7b91d03578c3a3c5c8d64d267f0d1648f33bd6" +prose_version: "threat-model.md@sha256:9aaa1da2ce1855376db0fca561d34c3b1bb6e0cb5ec56becd45bbca6269a5d02" model_status: under-review triage_policy: strict -confidence: {documented: 202, maintainer: 46, inferred: 91, assumption: 8} +confidence: {documented: 203, maintainer: 46, inferred: 91, assumption: 8} generation: model: "Claude Opus 5.5 (claude-opus-5-5)"