Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Punctuation put back in transcripts.** With transcripts cleaned up, Settings → Analysis → Put punctuation back
adds full stops, commas and question marks with a local model (kredor/punctuate-all, MIT, 12 languages), to
transcripts that have hardly any (imported captions) or to every one, keeping the marks they have. The model and
PyTorch (about 2 GB) are fetched the first time it's needed; off by default
([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)).
- **Clean up transcripts before analysis.** With Settings → Analysis → Clean up transcripts before analysis on (off
by default), a new Clean up step runs before Analyze: a word said three times in a row is kept once ("I I I looked"
→ "I looked"), doubled pronouns and articles once, "um" and "uh" go, and spacing, "i" and sentence capitals are
Expand Down
1 change: 1 addition & 0 deletions docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ never fetch a speech or voice model.
| Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` |
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| DuckDuckGo search | `web.search` is `duckduckgo` | ddgs (MIT, about 30 MB with lxml and primp), the client the assistant's [web search](assistant.md#web-search-and-reading-pages) reads DuckDuckGo's results with |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
Expand Down
8 changes: 8 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@ fetches nothing and sends nothing out:
- one space between words, none before a comma or a full stop, one after a comma;
- "i" becomes "I", and each sentence starts with a capital. A line that carries on the one before (same speaker, no
full stop) keeps its lower case.
- punctuation (`analysis.cleanup_punctuation`, Put punctuation back; off): a local model, kredor/punctuate-all (MIT;
English, German, French, Spanish, Italian, Dutch, Portuguese, Polish, Czech, Slovak, Slovenian and Bulgarian), adds
full stops, commas and question marks after words that have none, where it is at least 70% sure. "Where a
transcript has none" (`missing`) does it only for a transcript with fewer than one full stop in 40 words, such as
imported captions; "In every transcript" (`always`) for every one. It keeps the marks a transcript has, and gives
each line marks once. The model, Transformers and PyTorch (about 2 GB) are fetched the first time a clean step
needs them ([Components](components.md)); until then, and in other languages, the rules run alone and the job's
log says why.

These are English rules ("um" is a word in German and Portuguese), so they apply to English lines: the line's or the
asset's language when transcription gave one, else when the text reads as English. In other languages only a word
Expand Down
151 changes: 140 additions & 11 deletions fastapi_backend/app/domain/cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@
said three times or more in a row kept once, and their sentences capitalised. A document's or an image's text is kept
as it was read.

Punctuation (`analysis.cleanup_punctuation`, off by default): a transcript with hardly a full stop in it (imported
captions, some engines) gets full stops, commas and question marks back from a local model, kredor/punctuate-all
(MIT, XLM-RoBERTa, 12 languages), the one the punctuation script this comes from used. "missing" does it for those
transcripts only, "always" for every one; either way a mark the transcript has is kept, and each line is given marks
once. The model and PyTorch (about 2 GB) are fetched the first time a clean step needs them (components.py); until
then, or where it can't run, the rules run alone.

Every line it changes is kept in the asset's edit history, by "text cleanup": Change history lists them and Revert
puts a line back. A line someone corrected is never cleaned, since people's corrections outrank the rules, and running
the rules again on what they made changes nothing.
Expand Down Expand Up @@ -139,34 +146,156 @@ def clean_line(text, english_rules=True, fillers=True, starts_sentence=True):
return out.strip()


# ---------- punctuation, by a local model ----------
PUNCTUATION_MODEL = ("kredor/punctuate-all", "0fe37019de3f5e4fbd83289fd94e07fa588e47df") # pinned: MIT, 12 languages
PUNCTUATION_FILES = ["config.json", "pytorch_model.bin", "special_tokens_map.json", "tokenizer.json", "tokenizer_config.json"]
PUNCTUATION_LANGS = {"en", "de", "fr", "es", "bg", "it", "pl", "nl", "cs", "pt", "sk", "sl"}
PUNCTUATED = f"{PUNCTUATION_MODEL[0]}@{PUNCTUATION_MODEL[1][:12]}" # a line the model has given marks to
SURE = 0.7 # a mark the model is less sure of than this isn't added (the script it comes from used 0.7 too)
CHUNK, OVERLAP = 150, 5 # words the model reads at once (well inside its 512 tokens), and how many chunks share
OWN_MARK = re.compile(r"[.,;:!?…]['\"”’)]*$")
SENTENCE_END = re.compile(r"[.!?…]['\"”’)]*$")
NUMBER_MARK = re.compile(r"(?<!\d)[.,;:!?](?!\d)") # marks the model is shown without (not 3.5 or 1,000)
_PIPE: dict = {}


class Unavailable(RuntimeError):
"""The punctuation model can't run here (yet); the message says why."""


def fetch_punctuation():
"""Download the model at its pinned revision into the models folder (Hugging Face's cache there)."""
from huggingface_hub import snapshot_download

name, rev = PUNCTUATION_MODEL
return snapshot_download(name, revision=rev, allow_patterns=PUNCTUATION_FILES)


def punctuation_ready():
try:
from huggingface_hub import snapshot_download

name, rev = PUNCTUATION_MODEL
snapshot_download(name, revision=rev, allow_patterns=PUNCTUATION_FILES, local_files_only=True)
return True
except Exception: # noqa: BLE001 - not installed, or not downloaded
return False


def _pipe():
if "pipe" not in _PIPE:
try:
from huggingface_hub import snapshot_download
from transformers import pipeline

name, rev = PUNCTUATION_MODEL
path = snapshot_download(name, revision=rev, allow_patterns=PUNCTUATION_FILES, local_files_only=True)
_PIPE["pipe"] = pipeline("token-classification", model=path, aggregation_strategy="none", device=-1)
except Exception as e: # noqa: BLE001 - not fetched yet, or it won't load here
raise Unavailable(f"the punctuation model isn't ready here ({type(e).__name__}; Settings → Components)") from None
return _PIPE["pipe"]


def _labels(pipe, words):
"""The model's mark after each word ("0": none), read CHUNK words at a time with OVERLAP words shared, as
deepmultilingualpunctuation does: a word's last piece decides, and a mark it's less than SURE of counts as none."""
out, k = [], 0
while k < len(words):
batch = words[k : k + CHUNK]
keep = len(batch) if k + CHUNK >= len(words) else len(batch) - OVERLAP
ends, at = [], 0
for w in batch:
at += len(w)
ends.append(at)
at += 1
marks, j = ["0"] * len(batch), 0
for piece in pipe(" ".join(batch)):
while j < len(ends) and piece["end"] > ends[j]:
j += 1
if j < len(ends):
marks[j] = piece["entity"] if float(piece["score"]) >= SURE else "0"
out += marks[:keep]
k += keep
return out


def punctuate(texts):
"""The texts with the full stops, commas and question marks the model puts back after words that have no mark;
the marks they have are kept."""
pipe = _pipe()
words = [t.split() for t in texts]
flat = [NUMBER_MARK.sub("", w) or w for ws in words for w in ws]
labels = _labels(pipe, flat) if flat else []
out, k = [], 0
for ws in words:
line = [w + m if m in (".", ",", "?") and not OWN_MARK.search(w) else w for w, m in zip(ws, labels[k : k + len(ws)])]
out.append(" ".join(line))
k += len(ws)
return out


def lacks_punctuation(texts):
"""Hardly a sentence's end in it: fewer than one in 40 words."""
ws = [w for t in texts for w in t.split()]
return len(ws) >= 20 and sum(1 for w in ws if SENTENCE_END.search(w)) * 40 < len(ws)


def _punctuation(cfg, rec, segs):
"""(the lines' texts with marks put back, the ids of the lines given marks, a note for the job's log)."""
mode = (cfg.get("analysis") or {}).get("cleanup_punctuation") or "off"
texts = [s.get("text") or "" for s in segs]
todo = [k for k, s in enumerate(segs) if s.get("punctuated") != PUNCTUATED and texts[k].strip()]
lang = str(rec.get("language") or "").lower()[:2]
known = lang in PUNCTUATION_LANGS or (not lang and english(None, texts[:400]))
if mode == "off" or not todo or not known or (mode == "missing" and not lacks_punctuation(texts)):
return texts, [], None
try:
marked = punctuate(texts)
except Unavailable as e:
return texts, [], f"punctuation not restored: {e}"
for k in todo:
texts[k] = marked[k]
return texts, [segs[k]["id"] for k in todo], f"punctuation restored by {PUNCTUATION_MODEL[0]}"


# ---------- an asset's lines ----------
def plan(db, cfg, rid):
"""[(segment id, idx, text, cleaned)] for the lines the rules change; [] for a document or an image."""
"""What cleaning an asset does: {changes: [(segment id, idx, text, cleaned)], punctuated: [segment ids], note}.
Nothing for a document or an image."""
rec = db.one("SELECT source, language FROM $r", r=R("recording", rid)) or {}
if rec.get("source") not in SPEECH:
return []
segs = db.rows("SELECT record::id(id) AS id, idx, text, lang, speaker FROM segment WHERE recording = $r ORDER BY idx", r=rid)
return {"changes": [], "punctuated": [], "note": None}
segs = db.rows(
"SELECT record::id(id) AS id, idx, text, lang, speaker, punctuated FROM segment WHERE recording = $r ORDER BY idx", r=rid
)
people = set(db.values("SELECT VALUE idx FROM segment_edit WHERE recording = $r AND by != $c AND after.text != NONE", r=rid, c=CLEANER))
fillers = bool((cfg.get("analysis") or {}).get("cleanup_fillers", True))
whole = english(rec.get("language"), [s.get("text") or "" for s in segs[:400]])
marked, punctuated, note = _punctuation(cfg, rec, segs)
out, prev = [], None
for s in segs:
text = s.get("text") or ""
ended = prev is None or prev["speaker"] != s.get("speaker") or re.search(r"[.!?…]['\"”’)]*$", prev["text"]) is not None
for s, text in zip(segs, marked):
ended = prev is None or prev["speaker"] != s.get("speaker") or SENTENCE_END.search(prev["text"]) is not None
prev = {"speaker": s.get("speaker"), "text": text}
if s["idx"] in people or not text.strip():
continue
lang = s.get("lang")
en = english(lang, [text]) if lang and lang not in ("auto", "nospeech") else whole
new = clean_line(text, en, fillers, ended)
if new and new != text:
out.append((s["id"], s["idx"], text, new))
if new and new != (s.get("text") or ""):
out.append((s["id"], s["idx"], s.get("text") or "", new))
prev["text"] = new
return out
kept = {s["id"] for s in segs if s["idx"] in people}
return {"changes": out, "punctuated": [i for i in punctuated if i not in kept], "note": note}


def clean_recording(db, cfg, rid):
def clean_recording(db, cfg, rid, say=None):
"""Clean an asset's transcript lines; how many changed. Each change goes into its edit history."""
changes = plan(db, cfg, rid)
p = plan(db, cfg, rid)
if p["note"] and say:
say(p["note"])
changes = p["changes"]
if p["punctuated"]:
db.q("UPDATE segment SET punctuated = $m WHERE id IN $ids", m=PUNCTUATED, ids=[R("segment", i) for i in p["punctuated"]])
if not changes:
return 0
last = int(db.values("UPSERT $r SET n += $k RETURN VALUE n", r=R("seq", "segment_edit"), k=len(changes))[0])
Expand Down
38 changes: 38 additions & 0 deletions fastapi_backend/app/domain/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,36 @@ def fetch(self, cfg, m, say):
fetch_file(cfg, n, say)


class Punctuation(Component):
"""PyTorch, Transformers and the punctuation model (cleanup.py), when Settings → Analysis cleans up transcripts and
restores their punctuation."""

kind = "model"

def needed(self, cfg, m):
a = cfg.get("analysis") or {}
return bool(a.get("cleanup")) and (a.get("cleanup_punctuation") or "off") != "off"

def present(self, cfg):
from . import cleanup

return importable("transformers", "torch") and cleanup.punctuation_ready()

def fetch(self, cfg, m, say):
if not importable("transformers", "torch"):
pip_install(cfg, "punctuation", say)
from . import cleanup

if say:
say(f"downloading the punctuation model {cleanup.PUNCTUATION_MODEL[0]}")
cleanup.fetch_punctuation()

def detail(self, cfg):
from . import cleanup

return cleanup.PUNCTUATION_MODEL[0]


class Anytopdf(Component):
"""The anytopdf program (anytopdf.py), fetched when Settings → Documents asks for it to run here, or Settings →
Analysis asks it for places and dates (enrich.py), or Settings → Video for objects (objects.py) or faces."""
Expand Down Expand Up @@ -771,6 +801,14 @@ def detail(self, cfg):
license="Apache-2.0",
),
LAYA,
Punctuation(
"punctuation",
"Punctuation model",
"puts full stops, commas and question marks back in transcripts that lack them (analysis.cleanup_punctuation)",
steps={"clean"},
size_mb=1300,
license="MIT",
),
InjectionModel(
"injection-model",
"Prompt-injection classifier",
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ def _clean(db, cfg, rid, say, spec=None):
raise Skip("cleaning up transcripts is off (Settings → Analysis)")
if _source(db, rid) not in cleanup.SPEECH:
raise Skip("a document's or an image's text is kept as it was read")
n = cleanup.clean_recording(db, cfg, rid)
n = cleanup.clean_recording(db, cfg, rid, say)
say(f"cleaned up {n} line(s); Change history lists them" if n else "nothing to clean up")


Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@
("speakers", "embedder"): {"speechbrain", "none"},
("speakers", "cross_namespace"): {"suggest", "off"},
("analysis", "entities"): {"rules", "spacy"},
("analysis", "cleanup_punctuation"): {"off", "missing", "always"},
("search", "stemming"): {"english", "none"},
("search", "engine"): set(store.SEARCH_ENGINES),
("reports", "audio"): {"link", "embed", "none"},
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,14 +154,16 @@ def labels():
"cross_namespace": "suggest",
},
# anytopdf: the places and dates anytopdf (run here) finds, as hints for the extractor (enrich.py); cleanup: the
# clean step tidies transcripts before analysis (cleanup.py), cleanup_fillers taking out "um" and "uh" too
# clean step tidies transcripts before analysis (cleanup.py), cleanup_fillers taking out "um" and "uh" too, and
# cleanup_punctuation (off, missing, always) putting marks back with a local model
"analysis": {
"entities": "rules",
"spacy_model": "en_core_web_sm",
"gazetteer": [],
"anytopdf": False,
"cleanup": False,
"cleanup_fillers": True,
"cleanup_punctuation": "off",
},
# chat_models: the models people may pick in Chat; empty: whatever the model server lists (docs/configuration.md)
"llm": {
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ insightface = ["insightface", "onnxruntime"]
# Objects: YOLOX on ONNX Runtime (Apache-2.0; the model is a YOLOX .onnx file). Ultralytics YOLO also works if
# installed by hand (pip install ultralytics): it's AGPL-3.0, so it isn't an extra here.
objects = ["onnxruntime"]
# Punctuation put back in transcripts (analysis.cleanup_punctuation): kredor/punctuate-all on Transformers and PyTorch
punctuation = ["transformers", "torch"]
# A local prompt-injection classifier (ai.injection_check "classifier"): ProtectAI's DeBERTa model on ONNX Runtime
injection = ["onnxruntime", "tokenizers"]
# Checking IIIF manifests against the official schema
Expand Down
Loading
Loading