Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Clean up transcripts before analysis.** With Settings → Analysis → Clean up transcripts before analysis on (off
by default), a new Clean up step runs before Analyze: a word said three times in a row is kept once ("I I I looked"
→ "I looked"), doubled pronouns and articles once, "um" and "uh" go, and spacing, "i" and sentence capitals are
fixed, by rules, on English lines. Every changed line is in Change history with Revert, and lines someone corrected
are left alone. Reprocess, pipelines, `POST /api/v1/recordings/<id>/reprocess` and `lens clean` run it
([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)).
- **Automations run.** An automation saved over the API (`/api/v1/automations`) now reacts to new assets in its
namespaces: when its If passes, it runs its workflow as the person who made it, within its limits and budgets, and
each run shows what its actions did. Try one on an activity without keeping or sending anything, or run it now;
Expand Down
13 changes: 13 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,19 @@

Plans and progress for work in flight. Newest first.

## 2026-10-17 · Cleaning up transcripts before analysis

Adeel: a text cleaning step before entity extraction and transcript processing, from a punctuation and grammar
script he pasted. The `clean` step (cleanup.py) runs before `analyze` in the standard pipeline, off until Settings →
Analysis turns it on: rules take out fillers and stutters and tidy spacing and capitals, each change kept in the edit
history with Revert. The script's alternating "Speaker 1 / Speaker 2" labels aren't carried over: Lens tells voices
apart by diarization, or keeps the labels a transcript came with.

Next: punctuation restored by a local model (the script's deepmultilingualpunctuation, fetched on first use), grammar
from a LanguageTool server, spaCy fetched for named entities, and dates found and normalised.

Refine later: revert a whole cleanup at once; cleanup rules for other languages' fillers; per-namespace switch.

## 2026-10-14 · Long recordings in pieces a speech provider takes

Adeel: large audio files are segmented automatically to each transcription provider's limits. `speech.Engine` cuts a
Expand Down
30 changes: 29 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,40 @@ How assets move through Lens: where they come from, what each step does, and how
([Documents and images](configuration.md#documents-and-images)).
- `diarize` splits genuinely two-channel files by channel, otherwise clusters voice embeddings (or uses pyannote), then
matches voiceprints against the namespace's people.
- `clean` (off until Settings → Analysis → Clean up transcripts before analysis is on) tidies a transcript before
analysis reads it: [Cleaning up transcripts](#cleaning-up-transcripts).
- `analyze` finds entities, chapters, keywords and talk statistics; `summarize` (optional) calls any OpenAI-compatible
server for a summary, key points, action items (with who will do them), topics, people, tone and importance, each
key point and action item with the time of the line it comes from; `report` writes static HTML per asset and
per namespace, with word clouds.

Every step takes `--ns`, `--limit` and `--force`; `run` does them all, and a lock stops two runs overlapping.

### Cleaning up transcripts

Speech comes out of transcription as it was said, and the extractor, chapters, keywords and search all read it. With
`analysis.cleanup` on (Settings → Analysis → Clean up transcripts before analysis; off by default), the `clean` step,
which the standard pipeline runs just before `analyze`, tidies each line of a transcript with rules. It needs no model,
fetches nothing and sends nothing out:

- a word said three times or more in a row is kept once ("I I I looked" → "I looked"), and a doubled pronoun, article
or conjunction ("the the", "I, I") once; "had had", "that that" and "very very" stay;
- "um", "uh", "erm" and "hmm" go, with the comma after them (`analysis.cleanup_fillers`, Take out "um" and "uh"; on);
- one space between words, none before a comma or a full stop, one after a comma;
- "i" becomes "I", and each sentence starts with a capital. A line that carries on the one before (same speaker, no
full stop) keeps its lower case.

These are English rules ("um" is a word in German and Portuguese), so they apply to English lines: the line's or the
asset's language when transcription gave one, else when the text reads as English. In other languages only a word
said three times in a row is kept once, and sentences start with a capital. Documents and images keep their text as
it was read.

Every line it changes is in the asset's Change history (by "text cleanup"), with Revert. A line someone corrected,
or reverted, is never cleaned again, and running the step again on cleaned text changes nothing. Turn it on for a
custom pipeline by adding the Clean up step before Analyze (with Force it runs even while the setting is off); run it
on an asset with Reprocess → Clean up, `POST /api/v1/recordings/{id}/reprocess` with `{"steps": ["clean", "analyze"]}`,
or `lens clean`. Workers whose `workers.steps` list has `analyze` run it too.

## Importing text and transcripts {#importing-transcripts}

Formats: .txt, .md, .markdown, .mdx, .docx, .doc, .pdf, .srt, .vtt, .json (lens, Whisper or a list of segments) and
Expand Down Expand Up @@ -457,7 +484,8 @@ A pipeline is a named, versioned list of steps; each namespace can choose its de
analyze, summarize, report). Imports, watched folders and reprocessing use the namespace's pipeline; steps that need
audio skip themselves for imported text. Steps:

- `transcribe`, `diarize`, `analyze`, `summarize`, and `report` (the built-in report).
- `transcribe`, `diarize`, `clean` ([Cleaning up transcripts](#cleaning-up-transcripts)), `analyze`, `summarize`, and
`report` (the built-in report).
- `llm`: renders a prompt template, sends it to the configured model, checks the reply against the template's JSON
Schema and saves it as a named output (e.g. `meeting_notes`).
- `report` with a template: renders an HTML page into the namespace's reports. These pages may not run scripts.
Expand Down
15 changes: 12 additions & 3 deletions fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ def run_steps(db, cfg, which, ns=None, limit=0, force=False, recording=None, aud
log("transcribe:", ingest.transcribe_pending(db, cfg, ns, limit, force, log), "recording(s)")
if "diarize" in which:
log("diarize:", spk.diarize_pending(db, cfg, ns, limit, force, log), "recording(s)")
if "clean" in which:
from .domain import cleanup, settings

ccfg = settings.effective(db, cfg)
if (ccfg.get("analysis") or {}).get("cleanup") or force:
log("clean:", cleanup.clean_pending(db, ccfg, ns, limit, force, log), "asset(s)")
else:
log("clean: skipped; cleaning up transcripts is off (analysis.cleanup, Settings → Analysis)")
if "analyze" in which:
log("analyze:", analyze.analyze_pending(db, cfg, ns, limit, force, log), "asset(s)")
if "embed" in which:
Expand Down Expand Up @@ -51,10 +59,11 @@ def _main_base(argv=None):
("scan", "register new files in each namespace's folders"),
("transcribe", "transcribe new recordings"),
("diarize", "split speakers and match voice IDs within each namespace"),
("clean", "tidy transcripts before analysis: fillers, repeated words, spacing and capitals"),
("analyze", "named things, keywords, sections and speaking-time statistics"),
("embed", "index passages for search by meaning (with the configured embedding model)"),
("summarize", "optional LLM summaries"),
("run", "scan, transcribe, diarize, analyze, embed, summarize and report, resuming where it stopped"),
("run", "scan, transcribe, diarize, clean, analyze, embed, summarize and report, resuming where it stopped"),
):
p = sub.add_parser(name, help=helptext)
p.add_argument("--ns")
Expand Down Expand Up @@ -136,8 +145,8 @@ def _main_base(argv=None):
return
conn = store.connect(cfg, upgrade=a.cmd != "migrations") # so a failed upgrade can still be looked at
try:
if a.cmd in ("scan", "transcribe", "diarize", "analyze", "embed", "summarize", "run", "report"):
which = ["scan", "transcribe", "diarize", "analyze", "embed", "summarize", "report"] if a.cmd == "run" else [a.cmd]
if a.cmd in ("scan", "transcribe", "diarize", "clean", "analyze", "embed", "summarize", "run", "report"):
which = ["scan", "transcribe", "diarize", "clean", "analyze", "embed", "summarize", "report"] if a.cmd == "run" else [a.cmd]
with store.lock(cfg, "pipeline"):
run_steps(
conn,
Expand Down
202 changes: 202 additions & 0 deletions fastapi_backend/app/domain/cleanup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
"""Cleaning up a transcript before analysis: the clean step (Settings → Analysis → Clean up transcripts).

Speech comes out of transcription as it was said: "um" and "uh", stutters ("I I I looked"), a lower-case "i", stray
spaces before commas, sentences that start in lower case. The extractor, the chapters, the keywords and search all
read that text, so the clean step tidies each line first, with rules: no model, nothing to fetch, nothing sent out.

- repeated words: a word said three times or more in a row is kept once, and a doubled pronoun, article or
conjunction ("the the", "I, I") once. "had had", "that that" and "very very" stay as they are.
- fillers (`analysis.cleanup_fillers`, on with cleanup): "um", "uh", "erm" and "hmm" go, with the comma after them.
- spacing: one space between words, none before a comma or a full stop, one after a comma.
- capitals: "i" is "I", and each sentence starts with a capital (a line that carries on the one before doesn't).

The rules are English ones ("um" is a word in German and Portuguese), so they run on English lines: the line's or the
asset's language when transcription gave one, else when the text reads as English. Other languages only get a word
said three times or more in a row kept once, and their sentences capitalised. A document's or an image's text is kept
as it was read.

Every line it changes is kept in the asset's edit history, by "text cleanup": Change history lists them and Revert
puts a line back. A line someone corrected is never cleaned, since people's corrections outrank the rules, and running
the rules again on what they made changes nothing.
"""

from __future__ import annotations

import re

from . import store

R = store.R
CLEANER = "text cleanup" # who the edit history says changed a line
SPEECH = ("audio", "transcript") # documents and images keep their text as read
FILLERS = {"um", "umm", "uh", "uhh", "uhm", "erm", "er", "hmm"}
# said twice in a row these are nearly always a stumble; had, that, is, very and so are often doubled on purpose
# ("what it is is", "very very good")
DOUBLED = set(
"""i i'm i've i'll i'd a an the and but or to of in on at for with we you he she it they my our your their this
if it's we're you're they're there's""".split()
)
MARKERS = {"okay", "ok", "yeah", "yes", "no", "well", "so", "right", "oh", "alright", "anyway", "actually", "hey", "hi", "sorry"}
ENGLISH = set(
"""the and to of a in is it you that i was for on are with as be this have not but they at we what so or if can
all there my your do just about he she one would like get will an by me from how know think yeah okay""".split()
)
ABBREV = {"mr", "mrs", "ms", "dr", "prof", "st", "vs", "etc", "e.g", "i.e", "approx", "no", "fig", "inc", "ltd", "jr", "sr"}
WORD = re.compile(r"[^\W\d_][\w'’]*")
SENTENCE = re.compile(r"(?<![.…])([.!?])(\s+)([^\W\d_])")
STRAY = re.compile(r"\s+([,.;!?])(?=\s|$)")
AFTER_COMMA = re.compile(r"([,;])(?=[^\W\d_])")
I_ALONE = re.compile(r"(?<![\w'’-])i(?=$|[\s,.;:!?]|['’](?:m|ve|ll|d)\b)")
OWN_END = re.compile(r"(?<![.…])([.!?])$") # a token's own full stop, question or exclamation mark (not "um...")
FIRST = re.compile(r"[\"“'‘(]*([^\W\d_])")
LINK = re.compile(r"[\w.+-]+@|https?:|www\.")


def _bare(token):
return token.strip(",.;:!?-—–…\"“”'‘’()").lower().replace("’", "'")


def english(lang, texts):
"""Whether the English rules apply: to the language transcription gave, else to text that reads as English."""
if lang and str(lang).lower() not in ("auto", "nospeech", "unknown", "none"):
return str(lang).lower().startswith("en")
ws = [w.lower() for t in texts for w in WORD.findall(t)][:3000]
return len(ws) >= 3 and sum(w in ENGLISH for w in ws) / len(ws) >= 0.2


def _repeats(tokens, doubled):
"""A word three times or more in a row once; twice in a row once too when it's one of `doubled`."""
out, k = list(tokens), 0
while k < len(out):
w = _bare(out[k])
j = k
while w and j + 1 < len(out) and _bare(out[j + 1]) == w:
j += 1
if j - k >= 2 or (j - k == 1 and w in doubled):
last = out[j]
if out[k][:1].isupper() and not last[:1].isupper(): # "I i think", "The the"
last = last[:1].upper() + last[1:]
out[k : j + 1] = [last]
k += 1
return out


def _is_filler(token):
w = _bare(token)
core = token.strip(",.;:!?-—–…")
return w in FILLERS and not (core.isupper() and len(core) > 1) and re.fullmatch(r"[\w'’]+", core) is not None


def _fillers(tokens):
"""Without "um" and "uh": a sentence's end the filler carried moves to the word before, and a comma before it goes
with it unless it follows a word like "okay" or "well"."""
out, k = list(tokens), 0
while k < len(out):
if not _is_filler(out[k]):
k += 1
continue
t = out.pop(k)
if not k:
continue
prev, m = out[k - 1], OWN_END.search(t)
if m and prev[-1] not in ".!?":
out[k - 1] = prev.rstrip(",;:") + m.group(1)
elif t.endswith(",") and prev.endswith(",") and _bare(prev) not in MARKERS: # "was, uh, there"; "Okay, um, so"
out[k - 1] = prev[:-1]
return out


def _sentences(text):
"""Each sentence starting with a capital, past abbreviations ("Dr. Smith", "e.g. this")."""

def up(m):
before = text[: m.start()].rsplit(None, 1)
word = before[-1].lower().rstrip(".") if before else ""
if m.group(1) == "." and (word in ABBREV or len(word) == 1):
return m.group(0)
return m.group(1) + m.group(2) + m.group(3).upper()

return SENTENCE.sub(up, text)


def clean_line(text, english_rules=True, fillers=True, starts_sentence=True):
"""A transcript line as the rules leave it. `starts_sentence`: the line before ended a sentence (or there's none)."""
tokens = text.split()
if english_rules:
if fillers:
tokens = _fillers(tokens)
tokens = _repeats(tokens, DOUBLED)
else:
tokens = _repeats(tokens, ())
out = " ".join(tokens)
if english_rules:
out = AFTER_COMMA.sub(r"\1 ", STRAY.sub(r"\1", out))
out = I_ALONE.sub("I", out)
out = _sentences(out)
m = FIRST.match(out) # the line's first word, past an opening quote or bracket
if starts_sentence and m and m.group(1).islower() and not LINK.match(out[m.start(1) :]):
out = out[: m.start(1)] + m.group(1).upper() + out[m.end(1) :]
return out.strip()


def plan(db, cfg, rid):
"""[(segment id, idx, text, cleaned)] for the lines the rules change; [] for a document or an image."""
rec = db.one("SELECT source, language FROM $r", r=R("recording", rid)) or {}
if rec.get("source") not in SPEECH:
return []
segs = db.rows("SELECT record::id(id) AS id, idx, text, lang, speaker FROM segment WHERE recording = $r ORDER BY idx", r=rid)
people = set(db.values("SELECT VALUE idx FROM segment_edit WHERE recording = $r AND by != $c AND after.text != NONE", r=rid, c=CLEANER))
fillers = bool((cfg.get("analysis") or {}).get("cleanup_fillers", True))
whole = english(rec.get("language"), [s.get("text") or "" for s in segs[:400]])
out, prev = [], None
for s in segs:
text = s.get("text") or ""
ended = prev is None or prev["speaker"] != s.get("speaker") or re.search(r"[.!?…]['\"”’)]*$", prev["text"]) is not None
prev = {"speaker": s.get("speaker"), "text": text}
if s["idx"] in people or not text.strip():
continue
lang = s.get("lang")
en = english(lang, [text]) if lang and lang not in ("auto", "nospeech") else whole
new = clean_line(text, en, fillers, ended)
if new and new != text:
out.append((s["id"], s["idx"], text, new))
prev["text"] = new
return out


def clean_recording(db, cfg, rid):
"""Clean an asset's transcript lines; how many changed. Each change goes into its edit history."""
changes = plan(db, cfg, rid)
if not changes:
return 0
last = int(db.values("UPSERT $r SET n += $k RETURN VALUE n", r=R("seq", "segment_edit"), k=len(changes))[0])
t, first = store.now(), last - len(changes) + 1
ups = [{"id": R("segment", sid), "text": new} for sid, _, _, new in changes]
edits = [
{"recording": rid, "idx": idx, "before": {"text": old}, "after": {"text": new}, "by": CLEANER, "at": t, "n": first + k}
for k, (_, idx, old, new) in enumerate(changes)
]
for a in range(0, len(ups), 500):
db.run(
["FOR $u IN $ups { UPDATE $u.id SET text = $u.text; }", "INSERT INTO segment_edit $edits"],
ups=ups[a : a + 500],
edits=edits[a : a + 500],
)
return len(changes)


def clean_pending(db, cfg, ns=None, limit=0, force=False, log=print):
"""The command line's clean: transcripts not analysed yet (all of them with force)."""
where = "status IN ['transcribed', 'diarized', 'analyzed']" if force else "status IN ['transcribed', 'diarized']"
where += " AND source IN $src"
if ns:
where += " AND space = $s"
rows = db.rows(
f"SELECT record::id(id) AS id, title FROM recording WHERE {where} ORDER BY id",
src=list(SPEECH),
s=store.ns_id(db, ns, create=False) if ns else None,
)
rows = rows[: limit or None]
for r in rows:
log(f" {r['title']}: {clean_recording(db, cfg, r['id'])} line(s) cleaned")
return len(rows)
Loading
Loading