Repository navigation
feat(analysis): clean up transcripts before entity extraction - #476
Merged
Merged
Conversation
A new clean step tidies a transcript's lines with rules before analyze reads them: a word said
three times or more in a row once ("I I I looked" -> "I looked"), doubled pronouns and articles
once, "um" and "uh" taken out, spacing around commas, "i" -> "I" and sentence capitals. English
rules run on English lines only. The standard pipeline runs it before analyze, and it does
nothing until Settings -> Analysis -> Clean up transcripts before analysis is on.
Each changed line goes into the asset's edit history (by "text cleanup"), so Change history
lists it with Revert; a line someone corrected is never cleaned, and a second run changes
nothing. Reprocess, the pipeline editor, the canvas, the API and `lens clean` offer the step,
and workers whose steps list has analyze run it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: a transcript went from transcription straight into analysis, stutters, "um"s and stray spaces included, so the entity extractor and keywords read "I I I looked", "Dyno Therapeutics" after "and i think", and "segmentation ," as said.
After: with Settings → Analysis → Clean up transcripts before analysis on (off by default), a new Clean up step runs just before Analyze. On English lines it keeps one of a word said three times in a row ("I I I looked" → "I looked"), one of a doubled pronoun or article ("the the" → "the"; "very very" and "had had" stay), takes out "um" and "uh" (a second switch keeps them), and fixes spacing before commas, a lone "i" and sentence capitals. Other languages only get the three-in-a-row rule and capitals ("um" is German for "at"). Every changed line is in the asset's Change history with Revert, labelled "text cleanup"; lines someone corrected are left alone, and running it again changes nothing. Documents and images keep their text as read.
How:
cleanup.pyholds the rules and writes each change as asegment_editrow (the history people's own edits use), in batches of 500 lines.cleanjoinsjobs.STEPS, the standard pipeline, the import pipeline, the pipeline editor (Clean up, after Transcribe/Diarize, before Analyze),lens cleanand Reprocess. Workers with an olderworkers.stepslist that can analyze can clean.Motivation and Context
Adeel pasted a punctuation and grammar script and asked for a text-cleaning step before entity extraction and transcript processing. This is the first of a short series: punctuation put back by a local model, grammar from a LanguageTool server, spaCy for named entities and date extraction follow as their own PRs. The script's alternating "Speaker 1 / Speaker 2" labels aren't carried over, because Lens tells voices apart by diarization or keeps a transcript's own labels.
Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md.Done questions (AGENTS.md)
PUT /api/v1/settings/analysis(cleanup,cleanup_fillers) and the setup assistant'schange_settings, the step in pipelines andPOST /api/v1/recordings/<id>/reprocess, and the changes in the edit history API.tests/domain/test_cleanup.py, frontend pipeline and job tests),docs/processing.md, CHANGELOG and DEVLOG updated; OpenAPI unchanged (make openapi).Local CI (
ci/local-ci.sh): backend (embedded and server SurrealDB), Next.js and pre-commit passed.Generated by Claude Code