Skip to content

feat(analysis): clean up transcripts before entity extraction - #476

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-hyaa6h
Oct 17, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-hyaa6h

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: a transcript went from transcription straight into analysis, stutters, "um"s and stray spaces included, so the entity extractor and keywords read "I I I looked", "Dyno Therapeutics" after "and i think", and "segmentation ," as said.

After: with Settings → Analysis → Clean up transcripts before analysis on (off by default), a new Clean up step runs just before Analyze. On English lines it keeps one of a word said three times in a row ("I I I looked" → "I looked"), one of a doubled pronoun or article ("the the" → "the"; "very very" and "had had" stay), takes out "um" and "uh" (a second switch keeps them), and fixes spacing before commas, a lone "i" and sentence capitals. Other languages only get the three-in-a-row rule and capitals ("um" is German for "at"). Every changed line is in the asset's Change history with Revert, labelled "text cleanup"; lines someone corrected are left alone, and running it again changes nothing. Documents and images keep their text as read.

How: cleanup.py holds the rules and writes each change as a segment_edit row (the history people's own edits use), in batches of 500 lines. clean joins jobs.STEPS, the standard pipeline, the import pipeline, the pipeline editor (Clean up, after Transcribe/Diarize, before Analyze), lens clean and Reprocess. Workers with an older workers.steps list that can analyze can clean.

Motivation and Context

Adeel pasted a punctuation and grammar script and asked for a text-cleaning step before entity extraction and transcript processing. This is the first of a short series: punctuation put back by a local model, grammar from a LanguageTool server, spaCy for named entities and date extraction follow as their own PRs. The script's alternating "Speaker 1 / Speaker 2" labels aren't carried over, because Lens tells voices apart by diarization or keeps a transcript's own labels.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: a cleaning step before entity extraction, for transcripts; merged on green CI.
  • Agents: the settings are in PUT /api/v1/settings/analysis (cleanup, cleanup_fillers) and the setup assistant's change_settings, the step in pipelines and POST /api/v1/recordings/<id>/reprocess, and the changes in the edit history API.
  • Zero setup: rules only, no model or package.
  • Fewest clicks: one switch; the fillers switch shows only when it's on.
  • Configurable: both settings in Settings → Analysis, with defaults (off; fillers on).
  • Light: pure Python rules, nothing fetched.
  • Private and secure: runs locally on text already stored; no new surface.
  • Cost: no model calls.
  • User decides: off by default; people's corrections outrank the rules, and every change has Revert.
  • Upgrades: no stored-data shape change; existing pipelines keep their steps (only the standard and import pipelines gain Clean up, which skips while off).
  • Honest: docs describe what the rules do and don't (English-only fillers).
  • Tests (tests/domain/test_cleanup.py, frontend pipeline and job tests), docs/processing.md, CHANGELOG and DEVLOG updated; OpenAPI unchanged (make openapi).
  • Open standards; no new dependencies.
  • Reported in the project thread; refine-later items in DEVLOG (revert a whole cleanup at once, other languages' fillers, a per-namespace switch).

Local CI (ci/local-ci.sh): backend (embedded and server SurrealDB), Next.js and pre-commit passed.


Generated by Claude Code

A new clean step tidies a transcript's lines with rules before analyze reads them: a word said
three times or more in a row once ("I I I looked" -> "I looked"), doubled pronouns and articles
once, "um" and "uh" taken out, spacing around commas, "i" -> "I" and sentence capitals. English
rules run on English lines only. The standard pipeline runs it before analyze, and it does
nothing until Settings -> Analysis -> Clean up transcripts before analysis is on.

Each changed line goes into the asset's edit history (by "text cleanup"), so Change history
lists it with Revert; a line someone corrected is never cleaned, and a second run changes
nothing. Reprocess, the pipeline editor, the canvas, the API and `lens clean` offer the step,
and workers whose steps list has analyze run it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants