Skip to content

feat(analysis): put punctuation back in transcripts that lack it - #485

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-hyaa6h
Oct 17, 2026
Merged

adeelahmad merged 1 commit into
mainfrom
claude/project-thread-hyaa6h

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: a transcript with no punctuation (imported captions, some speech models) reached analysis as one long run of words, so sentences, questions and the extractor's sentence-based rules had nothing to go on.

After: with transcripts cleaned up, Settings → Analysis → Put punctuation back (off by default) adds full stops, commas and question marks with a local model, kredor/punctuate-all (MIT, 12 languages), the one the pasted script used. "Where it's missing" reads only transcripts with hardly any marks; "Always" reads every one. Marks a transcript has are kept, a mark the model is less than 70% sure of isn't added, and each line is given marks once. Every change is in Change history with Revert, like the rules' changes.

How: cleanup._punctuation runs Transformers' token-classification pipeline directly on overlapping chunks of words (deepmultilingualpunctuation no longer loads with Transformers 5), maps the marks back to lines, and the rules run after it. The model (pinned revision), Transformers and PyTorch's CPU build (about 2 GB) are the new punctuation extra and component, fetched the first time a clean step needs them; until then the rules run alone and the job's log says why. Lines given marks carry punctuated so they aren't read again.

Motivation and Context

Second of the series from Adeel's pasted punctuation and grammar script (cleanup merged in #476). spaCy entities, dates and grammar follow as their own PRs.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: punctuation put back before entity extraction, as the script did; merged on green CI.
  • Agents: PUT /api/v1/settings/analysis (cleanup_punctuation), the setup assistant's change_settings, the components API to fetch it, and the edit history API for the changes.
  • Zero setup: fetched on first use; nothing to install by hand.
  • Fewest clicks: one select under the cleanup switch.
  • Configurable: Settings → Analysis, default off.
  • Light: off by default; the 2 GB model and PyTorch are fetched only when picked, and the rules run without them.
  • Private and secure: runs locally; the model comes from Hugging Face at a pinned revision with no remote code (threat-model row added).
  • Cost: local model, no LLM calls.
  • User decides: off by default; existing marks and people's corrections are kept; every change has Revert.
  • Upgrades: segment.punctuated is a new optional field on a schemaless table; lines without it simply haven't been read yet. No upgrade step needed.
  • Honest: docs say what it adds and that it's a model's judgement.
  • Tests in tests/domain/test_cleanup.py (chunking, missing/always, fallback without the model, component needed only when picked); docs/processing.md, docs/components.md, CHANGELOG updated; OpenAPI unchanged.
  • Open: MIT model; Transformers (Apache-2.0) and PyTorch (BSD) as an opt-in extra.
  • Reported in the project thread; refine later: a GPU build when one's there, per-namespace switch.
  • Reuse: the punctuation model the script used, on Transformers' own pipeline; the clean step's edit history and the components fetcher Lens already has.

Local CI (ci/local-ci.sh): backend (embedded and server SurrealDB), Next.js, pre-commit and title passed.


Generated by Claude Code

The clean step can now restore full stops, commas and question marks with a local model,
kredor/punctuate-all (MIT, 12 languages), the one the pasted punctuation script used. Settings
-> Analysis -> Put punctuation back: off (the default), where a transcript has hardly any
(imported captions), or in every transcript. Marks a transcript already has are kept, a mark
the model is less than 70% sure of isn't added, and each line is given marks once. The
model runs on Transformers' token-classification pipeline directly, since
deepmultilingualpunctuation no longer loads with Transformers 5.

The model, Transformers and PyTorch's CPU build (about 2 GB) are fetched the first time a
clean step needs them (the punctuation extra and component); until then the rules run alone
and the job's log says why.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
@adeelahmad adeelahmad self-assigned this Oct 17, 2026
@adeelahmad
adeelahmad merged commit 1dfcdda into main Oct 17, 2026
18 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-hyaa6h branch October 17, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants