Repository navigation
feat(analysis): put punctuation back in transcripts that lack it - #485
Merged
Merged
Conversation
The clean step can now restore full stops, commas and question marks with a local model, kredor/punctuate-all (MIT, 12 languages), the one the pasted punctuation script used. Settings -> Analysis -> Put punctuation back: off (the default), where a transcript has hardly any (imported captions), or in every transcript. Marks a transcript already has are kept, a mark the model is less than 70% sure of isn't added, and each line is given marks once. The model runs on Transformers' token-classification pipeline directly, since deepmultilingualpunctuation no longer loads with Transformers 5. The model, Transformers and PyTorch's CPU build (about 2 GB) are fetched the first time a clean step needs them (the punctuation extra and component); until then the rules run alone and the job's log says why. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: a transcript with no punctuation (imported captions, some speech models) reached analysis as one long run of words, so sentences, questions and the extractor's sentence-based rules had nothing to go on.
After: with transcripts cleaned up, Settings → Analysis → Put punctuation back (off by default) adds full stops, commas and question marks with a local model, kredor/punctuate-all (MIT, 12 languages), the one the pasted script used. "Where it's missing" reads only transcripts with hardly any marks; "Always" reads every one. Marks a transcript has are kept, a mark the model is less than 70% sure of isn't added, and each line is given marks once. Every change is in Change history with Revert, like the rules' changes.
How:
cleanup._punctuationruns Transformers' token-classification pipeline directly on overlapping chunks of words (deepmultilingualpunctuation no longer loads with Transformers 5), maps the marks back to lines, and the rules run after it. The model (pinned revision), Transformers and PyTorch's CPU build (about 2 GB) are the newpunctuationextra and component, fetched the first time a clean step needs them; until then the rules run alone and the job's log says why. Lines given marks carrypunctuatedso they aren't read again.Motivation and Context
Second of the series from Adeel's pasted punctuation and grammar script (cleanup merged in #476). spaCy entities, dates and grammar follow as their own PRs.
Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md.Done questions (AGENTS.md)
PUT /api/v1/settings/analysis(cleanup_punctuation), the setup assistant'schange_settings, the components API to fetch it, and the edit history API for the changes.segment.punctuatedis a new optional field on a schemaless table; lines without it simply haven't been read yet. No upgrade step needed.tests/domain/test_cleanup.py(chunking, missing/always, fallback without the model, component needed only when picked);docs/processing.md,docs/components.md, CHANGELOG updated; OpenAPI unchanged.Local CI (
ci/local-ci.sh): backend (embedded and server SurrealDB), Next.js, pre-commit and title passed.Generated by Claude Code