Skip to content

feat(pipelines): clean up a recording's sound before it's transcribed - #569

Merged
adeelahmad merged 13 commits into
mainfrom
claude/project-thread-zrzew2
Oct 18, 2026
Merged

adeelahmad merged 13 commits into
mainfrom
claude/project-thread-zrzew2

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Before: speech-to-text heard a recording exactly as it was: fans, hiss, a quiet speaker, the far side of a call echoing back, and long silences that cloud engines bill for and Whisper imagines words in.

After: a new pipeline step, Clean up audio (clean_audio), runs a recording through WebRTC's audio processing with pywebrtc-audio (Apache-2.0, ~0.5 MB, numpy only, wheels for Linux x86_64/aarch64 and macOS): noise suppression (off/6/12/18/21 dB), automatic gain, a high-pass filter, echo cancellation for two-channel call recordings, and pauses of 2 s or longer left out of what's transcribed. The transcript's times (and word times) are put back on the recording's own timeline, so the player and citations stay right. No pipeline has the step until someone adds it, so defaults are unchanged.

How: the step measures speech with WebRTC's voice detection and keeps its options and what it found on the recording (audio_cleanup, also in GET /api/v1/recordings/{id}). transcribe_one then hears the cleaned audio, cut to its speech spans, and maps segment times back through the joins. The file is never changed and no copy is stored (cleaning is ~10 µs per 10 ms frame, so it's redone each time). Speaker separation and voice IDs still use the original. Options live in Settings → Transcription → Audio cleanup (transcribe.cleanup, validated server side). The library is an extra (webrtc) fetched on first use by a component, or inline by the step; a worker that can't get it transcribes the original and says so. The step is in the pipeline editor, canvas, Reprocess and batch Run on. It's named clean_audio to stay apart from #479's image enhance step.

Checked by hand on a real speech sample (Open Speech Repository): voice detection tracked the sentences; on synthetic tests echo cancellation took the far side down by >20x and the trimmed audio was 10 s of 22 s.

Done questions

  1. Goal: Yes: the node the thread asked for (noise suppression, gain, echo cancellation, voice detection that trims silence, before transcription, defaults unchanged). Merged on green.
  2. Agents: Yes: steps via the pipelines and reprocess APIs ({"steps": ["clean_audio", "transcribe"]}), results in audio_cleanup on the recording, settings via PUT /settings/transcribe (test_an_agent_reads_what_the_step_did).
  3. Zero setup: Yes: the 0.5 MB package is fetched the first time the step runs.
  4. Fewest clicks: Yes: one step added in the editor or one tick in Reprocess; every option has a default.
  5. Configurable: Yes: Settings → Transcription → Audio cleanup, not in the wizard.
  6. Light: Yes: nothing fetched until a pipeline runs the step; aarch64 wheels for a Pi; no model.
  7. Private and secure: Yes: all local DSP on decoded PCM, nothing sent out, no stored copy. No new attack surface (the library sees float arrays from ffmpeg, fetched at the uv.lock version like other extras), so the threat model is unchanged.
  8. Cost: Yes: no model calls; trimming silence lowers cloud transcription seconds, which the ledger already records from what's sent.
  9. User decides: Yes: off until a pipeline has the step; existing pipelines and defaults unchanged.
  10. Upgrades: n/a: a new optional field on recordings, nothing existing changes shape; old workers.steps lists with transcribe run the step too.
  11. Honest: Yes: docs describe only what's built; README lists it in Understand it with its checks.
  12. Tested and documented: Yes: tests/domain/test_audio_cleanup.py (12 tests), the Reprocess jest test, docs/processing.md#cleaning-up-audio, components table, CHANGELOG, docs/requirements.md; client regenerated with make openapi. Full backend suite: 1654 passed locally.
  13. Open: Yes: Apache-2.0 dependency, WebRTC's BSD code.
  14. Reported: Yes, in the thread. Refine later: per-step option overrides in the pipeline editor, listening to the cleaned audio in the player, using the cleaned audio (or the speech map) for diarization, other sample rates for echo delay hints.
  15. Reuse: Yes: WebRTC's own audio processing through an existing binding; Lens's decode, components fetching and pipeline step patterns.

Generated by Claude Code

A new pipeline step, enhance (Clean up audio), runs a recording through
WebRTC's audio processing with pywebrtc-audio (Apache-2.0, ~0.5 MB, fetched
the first time the step runs): noise suppression, automatic gain, a
high-pass filter, echo cancellation for two-channel call recordings, and
voice detection that leaves pauses out of what's transcribed. The
transcript's times are mapped back onto the recording's own timeline.

The step keeps its options and what it measured on the recording
(audio_cleanup); transcribing then hears the cleaned audio. The file is
never changed and no copy is stored. No pipeline has the step until one
adds it, so defaults are unchanged. Settings > Transcription > Audio
cleanup holds the options (transcribe.cleanup).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
…enhance step in #479

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
claude added 10 commits October 18, 2026 07:40
…-zrzew2

# Conflicts:
#	CHANGELOG.md
#	fastapi_backend/app/domain/pipelines.py
#	nextjs-frontend/components/pipelines/pipeline-canvas.tsx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014WcBxU5ARBoLWMYZU46St6
@adeelahmad
adeelahmad merged commit 173d5df into main Oct 18, 2026
20 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-zrzew2 branch October 18, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants