Skip to content

feat(enhance): clean up low quality sound before it's transcribed - #596

Closed
adeelahmad wants to merge 2 commits into
mainfrom
claude/project-thread-hnzflg
Closed

adeelahmad wants to merge 2 commits into
mainfrom
claude/project-thread-hnzflg

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Description

Before: the enhance step only measured and cleaned up images; recordings were transcribed straight from the uploaded audio, so a noisy, quiet or clipped recording transcribed badly.

After: the enhance step now listens to audio and video too. A low quality recording gets a cleaned-up copy (declipped, rumble and steady noise taken out, its level evened), and transcription hears that copy, kept in step with the original sample for sample so word timings still land where they were said. The original, its waveform and voice matching stay as they were. Off by default (Settings → Enhance). ffmpeg's filters are the default engine; WebRTC's noise suppression and gain (webrtc-noise-gain, Apache-2.0, fetched on first use) is an optional engine. Enhance and transcribe again from a recording's Details tab, the API (POST /api/v1/resources/{rid}/enhance), the MCP enhance tool, or the enhance automation action.

This folds in the previously parked "WebRTC audio cleanup node" idea as the webrtc engine.

Motivation and Context

Low quality sound is the biggest cause of bad transcripts, and the agent reads transcripts. Cleaning the audio before transcription raises quality at no cost to installs that leave it off. Follow-up to #479 (image enhancement), which built the enhance step and its settings surface.

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote a changelog.d/ fragment (the new scheme replacing ## Unreleased).

Done questions (AGENTS.md)

  • Goal: a low quality recording transcribes from a cleaned copy; delivered end to end.
  • Agents: the API route, the MCP enhance tool and the enhance automation action all enhance audio.
  • Zero setup: works after the install line; ffmpeg is already present, webrtc is fetched only if chosen.
  • Fewest clicks: one toggle; the Details tab re-enhances and re-transcribes in one action.
  • Configurable: enhance.audio_engine and thresholds in Settings → Enhance, with defaults, out of the wizard.
  • Light: off by default; the default (ffmpeg) adds nothing heavy; webrtc is an opt-in extra.
  • Private and secure: the copy is written encrypted beside the files; a new dependency row is in the threat model.
  • Cost: no model call; local signal processing only.
  • User decides: off by default, existing defaults unchanged.
  • Upgrades: no stored-data shape change; the enhanced copy is a derived rendition, cleaned up on deletion.
  • Honest: README and docs label it off by default; changelog fragment states what is built.
  • Tests, docs and generated files updated (test_enhance_audio.py, docs/enhance.md, make openapi).
  • Open standards; webrtc-noise-gain is Apache-2.0, an opt-in extra, so Lens stays MIT.
  • Result reported with evidence; tests green locally.
  • Reuse: reuses the enhance step, ffmpeg and the existing rendition/deletion machinery.

Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Closing as superseded by #569, which landed on main as the clean_audio pipeline step (WebRTC noise suppression, gain, high-pass, echo cancellation and voice detection before transcription, original untouched, off by default). That covers the same "clean up low quality sound before it's transcribed" need, so this PR would be a second parallel cleanup step, which AGENTS.md (reuse first) rules out.

The only behaviour here that clean_audio doesn't have: gating the cleanup on a measured quality check, and keeping the cleaned audio as a stored rendition. #569 deliberately keeps no copy and cleans whenever the step is added. If those additions are wanted they belong on top of clean_audio, not as a rival step. The image half of this work shipped separately in #479.


Generated by Claude Code

@adeelahmad adeelahmad closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants