Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions changelog.d/audio-quality-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Clean up only low quality recordings.** The `clean_audio` step can now measure each recording's signal-to-noise
and clipping first and leave a clean-sounding one as it is, cleaning only the noisy, quiet or clipped ones. Off by
default (every recording is still cleaned); turn on "Clean only low quality recordings" under Settings →
Transcription → Audio cleanup and set the signal-to-noise floor. What it measured (`snr_db`, `clip_pct`) is on the
recording in `GET /api/v1/recordings/{id}` ([Cleaning up audio](docs/processing.md#cleaning-up-audio)).
10 changes: 9 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,21 @@ until then. What it does is in Settings → Transcription → Audio cleanup (`tr
| `high_pass` | on | Takes out the rumble below speech: traffic, handling noise, a knocked desk |
| `echo` | `off` | Echo cancellation for a two-channel recording with what the speakers played on one channel (`left` or `right`) and the microphone on the other, as call recorders keep them; the far side's voice is taken out of the microphone, then both sides are mixed |
| `trim_silence` | on | Pauses of `min_silence_ms` (2000) or longer, by WebRTC's voice detection, are left out of what's transcribed |
| `only_if_poor` | off | Measure each recording and clean only the low quality ones; a clean-sounding recording is left as it is and transcribed from the original |
| `snr_floor_db` | 20 | With `only_if_poor`, a recording at or above this signal-to-noise (in dB) counts as clean; below it, or if it clips, it's cleaned |

With `only_if_poor` off (the default) every recording the step runs on is cleaned. Turned on, the step first reads the
recording's signal-to-noise and clipping with a quick measurement (numpy only, no model) and leaves a clean-sounding
one untouched, so a pipeline can carry the step for everything while only the noisy, quiet or clipped recordings are
actually cleaned. The measured `snr_db` and `clip_pct` are kept on the asset either way; a skipped recording keeps no
other cleanup options.

Leaving silence out means a cloud engine bills fewer minutes (the activity ledger's seconds are what was sent), a
local one finishes sooner and Whisper has no quiet stretch to imagine words in. The transcript's times are put back on
the recording's own timeline, so the player and every citation stay where the words were said.

The step measures how much of the recording is speech and keeps what it ran with on the asset (`audio_cleanup` in
`GET /api/v1/recordings/{id}`: the options, `speech_ms`, `kept_ms`, the gain it ended on). From then on transcribing
`GET /api/v1/recordings/{id}`: the options, `speech_ms`, `kept_ms`, `snr_db`, `clip_pct`, the gain it ended on). From then on transcribing
that asset hears it cleaned up that way, also on a later Reprocess. The file is never changed and no second copy is
stored: cleaning takes a fraction of a second per minute of audio, so it's done again each time. Speaker separation and
voice IDs still listen to the original. Run the step with every option off to have the asset transcribed as it is
Expand Down
2 changes: 1 addition & 1 deletion docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ request, to another check that proves the same thing; loosening a requirement or
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |
| Or by Moondream on this server, off by default, fetched at a pinned revision only where the chosen model can run, and saying why where it can't | [test_moondream.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_moondream.py) `test_moondream_describes_pictures_without_a_model_provider`, `test_where_each_moondream_can_run`, `test_on_a_cpu_moondream_3_is_never_fetched_and_the_step_says_why`, `test_moondream_is_fetched_pinned`, `test_admins_and_agents_choose_moondream` |
| A recording's sound cleaned up before transcription in a pipeline that has the step (noise, gain, echo), with silence left out of what's transcribed and the transcript's times kept on the recording's timeline | [test_audio_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_audio_cleanup.py) `test_speech_is_found_and_silence_left_out_of_transcription`, `test_echo_of_the_far_side_is_taken_out`, `test_without_the_step_or_with_every_option_off_it_is_transcribed_as_it_is`, `test_an_agent_reads_what_the_step_did` |
| A recording's sound cleaned up before transcription in a pipeline that has the step (noise, gain, echo), with silence left out of what's transcribed and the transcript's times kept on the recording's timeline, optionally only for low quality recordings | [test_audio_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_audio_cleanup.py) `test_speech_is_found_and_silence_left_out_of_transcription`, `test_echo_of_the_far_side_is_taken_out`, `test_without_the_step_or_with_every_option_off_it_is_transcribed_as_it_is`, `test_an_agent_reads_what_the_step_did`, `test_quality_gate_leaves_a_clean_recording_as_it_is`, `test_quality_gate_still_cleans_a_low_quality_recording` |
| Transcripts cleaned up before analysis (stutters, "um"s, spacing), off by default, each change in the edit history with Revert and people's corrections left alone | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_off_by_default_changes_nothing`, `test_clean_step_tidies_lines_and_keeps_them_in_the_history`, `test_people_s_corrections_outrank_the_rules` |
| Punctuation put back in transcripts that lack it by a local model, fetched only when picked | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_punctuation_put_back_where_it_is_missing`, `test_without_the_model_the_rules_run_alone`, `test_the_model_is_fetched_only_when_it_is_asked_for` |
| Grammar fixed by a LanguageTool server you run, when its address is set: grammar fixes only, never spelling or a word with a capital | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_grammar_fixes_from_languagetool_leave_names_and_spelling`, `test_without_its_server_the_rules_run_alone`, `test_the_grammar_server_is_an_address_without_a_password` |
Expand Down
56 changes: 54 additions & 2 deletions fastapi_backend/app/domain/audio_cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
from __future__ import annotations

import importlib.metadata
import math

import numpy as np

Expand All @@ -39,8 +40,12 @@
NOISE = {"off": None, "low": 0, "moderate": 1, "high": 2, "very_high": 3} # WebRTC's levels: 6, 12, 18 and 21 dB
ECHO = ("off", "left", "right") # which channel holds what the speakers played
DEFAULTS = store.DEFAULTS["transcribe"]["cleanup"]
KEYS = ("noise", "gain", "high_pass", "echo", "trim_silence", "min_silence_ms")
KEYS = ("noise", "gain", "high_pass", "echo", "trim_silence", "min_silence_ms", "only_if_poor", "snr_floor_db")
PACKAGE = "pywebrtc-audio"
QUIET_PCT = 10 # the loudness at this percentile of the 10 ms frames is taken as the noise floor
LOUD_PCT = 95 # the loudness at this percentile is taken as the speech level
CLIP = 0.99 # a sample at least this loud (of 1.0) counts as clipped
CLIP_MAX = 0.5 # more than this percent of samples clipped is poor quality worth cleaning


def check(value):
Expand All @@ -55,12 +60,15 @@ def check(value):
raise ValueError(f"transcribe.cleanup.noise is one of: {', '.join(NOISE)}")
if out["echo"] not in ECHO:
raise ValueError(f"transcribe.cleanup.echo is one of: {', '.join(ECHO)}")
for k in ("gain", "high_pass", "trim_silence"):
for k in ("gain", "high_pass", "trim_silence", "only_if_poor"):
if not isinstance(out[k], bool):
raise ValueError(f"transcribe.cleanup.{k} is true or false")
ms = out["min_silence_ms"]
if not (isinstance(ms, int) and not isinstance(ms, bool) and 500 <= ms <= 60000):
raise ValueError("transcribe.cleanup.min_silence_ms is a whole number of milliseconds from 500 to 60000")
snr = out["snr_floor_db"]
if not (isinstance(snr, (int, float)) and not isinstance(snr, bool) and 0 <= snr <= 60):
raise ValueError("transcribe.cleanup.snr_floor_db is a number of decibels from 0 to 60")
return out


Expand All @@ -81,6 +89,38 @@ def does_anything(o):
return o["noise"] != "off" or o["gain"] or o["high_pass"] or o["echo"] != "off" or o["trim_silence"]


def quality(mono):
"""A cheap read of a recording's sound before it's cleaned: (estimated SNR in dB or None, percent of samples
clipped). The noise floor and the speech level are the loudness at two percentiles of the 10 ms frames; numpy
only, no model. SNR is None when the recording is too short or too quiet to judge."""
if not len(mono):
return None, 0.0
x = np.abs(np.asarray(mono, dtype=np.float32))
clip_pct = round(100 * float(np.mean(x >= CLIP)), 2)
n = len(x) // FRAME
if n < 2:
return None, clip_pct
rms = np.sqrt(np.square(x[: n * FRAME].reshape(n, FRAME)).mean(axis=1))
rms = rms[rms > 0]
if len(rms) < 2:
return None, clip_pct
noise = float(np.percentile(rms, QUIET_PCT))
speech = float(np.percentile(rms, LOUD_PCT))
if noise <= 0 or speech <= 0:
return None, clip_pct
return round(20 * math.log10(speech / noise), 1), clip_pct


def is_poor(o, snr_db, clip_pct):
"""Whether a recording measures as low quality, so the quality gate still cleans it. Clipping is poor; an SNR that
couldn't be judged counts as poor too, so a doubtful recording is cleaned rather than skipped."""
if clip_pct > CLIP_MAX:
return True
if snr_db is None:
return True
return snr_db < o["snr_floor_db"]


def ready():
return components.importable("pywebrtc_audio")

Expand Down Expand Up @@ -275,6 +315,11 @@ def step(db, cfg, rid, say):
ensure(cfg, say)
path = ingest.audio_path(db, cfg, rec)
mono = ingest.decode(path)
snr_db, clip_pct = quality(mono)
if o["only_if_poor"] and not is_poor(o, snr_db, clip_pct):
del mono
db.q("UPDATE $r SET audio_cleanup = NONE", r=r)
return {"skipped": True, "snr_db": snr_db, "clip_pct": clip_pct, "snr_floor_db": o["snr_floor_db"]}
out, probs, gain, echo = clean(o, path, mono, rec.get("channels"))
del mono
total = len(out) * 1000 // SR
Expand All @@ -288,6 +333,8 @@ def step(db, cfg, rid, say):
"speech_ms": speech,
"kept_ms": kept,
"gain_db": gain,
"snr_db": snr_db,
"clip_pct": clip_pct,
"echo_note": echo if echo and echo not in ECHO else None,
"engine": f"{PACKAGE} {version() or ''}".strip(),
"at": store.now(),
Expand All @@ -301,8 +348,13 @@ def summary(saved):
"""One line for the step's log: what it found and what transcribing will hear."""
if not saved:
return "every audio cleanup option is off: it's transcribed as it is"
if saved.get("skipped"):
snr = "its noise couldn't be judged" if saved.get("snr_db") is None else f"{saved['snr_db']} dB signal-to-noise"
return f"good quality ({snr}, over the {saved['snr_floor_db']} dB floor): left as it is, transcribed from the original"
total = max(saved["duration_ms"], 1)
parts = [f"speech in {round(100 * saved['speech_ms'] / total)}% of {round(total / 60000, 1)} min"]
if saved.get("snr_db") is not None:
parts.append(f"{saved['snr_db']} dB signal-to-noise")
if saved["noise"] != "off":
parts.append(f"noise suppressed ({saved['noise'].replace('_', ' ')})")
if saved.get("gain_db") is not None:
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,8 @@ def labels():
"echo": "off",
"trim_silence": True,
"min_silence_ms": 2000,
"only_if_poor": False, # clean every recording the step runs on; True cleans only ones measured low quality
"snr_floor_db": 20, # with only_if_poor, a recording at or above this signal-to-noise is left as it is
},
"vocabulary": False, # give each recording's engine the names its namespace knows (vocabulary.py)
},
Expand Down
11 changes: 8 additions & 3 deletions fastapi_backend/app/schemas/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,14 @@ class AudioCleanup(ResponseModel):
echo: str = Field(description="echo cancellation: off, or the channel (left or right) holding what the speakers played")
trim_silence: bool = Field(description="silences of min_silence_ms or longer left out of what's transcribed")
min_silence_ms: int = Field(description="the shortest pause left out when trimming silence")
duration_ms: int = Field(description="how long the recording is")
speech_ms: int = Field(description="how much of it WebRTC's voice detection heard as speech")
kept_ms: int = Field(description="how much of it is transcribed: all of it unless silence is trimmed")
only_if_poor: bool = Field(False, description="clean only recordings measured as low quality; others are left as they are")
snr_floor_db: float = Field(20, description="with only_if_poor, a recording at or above this signal-to-noise is left as it is")
skipped: bool = Field(False, description="the quality gate left this recording as it is; it's transcribed from the original")
duration_ms: int | None = Field(None, description="how long the recording is")
speech_ms: int | None = Field(None, description="how much of it WebRTC's voice detection heard as speech")
kept_ms: int | None = Field(None, description="how much of it is transcribed: all of it unless silence is trimmed")
snr_db: float | None = Field(None, description="its estimated signal-to-noise, in dB (null when it couldn't be judged)")
clip_pct: float | None = Field(None, description="the percent of its samples that are clipped")
gain_db: float | None = Field(None, description="the gain applied by the end of the recording, in dB")
echo_note: str | None = Field(None, description="why echo wasn't cancelled though it was asked for")
engine: str | None = Field(None, description="what cleaned it up, with its version")
Expand Down
36 changes: 36 additions & 0 deletions fastapi_backend/tests/domain/test_audio_cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,8 @@ def test_settings_are_checked(db, cfg):
({"echo": "both"}, "echo is one of"),
({"gain": "yes"}, "gain is true or false"),
({"min_silence_ms": 10}, "min_silence_ms"),
({"only_if_poor": "sometimes"}, "only_if_poor is true or false"),
({"snr_floor_db": 80}, "snr_floor_db is a number"),
({"denoise": True}, "has no setting denoise"),
):
with pytest.raises(ValueError, match=says):
Expand Down Expand Up @@ -243,6 +245,40 @@ def test_an_agent_reads_what_the_step_did(client, db, cfg, recording):
assert pid.status_code in (200, 201), pid.text


def noisy():
"""The same speech and pauses as talk(), but under loud, steady noise: a low signal-to-noise recording."""
x = np.concatenate([quiet_for(2), babble(4, 1), quiet_for(10), babble(3, 2), quiet_for(3)])
return (x + np.random.default_rng(1).normal(0, 0.06, len(x))).astype(np.float32)


def test_quality_reads_signal_to_noise_and_clipping():
clean_snr, clip = audio_cleanup.quality(talk())
noisy_snr, _ = audio_cleanup.quality(noisy())
assert clean_snr is not None and noisy_snr is not None
assert clean_snr > noisy_snr # the clean recording measures a higher signal-to-noise
assert clip < 1 # talk() isn't clipped
clipped = np.clip(talk() * 20, -1, 1)
assert audio_cleanup.quality(clipped)[1] > audio_cleanup.CLIP_MAX # driven into clipping


def test_quality_gate_leaves_a_clean_recording_as_it_is(db, cfg, recording):
cfg["transcribe"]["cleanup"] = {"only_if_poor": True} # the clean-sounding talk() is over the default 20 dB floor
rid = recording()
log = _run(db, cfg, rid, "clean_audio")
assert db.one("SELECT audio_cleanup FROM $r", r=R("recording", rid)).get("audio_cleanup") is None
assert any("good quality" in line and "left as it is" in line for line in log)
assert Engine.heard[-1] == 22 * SR # it transcribed the whole original, nothing trimmed or cleaned


def test_quality_gate_still_cleans_a_low_quality_recording(db, cfg, recording):
cfg["transcribe"]["cleanup"] = {"only_if_poor": True, "trim_silence": False}
rid = recording(noisy())
_run(db, cfg, rid, "clean_audio")
saved = db.one("SELECT audio_cleanup FROM $r", r=R("recording", rid))["audio_cleanup"]
assert saved and not saved.get("skipped")
assert saved["snr_db"] < saved["snr_floor_db"] # under the floor, so it was cleaned


def _admin(db):
from tests.helpers import make_user

Expand Down
36 changes: 33 additions & 3 deletions nextjs-frontend/app/openapi-client/types.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1456,24 +1456,54 @@ export type AudioCleanup = {
* the shortest pause left out when trimming silence
*/
min_silence_ms: number;
/**
* Only If Poor
*
* clean only recordings measured as low quality; others are left as they are
*/
only_if_poor?: boolean;
/**
* Snr Floor Db
*
* with only_if_poor, a recording at or above this signal-to-noise is left as it is
*/
snr_floor_db?: number;
/**
* Skipped
*
* the quality gate left this recording as it is; it's transcribed from the original
*/
skipped?: boolean;
/**
* Duration Ms
*
* how long the recording is
*/
duration_ms: number;
duration_ms?: number | null;
/**
* Speech Ms
*
* how much of it WebRTC's voice detection heard as speech
*/
speech_ms: number;
speech_ms?: number | null;
/**
* Kept Ms
*
* how much of it is transcribed: all of it unless silence is trimmed
*/
kept_ms: number;
kept_ms?: number | null;
/**
* Snr Db
*
* its estimated signal-to-noise, in dB (null when it couldn't be judged)
*/
snr_db?: number | null;
/**
* Clip Pct
*
* the percent of its samples that are clipped
*/
clip_pct?: number | null;
/**
* Gain Db
*
Expand Down
2 changes: 2 additions & 0 deletions nextjs-frontend/components/settings/bodies.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ function AudioCleanup({ ctx }: { ctx: BodyCtx }) {
{ctx.form["transcribe.cleanup.trim_silence"] !== false && (
<F ctx={ctx} id="transcribe.cleanup.min_silence_ms" />
)}
<F ctx={ctx} id="transcribe.cleanup.only_if_poor" />
{ctx.form["transcribe.cleanup.only_if_poor"] === true && <F ctx={ctx} id="transcribe.cleanup.snr_floor_db" />}
</div>
</div>
);
Expand Down
17 changes: 17 additions & 0 deletions nextjs-frontend/components/settings/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,23 @@ export const FIELDS: FieldSpec[] = [
max: 60000,
unit: "ms",
},
{
section: "transcribe",
key: "cleanup.only_if_poor",
label: "Clean only low quality recordings",
kind: "switch",
hint: "The step measures each recording and leaves a clean-sounding one as it is, cleaning only the noisy, quiet or clipped ones",
},
{
section: "transcribe",
key: "cleanup.snr_floor_db",
label: "Clean below this signal-to-noise",
kind: "int",
min: 0,
max: 60,
unit: "dB",
hint: "A recording at or above this is counted as clean and left alone; lower is noisier",
},
{
section: "transcribe",
key: "vocabulary",
Expand Down
Loading
Loading