Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,13 +152,16 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an
asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos
by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by
Florence-2 or Moondream on your own server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing
and, with a local model, punctuation put back, and grammar by a LanguageTool server you run), entities found by spaCy
instead of the rules, and the dates people say read as days on the calendar, each off by default.
Florence-2 or Moondream on your own server. A recording's sound can be cleaned up before it's transcribed (noise,
volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before analysis
(stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server you
run), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each off
by default.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[cleaning up audio](docs/processing.md#cleaning-up-audio),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/audio-cleanup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Clean up a recording's sound before it's transcribed.** A new pipeline step, Clean up audio (`clean_audio`), runs a
recording through WebRTC's audio processing (pywebrtc-audio, Apache-2.0, about 0.5 MB, fetched the first time it
runs): noise suppression, automatic gain, a high-pass filter, echo cancellation for two-channel call recordings, and
pauses left out of what's transcribed, with the transcript's times put back on the recording's timeline. Settings →
Transcription → Audio cleanup holds what it does; no pipeline has it until you add it, the file is never changed, and
`GET /api/v1/recordings/{id}` says what it found (`audio_cleanup`)
([Cleaning up audio](docs/processing.md#cleaning-up-audio)).
1 change: 1 addition & 0 deletions docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ never fetch a speech or voice model.
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| spaCy | `analysis.entities` is `spacy` ([Entities](processing.md#entities-and-the-graph-explorer)) | spaCy (MIT) and the model in `analysis.spacy_model`: `en_core_web_sm`, `de_core_news_sm` or `xx_ent_wiki_sm` (MIT, 11 to 15 MB each), about 60 MB in all |
| WebRTC audio processing | a job first reaches a pipeline's `clean_audio` step ([Cleaning up audio](processing.md#cleaning-up-audio)) | pywebrtc-audio (Apache-2.0, about 0.5 MB, numpy only): WebRTC's noise suppression, gain control, echo cancellation and voice detection |
| DuckDuckGo search | `web.search` is `duckduckgo` | ddgs (MIT, about 30 MB with lxml and primp), the client the assistant's [web search](assistant.md#web-search-and-reading-pages) reads DuckDuckGo's results with |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
Expand Down
36 changes: 34 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ How assets move through Lens: where they come from, what each step does, and how
- `scan` finds audio under each namespace's paths and fingerprints it: moved files keep their history, duplicates are skipped,
and so are files whose asset someone deleted or moved to another namespace (at the same path, or a copy of the same
file).
- `clean_audio` (only in a pipeline that has it) cleans up the sound before it's transcribed: noise, volume, rumble,
echo, and silence left out of what's transcribed ([Cleaning up audio](#cleaning-up-audio)).
- `transcribe` uses SenseVoice, faster-whisper or mlx-whisper. When the configured engine isn't installed on a worker (or
doesn't import there), it uses the next one that is and says so in the job log; with none at all the step fails,
saying how to add one. A file that fails is marked and the batch carries on. For a
Expand All @@ -23,6 +25,36 @@ How assets move through Lens: where they come from, what each step does, and how

Every step takes `--ns`, `--limit` and `--force`; `run` does them all, and a lock stops two runs overlapping.

### Cleaning up audio

A phone on a table, a fan, a quiet voice across the room or a call with the other side coming back through the
speakers: speech-to-text hears all of it. The `clean_audio` step (Clean up audio in the pipeline editor and Reprocess)
runs a recording through WebRTC's audio processing, the code a browser runs on a call, with
[pywebrtc-audio](https://github.com/strands-labs/pywebrtc-audio) (Apache-2.0, about 0.5 MB; fetched the first time
the step runs, [Components](components.md)). No pipeline has it until you add it before Transcribe, so nothing changes
until then. What it does is in Settings → Transcription → Audio cleanup (`transcribe.cleanup`):

| Setting | Default | What it does |
|---|---|---|
| `noise` | `moderate` | Noise suppression: `off`, `low`, `moderate`, `high` or `very_high` (6, 12, 18 or 21 dB) |
| `gain` | on | Automatic gain control, so a quiet voice is as loud as a near one |
| `high_pass` | on | Takes out the rumble below speech: traffic, handling noise, a knocked desk |
| `echo` | `off` | Echo cancellation for a two-channel recording with what the speakers played on one channel (`left` or `right`) and the microphone on the other, as call recorders keep them; the far side's voice is taken out of the microphone, then both sides are mixed |
| `trim_silence` | on | Pauses of `min_silence_ms` (2000) or longer, by WebRTC's voice detection, are left out of what's transcribed |

Leaving silence out means a cloud engine bills fewer minutes (the activity ledger's seconds are what was sent), a
local one finishes sooner and Whisper has no quiet stretch to imagine words in. The transcript's times are put back on
the recording's own timeline, so the player and every citation stay where the words were said.

The step measures how much of the recording is speech and keeps what it ran with on the asset (`audio_cleanup` in
`GET /api/v1/recordings/{id}`: the options, `speech_ms`, `kept_ms`, the gain it ended on). From then on transcribing
that asset hears it cleaned up that way, also on a later Reprocess. The file is never changed and no second copy is
stored: cleaning takes a fraction of a second per minute of audio, so it's done again each time. Speaker separation and
voice IDs still listen to the original. Run the step with every option off to have the asset transcribed as it is
again. A worker that can't fetch pywebrtc-audio transcribes the original and says so in the job log. Run it on one
asset with Reprocess → Clean up audio, or `POST /api/v1/recordings/{id}/reprocess` with
`{"steps": ["clean_audio", "transcribe"]}`; workers whose `workers.steps` list has `transcribe` run it too.

### Your own names in transcription

Speech-to-text gets wrong the names only you use: family, colleagues, places, projects. With `transcribe.vocabulary`
Expand Down Expand Up @@ -712,8 +744,8 @@ A pipeline is a named, versioned list of steps; each namespace can choose its de
analyze, summarize, report). Imports, watched folders and reprocessing use the namespace's pipeline; steps that need
audio skip themselves for imported text. Steps:

- `transcribe`, `diarize`, `clean` ([Cleaning up transcripts](#cleaning-up-transcripts)), `analyze`, `summarize`, and
`report` (the built-in report).
- `clean_audio` ([Cleaning up audio](#cleaning-up-audio)), `transcribe`, `diarize`, `clean`
([Cleaning up transcripts](#cleaning-up-transcripts)), `analyze`, `summarize`, and `report` (the built-in report).
- `llm`: renders a prompt template, sends it to the configured model, checks the reply against the template's JSON
Schema and saves it as a named output (e.g. `meeting_notes`).
- `spec`: writes a technical spec from the transcript into a note linked to the asset, with the Spec from transcript
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ request, to another check that proves the same thing; loosening a requirement or
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |
| Or by Moondream on this server, off by default, fetched at a pinned revision only where the chosen model can run, and saying why where it can't | [test_moondream.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_moondream.py) `test_moondream_describes_pictures_without_a_model_provider`, `test_where_each_moondream_can_run`, `test_on_a_cpu_moondream_3_is_never_fetched_and_the_step_says_why`, `test_moondream_is_fetched_pinned`, `test_admins_and_agents_choose_moondream` |
| A recording's sound cleaned up before transcription in a pipeline that has the step (noise, gain, echo), with silence left out of what's transcribed and the transcript's times kept on the recording's timeline | [test_audio_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_audio_cleanup.py) `test_speech_is_found_and_silence_left_out_of_transcription`, `test_echo_of_the_far_side_is_taken_out`, `test_without_the_step_or_with_every_option_off_it_is_transcribed_as_it_is`, `test_an_agent_reads_what_the_step_did` |
| Transcripts cleaned up before analysis (stutters, "um"s, spacing), off by default, each change in the edit history with Revert and people's corrections left alone | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_off_by_default_changes_nothing`, `test_clean_step_tidies_lines_and_keeps_them_in_the_history`, `test_people_s_corrections_outrank_the_rules` |
| Punctuation put back in transcripts that lack it by a local model, fetched only when picked | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_punctuation_put_back_where_it_is_missing`, `test_without_the_model_the_rules_run_alone`, `test_the_model_is_fetched_only_when_it_is_asked_for` |
| Grammar fixed by a LanguageTool server you run, when its address is set: grammar fixes only, never spelling or a word with a capital | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_grammar_fixes_from_languagetool_leave_names_and_spelling`, `test_without_its_server_the_rules_run_alone`, `test_the_grammar_server_is_an_address_without_a_password` |
Expand Down
Loading
Loading