Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser))
- **Understand it.** Transcription, which can expect the names your namespace knows (off by default), people
recognised by voice (long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities
server (off by default, per namespace), low quality images and sound enhanced before they're read or transcribed (off by default), entities
and topics in a knowledge graph you can explore, query with Cypher or
SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
Expand All @@ -153,7 +153,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[enhancing images and sound](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
Expand Down Expand Up @@ -325,8 +325,8 @@ The guides are on the [docs site](https://adeelahmad.github.io/lense/) and in th
[the assistant by email](docs/assistant-email.md), [email replies](docs/email-replies.md), [podcasts](docs/podcasts.md), [notes](docs/notes.md),
[web clipper](docs/web-clipper.md), [the assistant in your browser](docs/browser-assistant.md),
[sensors](docs/sensors.md), [processing](docs/processing.md), [video](docs/video.md),
[NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md), [notifications](docs/notifications.md),
[automations](docs/automations.md), [timeline](docs/timeline.md)
[NSFW classification](docs/nsfw.md), [enhancing images and sound](docs/enhance.md),
[notifications](docs/notifications.md), [automations](docs/automations.md), [timeline](docs/timeline.md)
- **Knowledge:** [graph and Cypher](docs/graph.md), [graph history](docs/graph-history.md), [topics](docs/topics.md),
[the graph as files](docs/file-tree.md), [NFS share](docs/file-shares.md), [SMB share](docs/smb.md),
[linked data (RDF)](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md)
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/enhance-audio.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Clean up low quality sound before it's transcribed.** The `enhance` step now listens to audio and video too:
a noisy, quiet or clipped recording gets a cleaned-up copy (declipped, rumble and steady noise taken out, its level
evened) that transcription hears, in step with the original sample for sample, so words land where they were said;
the original, its waveform and voice matching stay as they were. Off by default: Settings → Enhance. ffmpeg's
filters are the default; WebRTC's noise suppression and gain (webrtc-noise-gain, Apache-2.0) is fetched the first
time it's chosen. Enhance and transcribe again from a recording's Details tab, the API, MCP or an automation
([Enhancing images and sound](docs/enhance.md)).
22 changes: 12 additions & 10 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1399,16 +1399,18 @@ Audited as `recording.nsfw`. `GET /resources/nsfw` (`ns` for one namespace) says
you read, how many resources are `flagged`, and how many of those the Library leaves out (`hidden`). Search hits and
photo search results carry `nsfw: true` for the ones the web app blurs.

### Enhancing images

The `enhance` step comes first in the standard pipeline: where [enhancing images](enhance.md) is on (off by
default), it measures an image and, when it's small, blurry, noisy, dull or has small faces, keeps an enhanced copy
that `read` and the steps after it work from; the original stays as it is. Other resources skip it. A resource
carries `enhanced`: `{kind, low, measured, enhanced, engine, scale, width, height, forced, at, download}` (`download`
a signed link to the copy), or `null` until the step looked at it. `POST /resources/{rid}/enhance` (the process
permission) enhances an image whatever the settings and its quality and runs its pipeline's later steps again,
answering the `job`; 400 for anything but an image. Audited as `recording.enhance`. `GET /resources/{rid}/enhanced`
is the copy, for members of its namespace.
### Enhancing images and sound

The `enhance` step comes first in the standard pipeline: where [enhancing](enhance.md) is on (off by default), it
measures an image and, when it's small, blurry, noisy, dull or has small faces, keeps an enhanced copy that `read`
and the steps after it work from; it listens to audio and video and, when the sound is noisy, quiet or clipped,
keeps a cleaned-up copy (FLAC, 16 kHz mono, in step with the original) that `transcribe` hears. The original stays
as it is. Documents and transcripts skip it. A resource carries `enhanced`: `{kind, low, measured, enhanced, engine,
forced, at, download}`, with `scale`, `width` and `height` for an image and `filters` or `speech` for sound
(`download` a signed link to the copy), or `null` until the step looked at it. `POST /resources/{rid}/enhance` (the
process permission) enhances an image or a recording whatever the settings and its quality and runs its pipeline's
later steps again, answering the `job`; 400 for a document or a transcript. Audited as `recording.enhance`.
`GET /resources/{rid}/enhanced` is the copy, for members of its namespace.

### Descriptions

Expand Down
2 changes: 1 addition & 1 deletion docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -704,7 +704,7 @@ or removed does nothing. What it writes is by the automation itself, `automation
| `http_request` | sends a `method` (GET, POST or PUT) to a `url`, with `headers` and a `body` (text, or an object sent as JSON) | out | `{status, ok, body, json, content_type, truncated, method, url, headers}`: the answer, as outside text |
| `run_workflow` | runs a recording `workflow` (its published `version`, or the one named) on an `asset` as a job | lens | `{job, workflow, version, asset}` |
| `run_pipeline` | processes an `asset` again with a `pipeline` (or the one its namespace uses), keeping its outputs | lens | `{job, pipeline, asset}` |
| `enhance` | [enhances](enhance.md) an image `asset` whatever the settings, keeping the original, and reads it again from the copy | lens | `{job, asset}` |
| `enhance` | [enhances](enhance.md) an image, audio or video `asset` whatever the settings, keeping the original, and reads or transcribes it again from the copy | lens | `{job, asset}` |
| `run_routine` | asks the scheduler to run a `routine` now (`propose_only` to only propose its changes) | lens, so it asks | `{routine, requested, propose_only}` |
| `set_field` | sets one of an `asset`'s custom fields (`field`: its id or label) to `value` (empty clears it) | lens | `{asset, field, label, value, changed}` |
| `add_to_collection` | moves an `asset` into a `collection` of its namespace | lens | `{asset, collection, name, moved, newly_see}` |
Expand Down
1 change: 1 addition & 0 deletions docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ never fetch a speech or voice model.
| Florence-2 | `llm.describe_engine` is `florence-2` | Florence-2-base-ft or -large-ft (MIT, `llm.florence_model`; about 280 or 830 MB), their 8-bit ONNX exports, the tokenizers library and ONNX Runtime, to [describe pictures](configuration.md#descriptions) on this machine |
| Moondream | `llm.describe_engine` is `moondream`, and this machine can run the chosen one | Moondream 3 Preview (BSL 1.1, about 19 GB, an NVIDIA GPU or a Mac with 32 GB) or Moondream 2 (Apache-2.0, about 3.7 GB), `llm.moondream_model`, at a pinned revision, with its tokenizer, PyTorch (the CPU build without an NVIDIA GPU) and transformers, to [describe pictures](configuration.md#descriptions) on this machine |
| Real-ESRGAN | [enhancing images](enhance.md) is on and `enhance.image_engine` is `realesrgan` (off by default) | Real-ESRGAN general x4v3 as an ONNX model (BSD-3-Clause, 5 MB), and ONNX Runtime |
| WebRTC audio processing | [enhancing sound](enhance.md) is on and `enhance.audio_engine` is `webrtc` (off by default), on Linux on x86-64 or ARM64 | webrtc-noise-gain (Apache-2.0, about 1 MB): WebRTC's noise suppression, automatic gain and voice detector |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| spaCy | `analysis.entities` is `spacy` ([Entities](processing.md#entities-and-the-graph-explorer)) | spaCy (MIT) and the model in `analysis.spacy_model`: `en_core_web_sm`, `de_core_news_sm` or `xx_ent_wiki_sm` (MIT, 11 to 15 MB each), about 60 MB in all |
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ Admins can change, in Settings: transcription, speaker separation, speech provid
provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word
search and search by meaning), reports and graph,
video, OCR, faces and objects, [NSFW classification](nsfw.md), [enhancing
images](enhance.md), workers, sign-in, components (and the edition, full or
images and sound](enhance.md), workers, sign-in, components (and the edition, full or
lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript
upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
automations, telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys,
Expand Down
106 changes: 78 additions & 28 deletions docs/enhance.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,26 @@
# Enhancing images
# Enhancing images and sound

Lens can look at each image before it's read and, when it's low quality, keep an enhanced copy that reading (OCR),
faces, objects, NSFW classification and descriptions work from instead. The original stays as it is. It's **off**
until an admin turns it on, and it runs on your own server: pictures aren't sent anywhere, and no model provider is
called.

Audio isn't enhanced yet.
faces, objects, NSFW classification and descriptions work from instead. It can listen to each audio or video file the
same way before it's transcribed, and keep a cleaned-up copy of its sound for transcription to hear. The original
stays as it is. Both are **off** until an admin turns them on, and they run on your own server: pictures and sound
aren't sent anywhere, and no model provider is called.

## Turning it on

Settings → Enhance, **Enhance low quality images** (`enhance.images`). The **enhance** step comes first in the
standard pipeline, so it looks at images added from then on. A pipeline of your own runs it when you add the
**enhance** step to it; a step with **force** set enhances every image it runs on, whatever the settings and its
quality.
Settings → Enhance, **Enhance low quality images** (`enhance.images`) and **Enhance low quality sound**
(`enhance.audio`). The **enhance** step comes first in the standard pipeline, so it looks at images and recordings
added from then on. A pipeline of your own runs it when you add the **enhance** step to it; a step with **force** set
enhances everything it runs on, whatever the settings and its quality. Documents and imported transcripts skip it.

Any image or recording can also be enhanced on request, even with it off: **Enhance** in an image's Details tab, or
**Enhance and transcribe again** in a recording's (editors), `POST /api/v1/resources/{rid}/enhance`, the `enhance`
tool for agents, or the `enhance` automation action. That makes the copy and runs the rest of its pipeline again on
it. Transcribing again makes a new transcript, so the Details tab says how many corrected lines it would replace.

Any image can also be enhanced on request, even with it off: **Enhance** in its Details tab (editors),
`POST /api/v1/resources/{rid}/enhance`, the `enhance` tool for agents, or the `enhance` automation action. That makes
the copy and runs the rest of its pipeline again on it.
## Images

## What counts as low quality
### What counts as low quality

The step measures the image and keeps why it looked low quality:

Expand All @@ -36,7 +38,7 @@ Settings → Video. With **Only images with faces** (`enhance.only_faces`), an i
An image with none of these is left as it is, and its job log says so: `good enough as it is (900×760, sharpness
312, noise 1.8, contrast 61)`.

## How it's enhanced
### How it's enhanced

A small image, or one with small faces, is made bigger: enough to bring its shorter side to `enhance.min_side` and
its smallest face to `enhance.min_face_px`, at most `enhance.max_scale` times and never past 4000 pixels a side. The
Expand All @@ -51,15 +53,55 @@ Real-ESRGAN and ONNX Runtime are fetched the first time the step runs with it ch
until then nothing is downloaded, so a Raspberry Pi that keeps `classic` stays small. An image over 1.5 megapixels is
enhanced by `classic` instead, so a small machine isn't swamped, and the job log says why.

## Sound

### What counts as low quality

The step listens to the whole recording (a video's soundtrack too) at 16 kHz mono, as transcription does, in 30 ms
frames, and keeps why it sounded low quality:

| Reason | When |
|---|---|
| noisy | its loudest frames are less than `enhance.min_snr_db` above its quietest (its noise floor) |
| quiet | the level of its loud half is under `enhance.min_level_db` dBFS |
| clipped | more than 0.1% of its samples are at full scale |

A recording with none of these is left as it is (`good enough as it is (signal to noise 31 dB, level -21 dBFS, 0%
clipped)`), and a silent one is skipped. So is one whose transcript came with it, unless it's asked for, since nothing
would hear the copy.

### How it's enhanced

The engine (`enhance.audio_engine`):

| Engine | What it does |
|---|---|
| `ffmpeg` (default) | ffmpeg's own filters, nothing to fetch: `adeclip` repairs clipping, a high-pass filter takes out rumble and hum below 70 Hz, `afftdn` takes out steady noise down from the floor it measured, and a quiet recording is turned up towards -20 dBFS (as far as its peak allows) and evened out with `dynaudnorm` |
| `webrtc` | WebRTC's audio processing through [webrtc-noise-gain](https://github.com/OHF-voice/webrtc-noise-gain) (Apache-2.0; WebRTC is BSD-3-Clause): its noise suppression, high-pass filter and, for a quiet recording, automatic gain, 10 ms at a time. Its voice detector says how much of it is speech |

Asked for with nothing wrong, it gets a gentle clean-up of everything: the high-pass filter, light noise reduction
and `dynaudnorm`.

The copy keeps the original's times sample for sample, so the transcript's words land where they were said in the
original: a filter that puts its output late (`afftdn` does) is measured once with a probe of clicks and
bursts, and the copy is moved back by that much. Speakers are still told apart from the original, so voiceprints stay
comparable between recordings, and the player's waveform is the original's.

WebRTC's processing is fetched the first time the step runs with it chosen, on Linux on x86-64 or ARM64; until then,
and elsewhere, ffmpeg does it and the job log says why. Echo cancellation needs the sound of the far end of a call,
which a recording doesn't have, so neither engine does it.

## The enhanced copy

The copy is a JPEG kept beside the original (`data_dir/renditions/<id>-enhanced.jpg`), encrypted where the
namespace's files are, re-keyed with them, and deleted with the resource. Its Details tab says what was found and
made (`Small, blurry · 240×180 to 960×720 by the classic engine`) with **Save the copy**. Only members of the
namespace can download it; visitors to a shared resource see its pages as before.
The copy is kept beside the original, a JPEG for an image and FLAC (16 kHz mono) for sound
(`data_dir/renditions/<id>-enhanced.jpg` or `.flac`), encrypted where the namespace's files are, re-keyed with them,
and deleted with the resource. Its Details tab says what was found and made (`Small, blurry · 240×180 to 960×720 by
the classic engine`, `Noisy · signal to noise 9.5 dB, level -24.1 dBFS · by ffmpeg's filters`) with **Save the
copy**. Only members of the namespace can download it; visitors to a shared resource see its pages or hear its media
as before.

Turning it off leaves copies alone until an image is processed again: then a copy the step made by itself is removed,
and one made on request is kept.
Turning it off leaves copies alone until an image or a recording is processed again: then a copy the step made by
itself is removed, and one made on request is kept.

## Settings

Expand All @@ -72,21 +114,29 @@ and one made on request is kept.
| `enhance.min_sharpness` | `100` | blurry below this sharpness (0 to 10000; 0 never counts one as blurry) |
| `enhance.min_face_px` | `64` | faces smaller than this many pixels tall are too small (0 to 1024; 0 turns it off) |
| `enhance.max_scale` | `4` | enlarge at most this many times (1 to 4) |
| `enhance.audio` | `false` | enhance low quality sound |
| `enhance.audio_engine` | `ffmpeg` | `ffmpeg` or `webrtc` |
| `enhance.min_snr_db` | `15` | noisy below this many dB of signal over noise (0 to 60) |
| `enhance.min_level_db` | `-35` | quiet below this level, dBFS (-70 to 0) |

## API and agents

- `POST /api/v1/resources/{rid}/enhance` (needs the process permission on the resource): enhance an image now and run
its pipeline's later steps again; answers the `job`. 400 for anything but an image. Audited as `recording.enhance`.
- A resource carries `enhanced`: `{kind, low, measured, enhanced, engine, scale, width, height, forced, at, download}`,
where `measured` is `{width, height, sharpness, noise, contrast, brightness, faces, smallest_face}` and `download` a
signed link to the copy; `null` until the step has looked at it.
- `POST /api/v1/resources/{rid}/enhance` (needs the process permission on the resource): enhance an image or a
recording now and run its pipeline's later steps again; answers the `job`. 400 for a document or a transcript.
Audited as `recording.enhance`.
- A resource carries `enhanced`: `{kind, low, measured, enhanced, engine, forced, at, download}`, with `scale`, `width`
and `height` for an image and `filters` (ffmpeg) or `speech` (WebRTC) for sound. `measured` is `{width, height,
sharpness, noise, contrast, brightness, faces, smallest_face}` for an image and `{duration, snr_db, level_db,
floor_db, peak_db, clipped}` for sound, and `download` a signed link to the copy; `null` until the step has looked
at it.
- `GET /api/v1/resources/{rid}/enhanced`: the copy (members of its namespace).
- [MCP](mcp.md): `enhance` (a write-scope token); `get_recording` says what the step found.
- [Automations](automations.md): the `enhance` action, on an asset.

## Refine later

- Enhance audio before it's transcribed: noise, level and clipping.
- Enhance a video's keyframes and a document's scanned pages, not only images.
- A face restorer for small faces, with a permissive license.
- Show the original and the copy side by side in the asset page.
- A speech enhancement model for heavy, changing noise (babble, music), with a permissive license.
- Enhance a video or episode imported by its address before it's downloaded (now: on request after).
- Play the enhanced sound in the player, and show the original and the copy side by side.
Loading
Loading