From c0ed7f5ca21a4429375e58580dbbd25bb19606d2 Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sat, 17 Oct 2026 01:38:01 +0000
Subject: [PATCH 1/8] feat(enhance): enhance low quality images before they're
read
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The new enhance step, first in the standard pipeline, measures each image
(size, sharpness, noise, contrast, and face size where the namespace finds
faces) and keeps an enhanced copy of a low quality one that read, faces,
objects and descriptions then work from. The original stays as it is.
Off by default (Settings → Enhance). The classic engine (Pillow) needs
nothing; Real-ESRGAN general x4v3 (BSD-3-Clause, 5 MB ONNX) is fetched the
first time it's chosen. The copy is encrypted like the namespace's files,
re-keyed and deleted with them, and downloadable by members only.
Agents: POST /resources/{rid}/enhance, the MCP tool enhance and the enhance
automation action enhance an image on request. The web app has the settings
section, the pipeline step and an Enhance row in the Details tab.
Co-Authored-By: Claude Opus 5.5 (1M context)
Claude-Session: https://claude.ai/code/session_012m72U9mPLCbuH8rSAZGSuZ
---
CHANGELOG.md | 7 +
README.md | 18 +-
docs/api.md | 13 +
docs/automations.md | 1 +
docs/components.md | 3 +-
docs/configuration.md | 2 +-
docs/enhance.md | 92 ++++
docs/mcp.md | 4 +-
fastapi_backend/app/api/mcp_tools.py | 23 +
fastapi_backend/app/api/media.py | 2 +-
.../app/api/v1/routes/recordings.py | 14 +
fastapi_backend/app/api/v1/routes/video.py | 17 +-
fastapi_backend/app/domain/actions_run.py | 35 +-
fastapi_backend/app/domain/components.py | 50 +++
fastapi_backend/app/domain/deletion.py | 14 +-
fastapi_backend/app/domain/documents.py | 5 +-
fastapi_backend/app/domain/enhance.py | 415 ++++++++++++++++++
fastapi_backend/app/domain/jobs.py | 51 ++-
fastapi_backend/app/domain/keyring.py | 24 +-
fastapi_backend/app/domain/ops_tools.py | 1 +
fastapi_backend/app/domain/pipelines.py | 34 +-
fastapi_backend/app/domain/settings.py | 6 +
fastapi_backend/app/domain/store.py | 16 +
fastapi_backend/app/schemas/recordings.py | 33 ++
fastapi_backend/tests/api/test_enhance.py | 286 ++++++++++++
fastapi_backend/tests/api/test_jobs.py | 4 +
fastapi_backend/tests/api/test_mcp.py | 1 +
mkdocs.yml | 1 +
nextjs-frontend/__tests__/enhance.test.tsx | 112 +++++
nextjs-frontend/app/openapi-client/index.ts | 11 +
nextjs-frontend/app/openapi-client/sdk.gen.ts | 38 ++
.../app/openapi-client/types.gen.ts | 185 ++++++++
nextjs-frontend/components/batches/format.ts | 12 +-
.../components/pipelines/pipeline-canvas.tsx | 4 +-
.../components/pipelines/pipeline-editor.tsx | 3 +
.../components/pipelines/pipeline-model.ts | 4 +
.../components/recording/details-tab.tsx | 2 +
.../components/recording/enhance-model.ts | 30 ++
.../components/recording/enhance.tsx | 63 +++
.../components/settings/bodies.tsx | 22 +
nextjs-frontend/components/settings/find.ts | 2 +-
nextjs-frontend/components/settings/model.ts | 71 +++
.../components/settings/settings-shell.tsx | 2 +
nextjs-frontend/components/ui/loop.tsx | 2 +
nextjs-frontend/openapi.json | 294 +++++++++++++
45 files changed, 1993 insertions(+), 36 deletions(-)
create mode 100644 docs/enhance.md
create mode 100644 fastapi_backend/app/domain/enhance.py
create mode 100644 fastapi_backend/tests/api/test_enhance.py
create mode 100644 nextjs-frontend/__tests__/enhance.test.tsx
create mode 100644 nextjs-frontend/components/recording/enhance-model.ts
create mode 100644 nextjs-frontend/components/recording/enhance.tsx
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d168bc8b5..b017accce 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
## Unreleased
+- **Enhance low quality images before they're read.** The new `enhance` step, first in the standard pipeline,
+ measures each image (size, sharpness, noise, contrast, and face size where the namespace finds faces) and keeps an
+ enhanced copy of a low quality one, which OCR, faces, objects and descriptions then read; the original stays as it
+ is. Off by default: Settings → Enhance. The light `classic` engine (Pillow) is the default; Real-ESRGAN general
+ x4v3 (BSD-3-Clause, 5 MB) is fetched the first time it's chosen. The Details tab says what was found, saves the
+ copy, and enhances an image on request; agents use `POST /api/v1/resources/{rid}/enhance`, the MCP tool `enhance`
+ or the `enhance` automation action ([Enhancing images](docs/enhance.md)).
- **Automations run.** An automation saved over the API (`/api/v1/automations`) now reacts to new assets in its
namespaces: when its If passes, it runs its workflow as the person who made it, within its limits and budgets, and
each run shows what its actions did. Try one on an activity without keeping or sending anything, or run it now;
diff --git a/README.md b/README.md
index 3fc476a80..aad1ed05f 100644
--- a/README.md
+++ b/README.md
@@ -101,15 +101,17 @@ Everything below is built and merged. Work that is still in a pull request is li
[the assistant in your browser](docs/browser-assistant.md))
- **Understand it.** Transcription, people recognised by voice (long recordings go to a speech provider in pieces that
fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
- server (off by default, per namespace), entities and topics in a knowledge graph you can explore, query with Cypher
- or SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
- (`/wiki//`), and entities, topics and people are edited and merged from their lists, the graph or
- an asset, with Undo instead of a confirm. Find what is most like a passage, an asset, a topic or an entity, in a
- Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show.
+ server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities
+ and topics in a knowledge graph you can explore, query with Cypher or SPARQL, and see as it was at any version.
+ Every entity has its own page at a wiki address (`/wiki//`), and entities, topics and people are
+ edited and merged from their lists, the graph or an asset, with Undo instead of a confirm. Find what is most like a
+ passage, an asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with
+ anytopdf, photos by what they show.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content),
[graph history](docs/graph-history.md), [topics](docs/topics.md), [video](docs/video.md),
- [NSFW classification](docs/nsfw.md), [long recordings](docs/speech-providers.md#long-recordings))
+ [NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md),
+ [long recordings](docs/speech-providers.md#long-recordings))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
source, read aloud in two voices. Keep notes beside every asset, entity and topic, written by you or the assistant, in
a tree where notes are renamed, moved and pinned (pins are your own), with today's note one click away, a search by
@@ -216,8 +218,8 @@ The guides are on the [docs site](https://adeelahmad.github.io/lense/) and in th
[the assistant by email](docs/assistant-email.md), [podcasts](docs/podcasts.md), [notes](docs/notes.md),
[web clipper](docs/web-clipper.md), [the assistant in your browser](docs/browser-assistant.md),
[sensors](docs/sensors.md), [processing](docs/processing.md), [video](docs/video.md),
- [NSFW classification](docs/nsfw.md), [notifications](docs/notifications.md), [automations](docs/automations.md),
- [timeline](docs/timeline.md)
+ [NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md), [notifications](docs/notifications.md),
+ [automations](docs/automations.md), [timeline](docs/timeline.md)
- **Knowledge:** [graph and Cypher](docs/graph.md), [graph history](docs/graph-history.md), [topics](docs/topics.md),
[the graph as files](docs/file-tree.md), [NFS share](docs/file-shares.md), [SMB share](docs/smb.md),
[linked data (RDF)](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md)
diff --git a/docs/api.md b/docs/api.md
index 6e8f3bec8..9c52d0410 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -560,6 +560,8 @@ PUT /api/v1/resources/{rid}/ip-groups/{gid}
DELETE /api/v1/resources/{rid}/ip-groups/{gid}
GET /api/v1/resources/{rid}/player
PUT /api/v1/resources/{rid}/nsfw
+POST /api/v1/resources/{rid}/enhance
+GET /api/v1/resources/{rid}/enhanced
GET /api/v1/resources/{rid}/embed-link
GET /api/v1/resources/{rid}/audio
GET /api/v1/resources/{rid}/wordcloud.svg
@@ -1307,6 +1309,17 @@ Audited as `recording.nsfw`. `GET /resources/nsfw` (`ns` for one namespace) says
you read, how many resources are `flagged`, and how many of those the Library leaves out (`hidden`). Search hits and
photo search results carry `nsfw: true` for the ones the web app blurs.
+### Enhancing images
+
+The `enhance` step comes first in the standard pipeline: where [enhancing images](enhance.md) is on (off by
+default), it measures an image and, when it's small, blurry, noisy, dull or has small faces, keeps an enhanced copy
+that `read` and the steps after it work from; the original stays as it is. Other resources skip it. A resource
+carries `enhanced`: `{kind, low, measured, enhanced, engine, scale, width, height, forced, at, download}` (`download`
+a signed link to the copy), or `null` until the step looked at it. `POST /resources/{rid}/enhance` (the process
+permission) enhances an image whatever the settings and its quality and runs its pipeline's later steps again,
+answering the `job`; 400 for anything but an image. Audited as `recording.enhance`. `GET /resources/{rid}/enhanced`
+is the copy, for members of its namespace.
+
### Descriptions
The `describe` step has a model that can see images, the one an admin chose as `llm.vision_model`, say in a few
diff --git a/docs/automations.md b/docs/automations.md
index 40a810eb9..45d9fa72d 100644
--- a/docs/automations.md
+++ b/docs/automations.md
@@ -514,6 +514,7 @@ or removed does nothing. What it writes is by the automation itself, `automation
| `http_request` | sends a `method` (GET, POST or PUT) to a `url`, with `headers` and a `body` (text, or an object sent as JSON) | out | `{status, ok, body, json, content_type, truncated, method, url, headers}`: the answer, as outside text |
| `run_workflow` | runs a recording `workflow` (its published `version`, or the one named) on an `asset` as a job | lens | `{job, workflow, version, asset}` |
| `run_pipeline` | processes an `asset` again with a `pipeline` (or the one its namespace uses), keeping its outputs | lens | `{job, pipeline, asset}` |
+| `enhance` | [enhances](enhance.md) an image `asset` whatever the settings, keeping the original, and reads it again from the copy | lens | `{job, asset}` |
| `run_routine` | asks the scheduler to run a `routine` now (`propose_only` to only propose its changes) | lens, so it asks | `{routine, requested, propose_only}` |
| `set_field` | sets one of an `asset`'s custom fields (`field`: its id or label) to `value` (empty clears it) | lens | `{asset, field, label, value, changed}` |
| `add_to_collection` | moves an `asset` into a `collection` of its namespace | lens | `{asset, collection, name, moved, newly_see}` |
diff --git a/docs/components.md b/docs/components.md
index 7be66590b..b3c3295d0 100644
--- a/docs/components.md
+++ b/docs/components.md
@@ -20,6 +20,7 @@ never fetch a speech or voice model.
| Face models | `video.face_engine` is `anytopdf` | SFace, for anytopdf to describe faces with (anytopdf too) |
| Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` |
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
+| Real-ESRGAN | [enhancing images](enhance.md) is on and `enhance.image_engine` is `realesrgan` (off by default) | Real-ESRGAN general x4v3 as an ONNX model (BSD-3-Clause, 5 MB), and ONNX Runtime |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
@@ -28,7 +29,7 @@ never fetch a speech or voice model.
- Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win.
- Models go into `data_dir/models`: the Hugging Face, ModelScope and PyTorch caches point there (unless `HF_HOME`,
`MODELSCOPE_CACHE` or `TORCH_HOME` say otherwise). They outlast container rebuilds and upgrades.
-- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model) are checked against their SHA-256.
+- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model, Real-ESRGAN) are checked against their SHA-256.
- A job step that needs something still being fetched waits for it instead of falling back or failing. If a fetch fails,
the step runs as before (for transcription, falling back to another engine), and the fetch is tried again an hour
later, when the settings change, or on **Check again now**.
diff --git a/docs/configuration.md b/docs/configuration.md
index 2c57e7d95..3710f5754 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -108,7 +108,7 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th
Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM
provider and key, a local model, the AI assistant, search (word search and search by meaning), reports and graph,
-video, OCR, faces and objects, [NSFW classification](nsfw.md), workers, sign-in, components (and the edition, full or
+video, OCR, faces and objects, [NSFW classification](nsfw.md), [enhancing images](enhance.md), workers, sign-in, components (and the edition, full or
lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript
upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
automations, telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys, and IIIF and metadata. The API
diff --git a/docs/enhance.md b/docs/enhance.md
new file mode 100644
index 000000000..4bf66a247
--- /dev/null
+++ b/docs/enhance.md
@@ -0,0 +1,92 @@
+# Enhancing images
+
+Lens can look at each image before it's read and, when it's low quality, keep an enhanced copy that reading (OCR),
+faces, objects, NSFW classification and descriptions work from instead. The original stays as it is. It's **off**
+until an admin turns it on, and it runs on your own server: pictures aren't sent anywhere, and no model provider is
+called.
+
+Audio isn't enhanced yet.
+
+## Turning it on
+
+Settings → Enhance, **Enhance low quality images** (`enhance.images`). The **enhance** step comes first in the
+standard pipeline, so it looks at images added from then on. A pipeline of your own runs it when you add the
+**enhance** step to it; a step with **force** set enhances every image it runs on, whatever the settings and its
+quality.
+
+Any image can also be enhanced on request, even with it off: **Enhance** in its Details tab (editors),
+`POST /api/v1/resources/{rid}/enhance`, the `enhance` tool for agents, or the `enhance` automation action. That makes
+the copy and runs the rest of its pipeline again on it.
+
+## What counts as low quality
+
+The step measures the image and keeps why it looked low quality:
+
+| Reason | When |
+|---|---|
+| small | its shorter side is under `enhance.min_side` pixels |
+| blurry | its sharpness (the variance of its Laplacian, where it's sharpest) is under `enhance.min_sharpness` |
+| noisy | its noise, estimated where it's plainest so edges and texture don't count, is over 6 levels out of 255 |
+| low contrast | the spread of its tones is under 30 levels out of 255: flat, too dark or too bright |
+| faces too small | a face on it is under `enhance.min_face_px` pixels tall, so recognition has little to go on |
+
+Faces are looked for only where the namespace finds faces (its faces mode isn't off), with the face engine in
+Settings → Video. With **Only images with faces** (`enhance.only_faces`), an image with no face is left alone.
+
+An image with none of these is left as it is, and its job log says so: `good enough as it is (900×760, sharpness
+312, noise 1.8, contrast 61)`.
+
+## How it's enhanced
+
+A small image, or one with small faces, is made bigger: enough to bring its shorter side to `enhance.min_side` and
+its smallest face to `enhance.min_face_px`, at most `enhance.max_scale` times and never past 4000 pixels a side. The
+engine (`enhance.image_engine`):
+
+| Engine | What it does |
+|---|---|
+| `classic` (default) | Pillow, nothing to fetch: a noisy image is smoothed with a median filter, enlarged with Lanczos, a dull one stretched to its full range of tones, and a soft or enlarged one sharpened (unsharp mask) |
+| `realesrgan` | [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) general x4v3 (BSD-3-Clause, about 5 MB, as Qualcomm AI Hub exports it to ONNX), run with ONNX Runtime on the CPU in tiles. It enlarges four times while taking out blur, noise and JPEG blocks, and the result is brought to the size wanted |
+
+Real-ESRGAN and ONNX Runtime are fetched the first time the step runs with it chosen ([Components](components.md));
+until then nothing is downloaded, so a Raspberry Pi that keeps `classic` stays small. An image over 1.5 megapixels is
+enhanced by `classic` instead, so a small machine isn't swamped, and the job log says why.
+
+## The enhanced copy
+
+The copy is a JPEG kept beside the original (`data_dir/renditions/-enhanced.jpg`), encrypted where the
+namespace's files are, re-keyed with them, and deleted with the resource. Its Details tab says what was found and
+made (`Small, blurry · 240×180 to 960×720 by the classic engine`) with **Save the copy**. Only members of the
+namespace can download it; visitors to a shared resource see its pages as before.
+
+Turning it off leaves copies alone until an image is processed again: then a copy the step made by itself is removed,
+and one made on request is kept.
+
+## Settings
+
+| Setting | Default | |
+|---|---|---|
+| `enhance.images` | `false` | enhance low quality images |
+| `enhance.image_engine` | `classic` | `classic` or `realesrgan` |
+| `enhance.only_faces` | `false` | only images with a face |
+| `enhance.min_side` | `720` | small below this many pixels on the shorter side (64 to 4000) |
+| `enhance.min_sharpness` | `100` | blurry below this sharpness (0 to 10000; 0 never counts one as blurry) |
+| `enhance.min_face_px` | `64` | faces smaller than this many pixels tall are too small (0 to 1024; 0 turns it off) |
+| `enhance.max_scale` | `4` | enlarge at most this many times (1 to 4) |
+
+## API and agents
+
+- `POST /api/v1/resources/{rid}/enhance` (needs the process permission on the resource): enhance an image now and run
+ its pipeline's later steps again; answers the `job`. 400 for anything but an image. Audited as `recording.enhance`.
+- A resource carries `enhanced`: `{kind, low, measured, enhanced, engine, scale, width, height, forced, at, download}`,
+ where `measured` is `{width, height, sharpness, noise, contrast, brightness, faces, smallest_face}` and `download` a
+ signed link to the copy; `null` until the step has looked at it.
+- `GET /api/v1/resources/{rid}/enhanced`: the copy (members of its namespace).
+- [MCP](mcp.md): `enhance` (a write-scope token); `get_recording` says what the step found.
+- [Automations](automations.md): the `enhance` action, on an asset.
+
+## Refine later
+
+- Enhance audio before it's transcribed: noise, level and clipping.
+- Enhance a video's keyframes and a document's scanned pages, not only images.
+- A face restorer for small faces, with a permissive license.
+- Show the original and the copy side by side in the asset page.
diff --git a/docs/mcp.md b/docs/mcp.md
index a92688f1b..2597de490 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -95,6 +95,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `nsfw_settings` | how [NSFW classification](nsfw.md) works in a namespace: on or off, the threshold, how flagged resources show, and how many are classified, flagged and not classified yet |
| `set_nsfw` | turn NSFW classification on or off, set its threshold and display, for a namespace (owners) or the archive (admins), and optionally classify what's there. Needs a write-scope token |
| `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access |
+| `enhance` | [enhance an image](enhance.md) whatever the settings, keeping the original, and read it again from the enhanced copy. Needs a write-scope token and the process permission |
| `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there |
| `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token |
| `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files |
@@ -136,7 +137,8 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed
`merge_topics`, `delete_topic` and `undo_topic_change`, which curate entities and topics as an editor does in the web
app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which
queues an episode, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
-editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring` and
+editors), `enhance`, which enhances an image and reads it again (the process permission),
+`set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring` and
`set_install_settings`, which have the machine change how Lens runs (admins).
Importing stays in the web app and the API.
diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py
index 4c6acdcc1..053586be6 100644
--- a/fastapi_backend/app/api/mcp_tools.py
+++ b/fastapi_backend/app/api/mcp_tools.py
@@ -43,6 +43,7 @@
approvals,
auth,
blobs,
+ enhance,
events,
feeds,
filetree,
@@ -736,6 +737,8 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]:
out["web_page"] = rec.web.model_dump(exclude_none=True)
if rec.nsfw:
out["nsfw"] = rec.nsfw.model_dump(exclude_none=True, include={"flagged", "score", "manual", "threshold", "checked", "of"})
+ if rec.enhanced:
+ out["enhanced"] = rec.enhanced.model_dump(exclude_none=True, exclude={"download"})
return {k: v for k, v in out.items() if v is not None}
@@ -1787,6 +1790,26 @@ def mark_nsfw(ctx: Context, recording_id: int, verdict: str) -> dict[str, Any]:
return {"recording_id": recording_id, "nsfw": got, "url": ctx.link(recording_id)}
+# ---------- enhancing low quality images ----------
+@tool(
+ "enhance",
+ "Enhance an image",
+ "Make an enhanced copy of an image asset (enlarged, sharpened, cleaned of noise, its contrast stretched), whatever "
+ "the settings and its quality, and read it again from that copy: its text, faces and objects. The original stays "
+ "as it is. Low quality images are enhanced on their own where an admin turned that on (enhance.images); "
+ "get_recording says what was found and done (enhanced). Editors, with a token that has the write scope.",
+ Arg("recording_id", "integer", "the image", required=True),
+ writes=True,
+)
+def enhance_image(ctx: Context, recording_id: int) -> dict[str, Any]:
+ if not ctx.user.can_write:
+ raise ToolError("this token is read-only: ask for a token with the write scope to enhance images")
+ ctx.acl.recording(recording_id, "process")
+ job = enhance.queue(ctx.db, recording_id, ctx.user.email)
+ auth.audit(ctx.db, ctx.user.as_audit(), "recording.enhance", f"recording:{recording_id}", {"job": job, "via": "mcp"})
+ return {"recording_id": recording_id, "job": job, "url": ctx.link(recording_id)}
+
+
# ---------- a namespace's own ways in for other apps (integrations.py) ----------
def _integrations(ctx: Context, namespace: str, sid: int) -> dict[str, Any]:
return {
diff --git a/fastapi_backend/app/api/media.py b/fastapi_backend/app/api/media.py
index 40875d780..f89c13755 100644
--- a/fastapi_backend/app/api/media.py
+++ b/fastapi_backend/app/api/media.py
@@ -22,7 +22,7 @@
from app.core.security import sign_path
from app.domain.store import API
-MEDIA_RX = re.compile(rf"^{re.escape(API)}/recordings/\d+/(audio|media|pdf|wordcloud\.svg|frames/[\w.-]+|files/\d+/download)$")
+MEDIA_RX = re.compile(rf"^{re.escape(API)}/recordings/\d+/(audio|media|pdf|enhanced|wordcloud\.svg|frames/[\w.-]+|files/\d+/download)$")
NS_MEDIA_RX = re.compile(rf"^{re.escape(API)}/namespaces/[a-z0-9][a-z0-9_-]*/wordcloud\.svg$")
REPORT_RX = re.compile(r"^/(reports|embed)/")
# the response fields the server fills with links (player audio, frames, face covers, posters, pages and their
diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py
index 4b4150488..ad84ecf9a 100644
--- a/fastapi_backend/app/api/v1/routes/recordings.py
+++ b/fastapi_backend/app/api/v1/routes/recordings.py
@@ -22,6 +22,7 @@
analyze,
auth,
deletion,
+ enhance,
hierarchy,
ipgroups,
jobs,
@@ -313,6 +314,7 @@ def get_recording(rid: int, acl: Acl, db: Db, cfg: Cfg) -> Recording:
if r.get("enrichment") and acl.member(r): # where a photo was taken: members only
d["enrichment"] = {k: v for k, v in r["enrichment"].items() if k != "photo_fp"}
d["nsfw"] = _nsfw(db, cfg, r, rid)
+ d["enhanced"] = enhance.brief(r.get("enhanced"), rid)
return Recording.model_validate(sign_urls(d, full=True))
@@ -617,6 +619,18 @@ def reprocess_recording(rid: int, acl: Acl, user: Writer, db: Db, body: Reproces
raise HTTPException(400, str(e)) from None
+@router.post("/{rid}/enhance")
+def enhance_recording(rid: int, acl: Acl, user: Writer, db: Db) -> JobQueued:
+ """Enhance an image whatever the settings and its quality (docs/enhance.md): an enhanced copy is made, and its
+ pipeline's steps after enhance (reading its text, faces, objects, …) run again on that copy. The original stays as
+ it is. 400 for anything but an image. Audited as `recording.enhance`."""
+ acl.recording(rid, "process")
+ with domain_errors():
+ job = enhance.queue(db, rid, user.email)
+ auth.audit(db, user.as_audit(), "recording.enhance", f"recording:{rid}", {"job": job})
+ return JobQueued(job=job)
+
+
@router.post("/{rid}/share")
def create_share(rid: int, acl: Acl, user: Writer, db: Db, body: ShareCreate | None = None) -> ShareLink:
"""A read-only link to this one recording, for people without an account, with a short ``/s/`` address too. Its
diff --git a/fastapi_backend/app/api/v1/routes/video.py b/fastapi_backend/app/api/v1/routes/video.py
index 6bef4583e..d346c63ec 100644
--- a/fastapi_backend/app/api/v1/routes/video.py
+++ b/fastapi_backend/app/api/v1/routes/video.py
@@ -19,7 +19,7 @@
from app.api.media import sign_urls
from app.api.streaming import LOCKED, picture_response, stored_file
from app.api.v1.routes.recordings import serve_audio
-from app.domain import auth, convert, documents, files, ingest, jobs, keyring, store, video
+from app.domain import auth, convert, documents, enhance, files, ingest, jobs, keyring, store, video
from app.domain import faces as facemod
from app.domain.store import API, DB, R
from app.schemas.common import Ok
@@ -81,6 +81,21 @@ def get_pdf(rid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, s: str = "")
return stored_file(db, cfg, path, request, "application/pdf", f"{name}.pdf", files.HEADERS)
+@router.get("/recordings/{rid}/enhanced", response_class=FileResponse, responses={200: {"content": {"image/jpeg": {}}}})
+def get_enhanced(rid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, s: str = "") -> Response:
+ """The enhanced copy the enhance step made of a low quality image (docs/enhance.md), to save. Members of its
+ namespace only (a bearer token, or a signed link the API gave one): visitors get its pages, with faces pixelated
+ where the namespace does that."""
+ rec = acl.recording(rid, share=s)
+ if not acl.member(rec):
+ raise HTTPException(404, "not found")
+ path = enhance.enhanced_path(cfg, rid)
+ if not path.is_file():
+ raise HTTPException(404, "not found")
+ name = pathlib.PurePosixPath(str((rec.get("remote") or {}).get("path") or rec.get("path") or "image")).stem or "image"
+ return stored_file(db, cfg, path, request, "image/jpeg", f"{name}-enhanced.jpg", files.HEADERS)
+
+
@router.get("/recordings/{rid}/frames/{name}", response_class=FileResponse, responses={200: {"content": {"image/jpeg": {}}}})
def get_frame(rid: int, name: str, acl: Acl, cfg: Cfg, db: Db, s: str = "") -> Response:
"""A still (shot frame, text-on-screen frame or face crop), or a document's or an image's page. Where the namespace
diff --git a/fastapi_backend/app/domain/actions_run.py b/fastapi_backend/app/domain/actions_run.py
index 2acb80162..e0a959185 100644
--- a/fastapi_backend/app/domain/actions_run.py
+++ b/fastapi_backend/app/domain/actions_run.py
@@ -1,5 +1,5 @@
-"""Automation actions that run Lens's own work (docs/automations.md, Actions): `run_workflow`, `run_pipeline` and
-`run_routine`.
+"""Automation actions that run Lens's own work (docs/automations.md, Actions): `run_workflow`, `run_pipeline`,
+`run_routine` and `enhance`.
Each changes something inside Lens (effect `lens`): a workflow or a pipeline on the asset the run is about, in the
automation's namespaces, runs without asking; on any other asset, and a routine always, it asks first. The work is
@@ -11,7 +11,7 @@
from __future__ import annotations
-from . import actions, budgets, jobs, pipelines, routines, workflows
+from . import actions, budgets, enhance, jobs, pipelines, routines, store, workflows
def _int(p, key, what):
@@ -62,6 +62,19 @@ def _run_pipeline(db, cfg, intent, user, dry):
return {**out, "job": jid}
+# ---------- enhance ----------
+def _enhance(db, cfg, intent, user, dry):
+ p = intent["params"]
+ acc = actions.person(db, intent, user)
+ rec = actions.asset_for(db, intent, acc, actions.asset_id(p.get("asset")), "process")
+ if (db.one("SELECT source FROM $r", r=store.R("recording", rec["id"])) or {}).get("source") != "image":
+ raise ValueError(f"{rec.get('title') or rec['id']} isn't an image: only images are enhanced so far")
+ out = {"asset": rec["id"]}
+ if dry:
+ return {**out, "job": None, "would": f"enhance {rec.get('title') or rec['id']} and read it again"}
+ return {**out, "job": enhance.queue(db, rec["id"], actions.actor(db, intent)["email"])}
+
+
# ---------- run_routine ----------
def _run_routine(db, cfg, intent, user, dry):
p = intent["params"]
@@ -145,6 +158,22 @@ def _run_routine(db, cfg, intent, user, dry):
)
)
+actions.register(
+ actions.ActionSpec(
+ name="enhance",
+ label="Enhance an image",
+ group="Run",
+ description="Makes an enhanced copy of an image (bigger, sharper, less noisy) and reads it again from that copy; "
+ "the original stays.",
+ params={"type": "object", "properties": {"asset": actions.asset_param("Image")}},
+ effect="lens",
+ subject=actions.asset_subject,
+ output={"type": "object", "properties": {"job": {"type": "integer"}, "asset": {"type": "integer"}}},
+ perform=_enhance,
+ describe=lambda p: f"Enhance {actions.on_asset(p)}",
+ )
+)
+
actions.register(
actions.ActionSpec(
name="run_routine",
diff --git a/fastapi_backend/app/domain/components.py b/fastapi_backend/app/domain/components.py
index 41183c279..e2bea00d9 100644
--- a/fastapi_backend/app/domain/components.py
+++ b/fastapi_backend/app/domain/components.py
@@ -75,6 +75,18 @@
"432763a6899ebc418c55f784b98f90565e5fc694c778d2ffcb0294b12f6a7404",
89,
),
+ # Real-ESRGAN general x4v3 (BSD-3-Clause), as Qualcomm AI Hub exports it to ONNX (its weights in a file of their
+ # own), at a fixed revision (enhance.py)
+ "realesr-general-x4v3/model.onnx": (
+ "https://huggingface.co/TheGuy444/Real-ESRGAN-ONNX/resolve/4c80f3da1a8a85f0757a9e7f79e08d0307220eb5/onnx/model.onnx",
+ "2d1810f8b75c6a899487decf933697241eafbe6327d7b723674ba6df73a08fda",
+ 1,
+ ),
+ "realesr-general-x4v3/model.data": (
+ "https://huggingface.co/TheGuy444/Real-ESRGAN-ONNX/resolve/4c80f3da1a8a85f0757a9e7f79e08d0307220eb5/onnx/model.data",
+ "512d0ec9940c2e9d85d27f2952f12a0b77b7841dc22df4ce9f3ea458bc98f37f",
+ 5,
+ ),
}
@@ -415,6 +427,11 @@ class Faces(Component):
def _files(self, cfg):
return FACE_FILES[1:] if cfg["video"].get("face_engine") == "anytopdf" else FACE_FILES
+ def serves(self, cfg, m):
+ from . import enhance
+
+ return self.steps | ({"enhance"} if enhance.wants_faces(cfg) else set()) # it looks for faces first
+
def needed(self, cfg, m):
v = cfg["video"]
if v.get("face_engine") == "anytopdf":
@@ -474,6 +491,31 @@ def fetch(self, cfg, m, say):
fetch_file(cfg, "vit-base-nsfw-detector-int8.onnx", say)
+class Enhance(Component):
+ """Real-ESRGAN and ONNX Runtime (enhance.py), when Settings → Enhance chooses it for images."""
+
+ kind = "model"
+
+ def needed(self, cfg, m):
+ from . import enhance
+
+ o = enhance.options(cfg)
+ return bool(o["images"] and o["image_engine"] == "realesrgan")
+
+ def present(self, cfg):
+ from . import enhance
+
+ return importable("onnxruntime") and bool(enhance.model_path(cfg))
+
+ def fetch(self, cfg, m, say):
+ from . import enhance
+
+ if not importable("onnxruntime"):
+ pip_install(cfg, "objects", say) # the objects extra is ONNX Runtime alone
+ for name in enhance.MODEL_FILES:
+ fetch_file(cfg, name, say)
+
+
class InjectionModel(Component):
"""The prompt-injection classifier (injection_model.py), fetched when ai.injection_check asks for it."""
@@ -744,6 +786,14 @@ def detail(self, cfg):
size_mb=105,
license="Apache-2.0",
),
+ Enhance(
+ "enhance",
+ "Real-ESRGAN",
+ "enhances low quality images before they're read (enhance.image_engine realesrgan)",
+ steps={"enhance"},
+ size_mb=6,
+ license="BSD-3-Clause",
+ ),
LAYA,
InjectionModel(
"injection-model",
diff --git a/fastapi_backend/app/domain/deletion.py b/fastapi_backend/app/domain/deletion.py
index ea6020d80..b6bb8ae99 100644
--- a/fastapi_backend/app/domain/deletion.py
+++ b/fastapi_backend/app/domain/deletion.py
@@ -81,12 +81,20 @@ def stop_jobs(db, rid, verb="delete"):
def _files(db, cfg, rec, rid, ns):
- """The files Lens made from the recording: its frames, its report pages, the PDF made of a document, the exports
- only it wrote, and a cached copy of a remote file that no other recording uses. Never the media file."""
+ """The files Lens made from the recording: its frames, its report pages, the PDF made of a document, its enhanced
+ copy, the exports only it wrote, and a cached copy of a remote file that no other recording uses. Never the media
+ file."""
+ from . import enhance
+
data = pathlib.Path(cfg["data_dir"])
reports = data / "reports" / (ns or "_")
name = f"{render.slug(rec.get('title'))}-{rid}"
- out = [reports / f"{name}.html", *reports.glob(f"{name}--*.html"), data / "renditions" / f"{int(rid)}.pdf"]
+ out = [
+ reports / f"{name}.html",
+ *reports.glob(f"{name}--*.html"),
+ data / "renditions" / f"{int(rid)}.pdf",
+ enhance.enhanced_path(cfg, rid),
+ ]
exports = data / "exports" / (ns or "_")
mine = {
(o.get("value") or {}).get("file")
diff --git a/fastapi_backend/app/domain/documents.py b/fastapi_backend/app/domain/documents.py
index 3e3139faa..53d0d1f1c 100644
--- a/fastapi_backend/app/domain/documents.py
+++ b/fastapi_backend/app/domain/documents.py
@@ -19,7 +19,7 @@
import tempfile
import xml.etree.ElementTree as ET
-from . import anytopdf, convert, ingest, keyring, store, tika, video, webcapture
+from . import anytopdf, convert, enhance, ingest, keyring, store, tika, video, webcapture
R = store.R
TYPES = {
@@ -484,6 +484,9 @@ def transcribe(db, cfg, rid, say):
path = ingest.audio_path(db, cfg, rec)
if not path or not os.path.exists(path):
raise FileNotFoundError(f"its file isn't there ({rec.get('path')})")
+ if rec["source"] == "image" and (better := enhance.picture(db, cfg, rid, path)) != path:
+ path = better # the enhance step's copy of a low quality image; the original stays as it was
+ say("read from its enhanced copy")
opts, (engine, why) = cfg["documents"], video.ocr_engine_why(cfg)
learnt, pdf, source, read = {}, path, rec["source"], None
if source == "document" and convert.needs(path):
diff --git a/fastapi_backend/app/domain/enhance.py b/fastapi_backend/app/domain/enhance.py
new file mode 100644
index 000000000..f6aa02c1c
--- /dev/null
+++ b/fastapi_backend/app/domain/enhance.py
@@ -0,0 +1,415 @@
+"""Enhancing low quality pictures (docs/enhance.md): the enhance step looks at an image before it's read and, when it's
+low quality, makes an enhanced copy that the read step (OCR), faces, objects, NSFW and descriptions then work from. The
+original stays as it is: the copy is a rendition, data_dir/renditions/-enhanced.jpg (encrypted where the
+namespace's files are), which the asset page offers to save.
+
+Off by default: Settings → Enhance, `enhance.images`. Run with `force` (an agent's `enhance` tool, POST
+/resources/{rid}/enhance, the `enhance` automation action, or a pipeline step that says so), it enhances an image
+whatever the settings and its quality.
+
+What counts as low quality (each a reason kept in recording.enhanced.low):
+- `small`: its shorter side is under enhance.min_side pixels.
+- `blurry`: the variance of its Laplacian (how sharply its tones change), where it's sharpest, is under
+ enhance.min_sharpness.
+- `noisy`: its noise, estimated with Immerkær's filter where it's plainest (so edges and texture don't count), is over
+ NOISE levels.
+- `dull`: its tones' standard deviation is under DULL: low contrast, or too dark or too bright.
+- `small_faces`: a face is under enhance.min_face_px tall, where the namespace finds faces (faces.mode isn't off), so
+ recognition has more to go on. With enhance.only_faces, only pictures with a face are enhanced at all.
+
+It's made bigger to bring its shorter side to min_side and its smallest face to min_face_px, at most enhance.max_scale
+times and MAX_SIDE pixels. The engine (`enhance.image_engine`):
+- `classic` (the default): Pillow, nothing to fetch. A noisy picture is smoothed with a median filter, enlarged with
+ Lanczos, a dull one stretched to its full range of tones, and a soft or enlarged one sharpened (unsharp mask).
+- `realesrgan`: Real-ESRGAN general x4v3 (BSD-3-Clause, 5 MB, as Qualcomm AI Hub exports it to ONNX), run by ONNX
+ Runtime on the CPU in tiles of 128 pixels. It enlarges four times while taking out blur, noise and JPEG blocks, and
+ the result is brought to the size wanted. components.py fetches it and ONNX Runtime the first time the step runs with
+ it chosen. A picture over REAL_MAX_PIXELS goes to classic instead, so a small machine isn't swamped.
+
+What a resource keeps (recording.enhanced): `kind`, `low` (the reasons), `measured` (width, height, sharpness, noise,
+contrast, brightness, and faces and smallest_face where they were looked for), `enhanced` (whether a copy was made),
+and for a copy its `engine`, `scale`, `width`, `height` and `forced`; `at`.
+"""
+
+from __future__ import annotations
+
+import io
+import os
+import pathlib
+import threading
+
+import numpy as np
+
+from . import convert, ingest, jobs, keyring, store
+
+R = store.R
+ENGINES = ("classic", "realesrgan")
+MODEL_NAME = "realesr-general-x4v3"
+MODEL_FILES = (f"{MODEL_NAME}/model.onnx", f"{MODEL_NAME}/model.data") # components.FILES fetches them
+MODELS = pathlib.Path("/opt/lens/models") # models put there by hand, or baked into an image
+TILE, PAD, SR = 128, 8, 4 # the model's input, the overlap around each tile, and how much bigger it makes them
+REAL_MAX_PIXELS = 1_500_000 # pictures larger than this are enhanced by classic (four times as big would be 24 MP)
+MAX_SIDE = 4000 # an enhanced picture's longest side
+MEASURE_SIDE = 2048 # a larger picture is measured at this size
+NOISE = 6.0 # noise (levels out of 255) over this is noisy
+DULL = 30.0 # a standard deviation of tones under this is dull
+MIN_SCALE = 1.1 # enlarging less than this isn't worth it
+QUALITY = 92 # JPEG quality of the enhanced copy
+LIMITS = {"min_side": (64, 4000), "min_sharpness": (0, 10000), "min_face_px": (0, 1024), "max_scale": (1, 4)}
+REASONS = {"small": "small", "blurry": "blurry", "noisy": "noisy", "dull": "low contrast", "small_faces": "faces too small"}
+_SESSIONS: dict[str, object] = {}
+_LOCK = threading.Lock()
+
+
+def options(cfg):
+ return {**store.DEFAULTS["enhance"], **(cfg.get("enhance") or {})}
+
+
+def check(key, value):
+ """A setting's value, checked: ValueError naming the problem."""
+ if key in ("images", "only_faces"):
+ if not isinstance(value, bool):
+ raise ValueError(f"enhance.{key} is true or false")
+ return value
+ if key == "image_engine":
+ if value not in ENGINES:
+ raise ValueError(f"enhance.image_engine is one of {', '.join(ENGINES)}")
+ return value
+ if key in LIMITS:
+ lo, hi = LIMITS[key]
+ if not (isinstance(value, (int, float)) and not isinstance(value, bool) and lo <= value <= hi):
+ raise ValueError(f"enhance.{key} is a number from {lo} to {hi}")
+ return int(value) if key in ("min_side", "min_face_px") else round(float(value), 2)
+ raise ValueError(f"there's no enhance setting called {key}")
+
+
+def wants_faces(cfg):
+ """Whether the step looks for faces: what has it wait for the face models (components.py)."""
+ o = options(cfg)
+ return bool(o["images"] and (o["only_faces"] or o["min_face_px"]))
+
+
+# ---------- the copy ----------
+def enhanced_path(cfg, rid):
+ return pathlib.Path(cfg["data_dir"]) / "renditions" / f"{int(rid)}-enhanced.jpg"
+
+
+def files(cfg, rid):
+ """The enhanced copies a resource has, as paths."""
+ p = enhanced_path(cfg, rid)
+ return [p] if p.is_file() else []
+
+
+def picture(db, cfg, rid, path):
+ """The file an image is read from: its enhanced copy when the enhance step made one, else its own file."""
+ p = enhanced_path(cfg, rid)
+ return keyring.working_copy(db, cfg, str(p)) if p.is_file() else path
+
+
+def clear(db, cfg, rid):
+ """Forget a resource's enhanced copy: the original is read again."""
+ for p in files(cfg, rid):
+ p.unlink(missing_ok=True)
+ db.q("UPDATE $r SET enhanced = NONE", r=R("recording", rid))
+
+
+def brief(e, rid):
+ """A resource's enhancement as the API shows it, with a link to save its copy."""
+ if not e:
+ return None
+ out = dict(e)
+ if e.get("enhanced"):
+ out["download"] = f"{store.API}/recordings/{int(rid)}/enhanced"
+ return out
+
+
+# ---------- how good it looks ----------
+def measure(img):
+ """How good an RGB picture looks: its size, sharpness (the variance of the Laplacian where it is sharpest), noise (Immerkær's estimate, out
+ of 255), contrast (standard deviation of its tones) and brightness (their mean)."""
+ from PIL import Image
+ from scipy import ndimage
+
+ w, h = img.size
+ g = img.convert("L")
+ if max(w, h) > MEASURE_SIDE:
+ g.thumbnail((MEASURE_SIDE, MEASURE_SIDE), Image.LANCZOS)
+ a = np.asarray(g, dtype=np.float32)
+ noise = 0.0
+ if min(a.shape) > 2:
+ k = np.array([[1, -2, 1], [-2, 4, -2], [1, -2, 1]], dtype=np.float32)
+ r = np.abs(ndimage.convolve(a, k))[1:-1, 1:-1]
+ g = np.hypot(ndimage.sobel(a, 0), ndimage.sobel(a, 1))[1:-1, 1:-1]
+ # where it's plainest (the flattest 30%), so edges and texture aren't taken for noise: the filter's response
+ # to noise of deviation s has deviation 6s, and its median magnitude is 0.6745 of that
+ noise = float(np.median(r[g <= np.percentile(g, 30)])) / (6 * 0.6745)
+ return {
+ "width": w,
+ "height": h,
+ "sharpness": round(_sharpness(ndimage.laplace(a)), 1),
+ "noise": round(noise, 2),
+ "contrast": round(float(a.std()), 1),
+ "brightness": round(float(a.mean()), 1),
+ }
+
+
+def _sharpness(lap):
+ """The variance of the Laplacian where the picture is sharpest: block by block, the 90th percentile, so a sharp
+ subject on a soft background, or a clean drawing with wide plain areas, isn't taken for blurry."""
+ n = max(16, min(64, min(lap.shape) // 4))
+ rows, cols = lap.shape[0] // n, lap.shape[1] // n
+ if rows * cols < 4:
+ return float(lap.var())
+ blocks = lap[: rows * n, : cols * n].reshape(rows, n, cols, n).var(axis=(1, 3))
+ return float(np.percentile(blocks, 90))
+
+
+def assess(m, faces, opts):
+ """(the reasons a picture is low quality, how many times bigger to make it): `faces`, the heights of its faces in
+ pixels, or None where they weren't looked for."""
+ low, scale = [], 1.0
+ w, h = m["width"], m["height"]
+ if min(w, h) < opts["min_side"]:
+ low.append("small")
+ scale = max(scale, opts["min_side"] / min(w, h))
+ if m["sharpness"] < opts["min_sharpness"]:
+ low.append("blurry")
+ if m["noise"] > NOISE:
+ low.append("noisy")
+ if m["contrast"] < DULL:
+ low.append("dull")
+ if faces and opts["min_face_px"] and min(faces) < opts["min_face_px"]:
+ low.append("small_faces")
+ scale = max(scale, opts["min_face_px"] / max(1, min(faces)))
+ scale = min(scale, opts["max_scale"], MAX_SIDE / max(w, h))
+ return low, round(scale, 2) if scale >= MIN_SCALE else 1.0
+
+
+def _face_heights(db, cfg, sid, img):
+ """(the heights of the faces on a picture in pixels, None) or (None, why they weren't looked for)."""
+ from . import faces, video
+
+ if faces.mode(db, sid) == "off":
+ return None, "face detection is off in this namespace"
+ engine = video.face_engine(cfg)
+ if not engine:
+ return None, "no face engine is here yet (Settings → Video)"
+ with keyring.scratch(cfg) as d:
+ p = d / "picture.png"
+ img.save(p)
+ found = engine.faces(str(p))
+ return [max(1, round(f["box"][3] * img.height)) for f in found], None
+
+
+# ---------- engines ----------
+def model_path(cfg):
+ """Real-ESRGAN's model: one put in MODELS by hand, else the one components.py fetched; None without one (its
+ weights are a file of their own beside it)."""
+ from . import components
+
+ for d in (MODELS, components.models_dir(cfg)):
+ if all((d / f).is_file() for f in MODEL_FILES):
+ return str(d / MODEL_FILES[0])
+ return None
+
+
+def _session(path):
+ import onnxruntime as ort
+
+ with _LOCK: # one per process
+ if path not in _SESSIONS:
+ _SESSIONS[path] = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
+ return _SESSIONS[path]
+
+
+class RealEsrgan:
+ name = MODEL_NAME
+
+ def __init__(self, cfg):
+ path = model_path(cfg)
+ if not path:
+ raise RuntimeError("Real-ESRGAN isn't here yet (it's fetched the first time the step runs with it chosen)")
+ try:
+ import onnxruntime # noqa: F401
+ except ImportError:
+ raise RuntimeError("ONNX Runtime isn't installed (it's fetched on first use, or pip install \"lens[objects]\")") from None
+ self.session = _session(path)
+ self.input = self.session.get_inputs()[0].name
+
+ def upscale(self, img):
+ """An RGB picture four times as big, tile by tile: each tile overlaps its neighbours by PAD pixels, which are
+ cut off again, so no seams show."""
+ from PIL import Image
+
+ a = np.asarray(img, dtype=np.float32) / 255.0
+ h, w = a.shape[:2]
+ step = TILE - 2 * PAD
+ H, W = -(-h // step) * step, -(-w // step) * step
+ padded = np.pad(a, ((PAD, H - h + PAD), (PAD, W - w + PAD), (0, 0)), mode="edge")
+ out = np.empty((H * SR, W * SR, 3), dtype=np.uint8)
+ inner = slice(PAD * SR, (PAD + step) * SR)
+ for y in range(0, H, step):
+ for x in range(0, W, step):
+ tile = np.ascontiguousarray(padded[y : y + TILE, x : x + TILE].transpose(2, 0, 1)[None])
+ got = self.session.run(None, {self.input: tile})[0][0][:, inner, inner]
+ out[y * SR : (y + step) * SR, x * SR : (x + step) * SR] = (got.transpose(1, 2, 0).clip(0, 1) * 255 + 0.5).astype(np.uint8)
+ return Image.fromarray(out[: h * SR, : w * SR])
+
+
+def _size(img, scale):
+ return max(1, round(img.width * scale)), max(1, round(img.height * scale))
+
+
+def classic(img, low, scale):
+ """Pillow alone: smoothed if noisy, enlarged, stretched if dull, sharpened if soft or enlarged."""
+ from PIL import Image, ImageFilter, ImageOps
+
+ if "noisy" in low:
+ img = img.filter(ImageFilter.MedianFilter(3))
+ if scale > 1:
+ img = img.resize(_size(img, scale), Image.LANCZOS)
+ if "dull" in low:
+ img = ImageOps.autocontrast(img, cutoff=1, preserve_tone=True)
+ if "blurry" in low or scale > 1:
+ img = img.filter(ImageFilter.UnsharpMask(radius=2, percent=150 if "blurry" in low else 80, threshold=3))
+ return img
+
+
+def image_engine(cfg, img):
+ """(Real-ESRGAN, None) where it's chosen and can take this picture, else (None, why classic does it)."""
+ if options(cfg)["image_engine"] != "realesrgan":
+ return None, None
+ if img.width * img.height > REAL_MAX_PIXELS:
+ return None, f"it's over {REAL_MAX_PIXELS // 1_000_000:g} MP, too large for Real-ESRGAN here: enhanced by classic"
+ try:
+ return RealEsrgan(cfg), None
+ except RuntimeError as e:
+ return None, f"{e}: enhanced by classic"
+
+
+def enhance_image(cfg, img, low, scale, say):
+ """(the enhanced picture, the engine's name)."""
+ from PIL import Image, ImageOps
+
+ real, why = image_engine(cfg, img)
+ if why:
+ say(why)
+ if not real:
+ return classic(img, low, scale), "classic"
+ big = real.upscale(img)
+ if scale != SR:
+ big = big.resize(_size(img, scale), Image.LANCZOS)
+ if "dull" in low:
+ big = ImageOps.autocontrast(big, cutoff=1, preserve_tone=True)
+ return big, real.name
+
+
+# ---------- the step ----------
+def _keep(db, rid, e):
+ db.q("UPDATE $r SET enhanced = $e", r=R("recording", rid), e=store.clean(e))
+
+
+def _said(m):
+ return f"{m['width']}×{m['height']}, sharpness {m['sharpness']:g}, noise {m['noise']:g}, contrast {m['contrast']:g}"
+
+
+def step(db, cfg, rid, say, spec=None):
+ """The enhance step: an image that looks low quality gets an enhanced copy, which it's read from."""
+ rec = db.one("SELECT space, path, remote, source, enhanced FROM $r", r=R("recording", rid)) or {}
+ force = bool((spec or {}).get("force"))
+ if rec.get("source") != "image":
+ raise jobs.Skip("only images are enhanced so far")
+ opts = options(cfg)
+ if not (opts["images"] or force):
+ if (rec.get("enhanced") or {}).get("forced"): # someone asked for this one: kept
+ raise jobs.Skip("enhancing images is off (Settings → Enhance); kept the copy made on request")
+ clear(db, cfg, rid)
+ raise jobs.Skip("enhancing images is off (Settings → Enhance)")
+ path = ingest.audio_path(db, cfg, rec)
+ if not path or not os.path.exists(path):
+ raise FileNotFoundError(f"its file isn't there ({rec.get('path')})")
+ if convert.ext_of(path) in store.CAMERA_EXT:
+ clear(db, cfg, rid)
+ raise jobs.Skip("camera RAW and HEIC photos aren't enhanced (anytopdf reads them as they are)")
+ img = _open(path)
+ if img is None:
+ clear(db, cfg, rid)
+ raise jobs.Skip("it has several frames (a multi-page TIFF or an animation): only single pictures are enhanced")
+ heights = None
+ if opts["only_faces"] or opts["min_face_px"]:
+ heights, why = _face_heights(db, cfg, rec["space"], img)
+ if opts["only_faces"] and not force and not heights:
+ clear(db, cfg, rid)
+ raise jobs.Skip(f"{why} (enhance.only_faces)" if heights is None else "there's no face in it (enhance.only_faces)")
+ m = measure(img)
+ if heights is not None:
+ m["faces"] = len(heights)
+ if heights:
+ m["smallest_face"] = min(heights)
+ low, scale = assess(m, heights, opts)
+ now = store.now()
+ if scale == 1 and set(low) <= {"small", "small_faces"}: # only enlarging would help, and max_scale is 1
+ low = []
+ if not low and not force:
+ for p in files(cfg, rid):
+ p.unlink(missing_ok=True)
+ _keep(db, rid, {"kind": "image", "low": [], "measured": m, "enhanced": False, "at": now})
+ return say(f"good enough as it is ({_said(m)})")
+ out, engine = enhance_image(cfg, img, low, scale, say)
+ buf = io.BytesIO()
+ out.save(buf, "JPEG", quality=QUALITY)
+ keyring.keep(db, cfg, rec["space"], enhanced_path(cfg, rid), buf.getvalue())
+ _keep(
+ db,
+ rid,
+ {
+ "kind": "image",
+ "low": low,
+ "measured": m,
+ "enhanced": True,
+ "engine": engine,
+ "scale": scale,
+ "width": out.width,
+ "height": out.height,
+ "forced": force or None,
+ "at": now,
+ },
+ )
+ why = ", ".join(REASONS[x] for x in low) if low else "on request"
+ say(f"enhanced ({why}): {m['width']}×{m['height']} to {out.width}×{out.height} by {engine}")
+
+
+def _open(path):
+ """An image file as one flat RGB picture (documents.py's way: its EXIF orientation, transparency on white), or
+ None when it has several frames."""
+ from PIL import Image
+
+ from . import documents
+
+ old, Image.MAX_IMAGE_PIXELS = Image.MAX_IMAGE_PIXELS, documents.MAX_PIXELS
+ try:
+ with Image.open(path) as im:
+ if getattr(im, "n_frames", 1) > 1:
+ return None
+ return documents._flat(im)
+ except Image.DecompressionBombError:
+ raise ValueError(f"this image is too large to open (more than {documents.MAX_PIXELS // 1_000_000} million pixels)") from None
+ except OSError as e:
+ raise ValueError(f"this image can't be read ({e})") from None
+ finally:
+ Image.MAX_IMAGE_PIXELS = old
+
+
+def queue(db, rid, by=None):
+ """Enhance an image whatever the settings and its quality, then run what reads it again: the steps of its pipeline
+ after enhance (the read step, faces, objects, …). The job's id."""
+ from . import content_types, pipelines
+
+ rec = db.one("SELECT space, source FROM $r", r=R("recording", rid))
+ if not rec:
+ raise KeyError(rid)
+ if rec.get("source") != "image":
+ raise ValueError("only images are enhanced so far")
+ steps, _ = pipelines.resolve(db, rec["space"], None, content_types.of_recording(db, rid)[0])
+ rest = [s for s in steps if (s if isinstance(s, str) else s.get("type")) != "enhance"]
+ return jobs.add_steps(db, rid, [{"type": "enhance", "force": True}, *rest], by)
diff --git a/fastapi_backend/app/domain/jobs.py b/fastapi_backend/app/domain/jobs.py
index bf3b8e859..ef8fb000b 100644
--- a/fastapi_backend/app/domain/jobs.py
+++ b/fastapi_backend/app/domain/jobs.py
@@ -17,14 +17,38 @@
from . import activity, analyze, ingest, keyring, pipelines, render, speakers as spk, store, telemetry
R = store.R
-PIPELINE = ["transcribe", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe", "analyze", "embed", "summarize", "report"]
+PIPELINE = [
+ "enhance",
+ "transcribe",
+ "diarize",
+ "shots",
+ "ocr",
+ "faces",
+ "objects",
+ "nsfw",
+ "describe",
+ "analyze",
+ "embed",
+ "summarize",
+ "report",
+]
AFTER_IMPORT = ["analyze", "embed", "summarize", "report"]
ACTIVE = ["queued", "running", "held"] # held: over budget, waiting for someone to pick (budgets.py)
FIELDS = (
"record::id(id) AS id, recording, space, batch, pipeline, steps, step_index, next_step, status, worker, error, attempts, "
"created_by, created_at, started_at, finished_at, updated_at, cancel_requested, log_total, cost_usd, tokens, cost_estimate, hold"
)
-MEDIA_STEPS = {"transcribe", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe"} # they take longer the longer the recording
+MEDIA_STEPS = {
+ "enhance",
+ "transcribe",
+ "diarize",
+ "shots",
+ "ocr",
+ "faces",
+ "objects",
+ "nsfw",
+ "describe",
+} # they take longer the longer the recording
FILED = ("audio", "document", "image") # sources with a file of their own for the steps to work on
TIMINGS = 25 # recent timings kept per kind of step, for estimates
@@ -41,6 +65,13 @@ def _spec(s):
return {"type": s} if isinstance(s, str) else dict(s)
+def _enhance(db, cfg, rid, say, spec=None):
+ """A low quality image's enhanced copy, which the steps after it read (enhance.py)."""
+ from . import enhance
+
+ enhance.step(db, cfg, rid, say, spec)
+
+
def _read(db, cfg, rid, say, spec=None):
"""A document's or an image's step: draw its pages and read their text (documents.py). It needs no speech model,
so a document never waits for one to be fetched."""
@@ -183,6 +214,7 @@ def _describe(db, cfg, rid, say, spec=None):
VIDEO_STEPS = {"shots", "ocr", "faces", "objects", "describe"} # added after workers.steps lists were first written
STEPS = {
+ "enhance": _enhance,
"transcribe": _transcribe,
"read": _read,
"diarize": _diarize,
@@ -203,11 +235,18 @@ def _describe(db, cfg, rid, say, spec=None):
}
+# The sources the enhance step works on (docs/enhance.md).
+ENHANCED = {"image"}
+
+
def for_source(steps, source):
"""A pipeline's steps as they run on a resource of this source: a document or an image is read, not transcribed,
and has no voices to tell apart, so a speech or voice model is never fetched for it. An imported transcript has
nothing to transcribe and its speakers came with it, so it skips both steps too (unless they're all that was asked
- for: those still run, to say why there's nothing to do)."""
+ for: those still run, to say why there's nothing to do). Only images are enhanced so far, so the others skip enhance
+ the same way."""
+ if source not in ENHANCED:
+ steps = [s for s in steps if s.get("type") != "enhance"] or steps
if source == "transcript":
return [s for s in steps if s.get("type") not in ("transcribe", "diarize")] or steps
if source not in ("document", "image"):
@@ -733,10 +772,10 @@ def migrate_document_steps(db):
a=ACTIVE,
)
for j in rows:
- if _source(db, j["recording"]) not in ("document", "image"):
+ if (src := _source(db, j["recording"])) not in ("document", "image"):
continue
i, steps = j.get("step_index") or 0, [_spec(s) for s in j.get("steps") or []]
- left = for_source(steps[i:], "document")
+ left = for_source(steps[i:], src)
if left == steps[i:] or not left:
continue
db.q(
@@ -913,6 +952,8 @@ def __init__(self, db, cfg_fn, name=None, steps=None, log=None):
self.can.add("embed")
if not steps and "objects" in self.can: # NSFW classification runs on ONNX Runtime too; older lists run it as well
self.can.add("nsfw")
+ if not steps and self.can & {"transcribe", "read"}: # enhancing comes first; lists written before it run it too
+ self.can.add("enhance")
if not steps and log and (missing := sorted(set(STEPS) - self.can)):
log(f"worker {self.name}: workers.steps leaves out {', '.join(missing)}; jobs with those steps wait for another worker")
self.was_paused = False
diff --git a/fastapi_backend/app/domain/keyring.py b/fastapi_backend/app/domain/keyring.py
index f5796a454..c3ee69328 100644
--- a/fastapi_backend/app/domain/keyring.py
+++ b/fastapi_backend/app/domain/keyring.py
@@ -794,6 +794,17 @@ def plain_picture(db, cfg, path):
shutil.rmtree(d, ignore_errors=True)
+@contextlib.contextmanager
+def scratch(cfg):
+ """A private folder for plain files a step needs for a moment (a picture handed to a tool), removed straight
+ after."""
+ d = pathlib.Path(tempfile.mkdtemp(dir=_work_dir(cfg).parent, prefix="pic-"))
+ try:
+ yield d
+ finally:
+ shutil.rmtree(d, ignore_errors=True)
+
+
def _make(db, cfg, path, folder, out):
if out.exists():
return
@@ -875,12 +886,16 @@ def stored_files(db, cfg):
def made_files(db, cfg):
"""(space, path) for the files Lens makes from them: documents' PDF renditions, frames, page images and face
- crops, reports and exports."""
+ crops, enhanced copies, reports and exports."""
+ from . import enhance
+
data = pathlib.Path(cfg["data_dir"])
for r in db.rows("SELECT record::id(id) AS id, space, source FROM recording"):
p = data / "renditions" / f"{int(r['id'])}.pdf"
if r.get("source") == "document" and p.is_file():
yield r["space"], str(p)
+ for p in enhance.files(cfg, r["id"]): # an enhanced copy (enhance.py)
+ yield r["space"], str(p)
d = data / "frames" / str(int(r["id"]))
if d.is_dir():
for f in sorted(d.glob("*.jpg")):
@@ -908,9 +923,9 @@ def made_files(db, cfg):
def recording_files(db, cfg, rid):
"""The files Lens keeps of one recording, where they are for its namespace now: its own file under data_dir, its
- supplementary files, its rendition, frames, page images and face crops, its cached copy from a storage source,
- and its report pages and exports."""
- from . import files as filemod, render, sources
+ supplementary files, its rendition and enhanced copy, frames, page images and face crops, its cached copy from a
+ storage source, and its report pages and exports."""
+ from . import enhance, files as filemod, render, sources
rec = db.one("SELECT title, space, path, remote, source FROM $r", r=store.R("recording", rid))
if not rec:
@@ -927,6 +942,7 @@ def recording_files(db, cfg, rid):
p = data / "renditions" / f"{int(rid)}.pdf"
if p.is_file():
yield str(p)
+ yield from (str(p) for p in enhance.files(cfg, rid))
d = data / "frames" / str(int(rid))
if d.is_dir():
yield from (str(f) for f in sorted(d.glob("*.jpg")))
diff --git a/fastapi_backend/app/domain/ops_tools.py b/fastapi_backend/app/domain/ops_tools.py
index 098bea380..44c363e74 100644
--- a/fastapi_backend/app/domain/ops_tools.py
+++ b/fastapi_backend/app/domain/ops_tools.py
@@ -27,6 +27,7 @@
"analysis",
"graph",
"video",
+ "enhance",
"uploads",
"files",
"ai",
diff --git a/fastapi_backend/app/domain/pipelines.py b/fastapi_backend/app/domain/pipelines.py
index 1239ae528..2998ae094 100644
--- a/fastapi_backend/app/domain/pipelines.py
+++ b/fastapi_backend/app/domain/pipelines.py
@@ -1,5 +1,6 @@
"""Pipelines: named, versioned lists of steps a recording goes through. Each namespace can choose its default; without
-one, recordings get the standard pipeline. Steps: transcribe (speech to text), read (a document's or an
+one, recordings get the standard pipeline. Steps: enhance (an enhanced copy of a low quality image, enhance.py, which
+the steps after it read), transcribe (speech to text), read (a document's or an
image's pages and their text), diarize, shots, ocr, faces, objects, nsfw (NSFW classification, nsfw.py), describe,
analyze, embed (passages for search by meaning, semantic.py), summarize, llm (a prompt template whose structured
result is saved as a named output), report (built in, or from a report template) and export (a template rendered to
@@ -23,14 +24,39 @@
from . import keyring, llm, render, sources, store, templates
R = store.R
-STANDARD = ["transcribe", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe", "analyze", "embed", "summarize", "report"]
+STANDARD = [
+ "enhance",
+ "transcribe",
+ "diarize",
+ "shots",
+ "ocr",
+ "faces",
+ "objects",
+ "nsfw",
+ "describe",
+ "analyze",
+ "embed",
+ "summarize",
+ "report",
+]
TYPES = {
- "transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe", "analyze", "embed", "summarize", "llm",
+ "enhance", "transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe", "analyze", "embed", "summarize", "llm",
"report",
"export",
"workflow",
} # fmt: skip
-ASSET_STEPS = ("transcribe", "read", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe") # they make something of the media
+ASSET_STEPS = (
+ "enhance",
+ "transcribe",
+ "read",
+ "diarize",
+ "shots",
+ "ocr",
+ "faces",
+ "objects",
+ "nsfw",
+ "describe",
+) # they make something of the media
KEYS = {"type", "name", "when", "template", "version", "key", "filename", "destination", "model", "force", "workflow"}
NODE_RX = re.compile(r"^[A-Za-z0-9_-]{1,40}$")
WHEN = {"min_minutes": (int, float), "max_minutes": (int, float), "source": str, "languages": list}
diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py
index 72573411b..9d0604a2a 100644
--- a/fastapi_backend/app/domain/settings.py
+++ b/fastapi_backend/app/domain/settings.py
@@ -103,6 +103,7 @@
"object_min_score",
),
"nsfw": None,
+ "enhance": None,
"server": ("embed_frame_ancestors", "max_upload_mb", "allowed_hosts", "session_hours", "secure_cookies", "trusted_proxies"),
"uploads": None,
"files": None,
@@ -161,6 +162,7 @@
("video", "ocr_engine"): {"auto", "tesseract", "apple-vision", "rapidocr", "doctr", "none"},
("video", "face_engine"): {"opencv", "insightface", "anytopdf", "none"},
("video", "object_engine"): {"yolox", "ultralytics", "anytopdf", "off"},
+ ("enhance", "image_engine"): {"classic", "realesrgan"},
("decisions", "engine"): {"auto", "jev", "laya", "llm", "off"},
("voice", "input"): {"auto", "server", "browser"},
("voice", "stt"): {"same", "sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"},
@@ -498,6 +500,10 @@ def _check(section, key, value, default):
return _component_setting(key, value)
if section == "nsfw":
return _nsfw_setting(key, value)
+ if section == "enhance" and key != "image_engine":
+ from . import enhance
+
+ return enhance.check(key, value)
if section == "mail" and key != "security":
return _mail_setting(key, value)
if section == "bridge" and key not in ("answer", "enabled", "approve"):
diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py
index 8608aa685..3e88255f4 100644
--- a/fastapi_backend/app/domain/store.py
+++ b/fastapi_backend/app/domain/store.py
@@ -255,6 +255,7 @@ def labels():
"stale_minutes": 15,
"max_attempts": 3,
"steps": [
+ "enhance",
"transcribe",
"read",
"diarize",
@@ -510,6 +511,21 @@ def labels():
# each picture; a resource is flagged when one scores at least the threshold. display is how flagged resources show
# in the web app: blur (click to see), hide (left out of the Library unless asked for) or show.
"nsfw": {"enabled": False, "threshold": 0.8, "display": "blur", "max_frames": 300},
+ # enhancing low quality images before they're read (enhance.py, docs/enhance.md): off until an admin turns it on.
+ # An image smaller than min_side (its shorter side, in pixels), blurrier than min_sharpness (the variance of its
+ # Laplacian), noisy, dull, or with a face under min_face_px tall gets an enhanced copy, at most max_scale times
+ # bigger, which the steps after it read; the original stays. image_engine: classic (Pillow, nothing to fetch) or
+ # realesrgan (Real-ESRGAN general x4v3, BSD-3-Clause, 5 MB, fetched on first use). only_faces: only pictures with a
+ # face, where the namespace finds faces.
+ "enhance": {
+ "images": False,
+ "image_engine": "classic",
+ "only_faces": False,
+ "min_side": 720,
+ "min_sharpness": 100,
+ "min_face_px": 64,
+ "max_scale": 4,
+ },
# rclone and local_roots are bootstrap-only on purpose: the web app must not be able to pick an executable
# or open up arbitrary folders on the server. Local folders can only be watched inside local_roots.
"sources": {"rclone": None, "local_roots": [], "check_seconds": 15, "cache_dir": None},
diff --git a/fastapi_backend/app/schemas/recordings.py b/fastapi_backend/app/schemas/recordings.py
index 81642fe82..00fb4dc7c 100644
--- a/fastapi_backend/app/schemas/recordings.py
+++ b/fastapi_backend/app/schemas/recordings.py
@@ -189,6 +189,38 @@ class Rendition(ResponseModel):
by: Literal["libreoffice", "chromium", "anytopdf"]
+class EnhanceMeasured(ResponseModel):
+ """How good a picture looked to the enhance step (docs/enhance.md)."""
+
+ width: int
+ height: int
+ sharpness: float = Field(description="the variance of its Laplacian: how sharply its tones change")
+ noise: float = Field(description="its estimated noise, in levels out of 255")
+ contrast: float = Field(description="the standard deviation of its tones (0 to 255)")
+ brightness: float = Field(description="the mean of its tones (0 to 255)")
+ faces: int | None = Field(None, description="how many faces it has, where they were looked for")
+ smallest_face: int | None = Field(None, description="the height of its smallest face, in pixels")
+
+
+class Enhanced(ResponseModel):
+ """What the enhance step found and did (docs/enhance.md): whether the picture looked low quality and why, and the
+ enhanced copy it's read from, if one was made. The original stays as it is."""
+
+ kind: Literal["image"]
+ low: list[Literal["small", "blurry", "noisy", "dull", "small_faces"]] = Field(
+ default_factory=list, description="why it looked low quality; empty when it looked good enough"
+ )
+ measured: EnhanceMeasured
+ enhanced: bool = Field(description="an enhanced copy was made, and the steps after it read that")
+ engine: str | None = Field(None, description="what enhanced it: classic, or realesr-general-x4v3 (Real-ESRGAN)")
+ scale: float | None = Field(None, description="how many times bigger the copy is")
+ width: int | None = Field(None, description="the copy's width in pixels")
+ height: int | None = None
+ forced: bool | None = Field(None, description="enhanced on request, whatever the settings and its quality")
+ at: str | None = None
+ download: str | None = Field(None, description="a signed link to save the enhanced copy")
+
+
class Described(ResponseModel):
"""What a document says about itself, read by a Tika server (documents.tika_metadata), as Dublin Core terms."""
@@ -285,6 +317,7 @@ class Recording(ResponseModel):
web: WebPage | None = Field(None, description="a web page captured as a document")
enrichment: Enrichment | None = Field(None, description="the places and dates anytopdf found (analysis.anytopdf)")
nsfw: NsfwState | None = Field(None, description="NSFW classification, where it's on in its namespace and it was classified")
+ enhanced: Enhanced | None = Field(None, description="an image the enhance step looked at: what it found, and its enhanced copy")
class Page(ResponseModel):
diff --git a/fastapi_backend/tests/api/test_enhance.py b/fastapi_backend/tests/api/test_enhance.py
new file mode 100644
index 000000000..592e29b8b
--- /dev/null
+++ b/fastapi_backend/tests/api/test_enhance.py
@@ -0,0 +1,286 @@
+"""Enhancing low quality images (docs/enhance.md): off by default; turned on, the enhance step measures an image before
+it's read and, when it's small, blurry, noisy, dull or has faces too small to recognise, makes an enhanced copy that the
+read step and the steps after it work from, keeping the original. Asked for (the API, MCP, an automation), it enhances
+whatever the settings and the picture's quality. Real-ESRGAN is read the way its export expects: RGB from 0 to 1, in
+tiles of 128 pixels, four times as big."""
+
+from __future__ import annotations
+
+import io
+
+import numpy as np
+import pytest
+from PIL import Image, ImageFilter
+
+from app.api import mcp_tools
+from app.domain import actions, components, enhance, faces, jobs, keyring, machine, settings, store, video
+from tests.api.test_documents import _png, _upload
+from tests.api.test_mcp import tool, tool_error
+from tests.helpers import drain, login, make_user
+
+R = store.R
+
+
+@pytest.fixture
+def env(client, db):
+ make_user(db, "root@x.io", "root password 1", admin=True)
+ make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"})
+ make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"})
+ return {
+ "ha": login(client, "root@x.io", "root password 1"),
+ "he": login(client, "ed@x.io", "editor password 1"),
+ "hv": login(client, "vi@x.io", "viewer password 1"),
+ }
+
+
+def on(cfg, **more):
+ return {**cfg, "enhance": {**cfg["enhance"], "images": True, **more}}
+
+
+def _scene(w, h, seed=1):
+ """A picture with edges and texture: rectangles on a gradient."""
+ rng = np.random.default_rng(seed)
+ a = np.tile(np.linspace(40, 210, w, dtype=np.float32), (h, 1))[..., None].repeat(3, axis=2)
+ for _ in range(12):
+ x, y = int(rng.integers(0, w - 8)), int(rng.integers(0, h - 8))
+ a[y : y + int(rng.integers(4, h // 3)), x : x + int(rng.integers(4, w // 3))] = rng.integers(0, 255, 3)
+ return Image.fromarray(a.clip(0, 255).astype(np.uint8))
+
+
+def _photo(client, h, img, name="photo.png", ns="pods"):
+ return _upload(client, h, _png(img), name, ns=ns)["recording"]
+
+
+def _log(db, rid):
+ return " ".join(x for j in db.rows("SELECT log FROM job WHERE recording = $r", r=rid) for x in j["log"])
+
+
+def _page(db, rid):
+ return db.one("SELECT width, height FROM page WHERE recording = $r AND idx = 0", r=rid)
+
+
+def test_off_by_default_the_original_is_read(client, env, db, cfg):
+ assert store.DEFAULTS["enhance"] == {
+ "images": False,
+ "image_engine": "classic",
+ "only_faces": False,
+ "min_side": 720,
+ "min_sharpness": 100,
+ "min_face_px": 64,
+ "max_scale": 4,
+ }
+ assert store.DEFAULTS["workers"]["steps"][0] == "enhance" and jobs.PIPELINE[0] == "enhance"
+ rid = _photo(client, env["he"], _scene(160, 120).filter(ImageFilter.GaussianBlur(3)))
+ drain(db, cfg)
+ assert "enhance skipped: enhancing images is off (Settings → Enhance)" in _log(db, rid)
+ assert _page(db, rid) == {"width": 160, "height": 120}
+ assert client.get(f"/api/v1/recordings/{rid}", headers=env["hv"]).json()["enhanced"] is None
+ assert not enhance.files(cfg, rid)
+ assert not components.BY_ID["enhance"].wanted_by(db, cfg, machine.probe(cfg["data_dir"]))
+
+
+def test_a_low_quality_image_is_read_from_its_enhanced_copy(client, env, db, cfg):
+ he, hv = env["he"], env["hv"]
+ small = _scene(240, 180).filter(ImageFilter.GaussianBlur(2))
+ rid = _photo(client, he, small)
+ drain(db, on(cfg))
+ log = _log(db, rid)
+ assert "enhanced (small, blurry): 240×180 to 960×720 by classic" in log and "read from its enhanced copy" in log
+ assert _page(db, rid) == {"width": 960, "height": 720} # its page is drawn from the copy
+ e = db.one("SELECT enhanced FROM $r", r=R("recording", rid))["enhanced"]
+ assert (e["low"], e["scale"], e["engine"], e["enhanced"]) == (["small", "blurry"], 4.0, "classic", True)
+ assert e["measured"]["width"] == 240 and e["measured"]["sharpness"] < 100
+
+ got = client.get(f"/api/v1/recordings/{rid}", headers=hv).json()["enhanced"]
+ assert (got["width"], got["height"], got["low"]) == (960, 720, ["small", "blurry"])
+ r = client.get(got["download"]) # a signed link, for a member
+ assert r.status_code == 200 and r.headers["content-type"] == "image/jpeg"
+ assert Image.open(io.BytesIO(r.content)).size == (960, 720)
+ assert client.get(f"/api/v1/recordings/{rid}/enhanced", headers=hv).status_code == 200
+ # the original stays as it was
+ assert Image.open(io.BytesIO(client.get(f"/api/v1/recordings/{rid}/media", headers=hv).content)).size == (240, 180)
+ # a share link's visitor sees its pages, not the copy
+ link = client.post(f"/api/v1/recordings/{rid}/share", headers=he, json={}).json()
+ assert client.get(f"/api/v1/recordings/{rid}/enhanced?s={link['token']}").status_code == 404
+
+ # deleting the resource removes the copy
+ path = enhance.enhanced_path(cfg, rid)
+ assert path.is_file()
+ assert client.delete(f"/api/v1/recordings/{rid}", headers=env["ha"]).status_code == 200
+ assert not path.exists()
+
+
+def test_a_good_image_is_left_as_it_is(client, env, db, cfg):
+ rid = _photo(client, env["he"], _scene(900, 760))
+ drain(db, on(cfg))
+ assert "good enough as it is (900×760" in _log(db, rid)
+ e = db.one("SELECT enhanced FROM $r", r=R("recording", rid))["enhanced"]
+ assert e["enhanced"] is False and e["low"] == [] and not enhance.files(cfg, rid)
+ assert _page(db, rid) == {"width": 900, "height": 760}
+ assert client.get(f"/api/v1/recordings/{rid}", headers=env["hv"]).json()["enhanced"].get("download") is None
+
+
+def test_asked_for_it_is_enhanced_whatever_the_settings_and_kept(client, env, db, cfg):
+ he, hv = env["he"], env["hv"]
+ rid = _photo(client, he, _scene(900, 760))
+ drain(db, cfg)
+ assert client.post(f"/api/v1/recordings/{rid}/enhance", headers=hv).status_code == 403 # viewers don't process
+ job = client.post(f"/api/v1/recordings/{rid}/enhance", headers=he).json()["job"]
+ steps = [s["type"] for s in db.one("SELECT steps FROM $r", r=R("job", job))["steps"]]
+ assert steps[:2] == ["enhance", "read"] and "enhance" not in steps[1:]
+ drain(db, cfg)
+ e = db.one("SELECT enhanced FROM $r", r=R("recording", rid))["enhanced"]
+ assert e["forced"] is True and e["enhanced"] is True and e["low"] == [] and enhance.files(cfg, rid)
+ assert "enhanced (on request)" in _log(db, rid)
+ # processed again with enhancing off, the copy made on request stays
+ jobs.enqueue(db, rid, ["enhance", "read"])
+ drain(db, cfg)
+ assert "kept the copy made on request" in _log(db, rid) and enhance.files(cfg, rid)
+
+ # documents and recordings aren't enhanced
+ doc = _upload(client, he, b"%PDF-1.4\n", "x.pdf")["recording"]
+ r = client.post(f"/api/v1/recordings/{doc}/enhance", headers=he)
+ assert r.status_code == 400 and "only images" in r.json()["detail"]
+
+
+def test_agents_and_automations_enhance_too(client, env, db, cfg, app):
+ he, hv = env["he"], env["hv"]
+ rid = _photo(client, he, _scene(200, 150))
+ drain(db, cfg)
+ assert "enhance" in mcp_tools.TOOLS and mcp_tools.TOOLS["enhance"].writes
+ assert tool_error(client, hv, "enhance", recording_id=rid) == "needs the process permission for this asset"
+ got = tool(client, he, "enhance", recording_id=rid)
+ assert got["job"] and got["url"].endswith(f"/resources/{rid}")
+ drain(db, cfg)
+ assert tool(client, hv, "get_recording", recording_id=rid)["enhanced"]["scale"] == 4.0 # 720 / 150, at most max_scale
+
+ # the automation action, on the asset the activity is about
+ me = db.one("SELECT record::id(id) AS id FROM account WHERE email = 'ed@x.io'")["id"]
+ intent = {
+ "key": "1-n1",
+ "action": "enhance",
+ "params": {"asset": f"lens:recording/{rid}"},
+ "effect": "lens",
+ "risk": "routine",
+ "untrusted": False,
+ "spaces": [store.ns_id(db, "pods")],
+ "run_as": me,
+ "node": "n1",
+ "automation": 4,
+ "run": 1,
+ }
+ assert actions.perform(db, cfg, intent, None, True)["would"].startswith("enhance photo")
+ out = actions.perform(db, cfg, intent, None, False)
+ assert db.one("SELECT created_by FROM $r", r=R("job", out["job"]))["created_by"] == "automation: Automation 4"
+ assert actions.describe("enhance", {"asset": 7}) == "Enhance asset 7"
+
+
+def test_faces_too_small_to_recognise_and_only_faces(client, env, db, cfg, monkeypatch):
+ class FakeFaces:
+ name = "fake"
+
+ def faces(self, path):
+ img = Image.open(path)
+ return (
+ [{"box": [0.4, 0.4, 0.05, 30 / img.height], "score": 0.9, "embedding": np.ones(4)}] if img.getpixel((0, 0))[0] > 100 else []
+ )
+
+ monkeypatch.setattr(video, "face_engine", lambda cfg: FakeFaces())
+ he = env["he"]
+ with_face = _scene(1200, 900, seed=3)
+ with_face.putpixel((0, 0), (250, 250, 250))
+ no_face = _scene(1200, 900, seed=4)
+ no_face.putpixel((0, 0), (5, 5, 5))
+ a, b = _photo(client, he, with_face, "a.png"), _photo(client, he, no_face, "b.png")
+ drain(db, on(cfg, only_faces=True))
+ # the namespace doesn't find faces: none were looked for
+ assert "face detection is off in this namespace (enhance.only_faces)" in _log(db, a)
+ faces.set_mode(db, store.ns_id(db, "pods"), "detect")
+ jobs.enqueue(db, a, ["enhance"])
+ jobs.enqueue(db, b, ["enhance"])
+ drain(db, on(cfg, only_faces=True))
+ assert "there's no face in it (enhance.only_faces)" in _log(db, b)
+ e = db.one("SELECT enhanced FROM $r", r=R("recording", a))["enhanced"]
+ assert e["low"] == ["small_faces"] and e["measured"]["smallest_face"] == 30 and e["scale"] == round(64 / 30, 2)
+
+
+def test_measuring(cfg):
+ sharp = _scene(400, 300)
+ m = enhance.measure(sharp)
+ assert m["sharpness"] > 100 and m["noise"] < 2 and m["contrast"] > 30
+ assert enhance.measure(sharp.filter(ImageFilter.GaussianBlur(3)))["sharpness"] < 100
+ rng = np.random.default_rng(0)
+ flat = np.full((300, 400, 3), 128, dtype=np.float32) + rng.normal(0, 12, (300, 400, 1))
+ noisy = enhance.measure(Image.fromarray(flat.clip(0, 255).astype(np.uint8)))
+ assert 10 < noisy["noise"] < 14 # Immerkær's estimate finds the noise it was given
+ assert enhance.measure(Image.new("RGB", (400, 300), (90, 90, 90)))["contrast"] < enhance.DULL
+ opts = enhance.options(cfg)
+ assert enhance.assess(m, None, opts) == (["small"], 2.4)
+ assert enhance.assess({**m, "width": 4000, "height": 3000}, [20], opts) == (["small_faces"], 1.0) # MAX_SIDE
+ assert enhance.assess(m, [80], {**opts, "max_scale": 1})[1] == 1.0
+ assert enhance.classic(sharp, ["small"], 2.4).size == (960, 720)
+
+
+class Nearest:
+ """Stands in for Real-ESRGAN's session: each 128-pixel tile four times as big, nearest neighbour."""
+
+ def __init__(self):
+ self.calls = 0
+
+ def get_inputs(self):
+ return [type("I", (), {"name": "image", "shape": [1, 3, 128, 128]})()]
+
+ def run(self, _, feeds):
+ x = feeds["image"]
+ assert x.shape == (1, 3, 128, 128) and x.dtype == np.float32 and 0 <= x.min() and x.max() <= 1
+ self.calls += 1
+ return [x.repeat(4, axis=2).repeat(4, axis=3)]
+
+
+def test_real_esrgan_is_run_in_tiles_without_seams(cfg, monkeypatch):
+ fake = Nearest()
+ for name in enhance.MODEL_FILES:
+ p = components.models_dir(cfg) / name
+ p.parent.mkdir(parents=True, exist_ok=True)
+ p.write_bytes(b"x")
+ monkeypatch.setattr(enhance, "_session", lambda path: fake)
+ img = _scene(250, 130)
+ real = enhance.RealEsrgan(cfg)
+ big = real.upscale(img)
+ assert big.size == (1000, 520) and fake.calls == 3 * 2
+ want = np.asarray(img).repeat(4, axis=0).repeat(4, axis=1)
+ assert np.abs(np.asarray(big).astype(int) - want).max() <= 1 # every tile in its place
+ said = []
+ out, engine = enhance.enhance_image(on(cfg, image_engine="realesrgan"), img, ["small"], 2.0, said.append)
+ assert (out.size, engine, said) == ((500, 260), "realesr-general-x4v3", [])
+ big = Image.new("RGB", (1300, 1200))
+ out, engine = enhance.enhance_image(on(cfg, image_engine="realesrgan"), big, ["blurry"], 1.0, said.append)
+ assert engine == "classic" and "too large for Real-ESRGAN" in said[0]
+ assert components.BY_ID["enhance"].needed(on(cfg, image_engine="realesrgan"), machine.probe(cfg["data_dir"]))
+
+
+def test_settings_are_checked(db, cfg):
+ base = cfg
+ settings.save(db, base, "enhance", {"images": True, "image_engine": "realesrgan", "min_side": 1000, "max_scale": 2.5})
+ eff = settings.effective(db, base)["enhance"]
+ assert (eff["images"], eff["image_engine"], eff["min_side"], eff["max_scale"]) == (True, "realesrgan", 1000, 2.5)
+ for bad, said in (
+ ({"image_engine": "anime4k"}, "enhance.image_engine must be one of"),
+ ({"min_side": 10}, "enhance.min_side is a number from 64 to 4000"),
+ ({"max_scale": 8}, "enhance.max_scale is a number from 1 to 4"),
+ ({"images": "yes"}, "enhance.images is true or false"),
+ ({"denoise": True}, "unknown setting enhance.denoise"),
+ ):
+ with pytest.raises(ValueError, match=said):
+ settings.save(db, base, "enhance", bad)
+
+
+def test_the_copy_is_encrypted_where_the_files_are(client, env, db, cfg):
+ eff = settings.effective(db, cfg) # a new archive encrypts its files
+ rid = _photo(client, env["he"], _scene(200, 150))
+ drain(db, on(eff))
+ path = enhance.enhanced_path(cfg, rid)
+ assert keyring.is_encrypted(path)
+ assert str(path) in list(keyring.recording_files(db, cfg, rid))
+ r = client.get(f"/api/v1/recordings/{rid}/enhanced", headers=env["he"])
+ assert r.status_code == 200 and Image.open(io.BytesIO(r.content)).size == (800, 600)
diff --git a/fastapi_backend/tests/api/test_jobs.py b/fastapi_backend/tests/api/test_jobs.py
index ca791ccd6..1e88cbfc7 100644
--- a/fastapi_backend/tests/api/test_jobs.py
+++ b/fastapi_backend/tests/api/test_jobs.py
@@ -398,6 +398,10 @@ def test_a_document_is_read_not_transcribed(db, cfg, folder):
{"type": "read"},
{"type": "report"},
]
+ # only images are enhanced, so the others drop the step unless it's all that was asked for
+ assert jobs.for_source([{"type": "enhance"}, {"type": "transcribe"}], "audio") == [{"type": "transcribe"}]
+ assert jobs.for_source([{"type": "enhance"}, {"type": "transcribe"}], "image") == [{"type": "enhance"}, {"type": "read"}]
+ assert jobs.for_source([{"type": "enhance"}], "document") == [{"type": "enhance"}]
# a worker whose steps list predates the read step runs it too
assert "read" in jobs.Worker(db, lambda: {**cfg, "workers": {**cfg["workers"], "steps": ["transcribe"]}}, name="old").can
diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py
index 6e720b9df..88f3fa5f3 100644
--- a/fastapi_backend/tests/api/test_mcp.py
+++ b/fastapi_backend/tests/api/test_mcp.py
@@ -85,6 +85,7 @@ def test_clients_are_sent_to_sign_in(client, app, env):
"create_podcast",
"set_nsfw",
"mark_nsfw",
+ "enhance",
"set_namespace_integrations",
"set_file_storage",
"set_monitoring",
diff --git a/mkdocs.yml b/mkdocs.yml
index 9d8d1d712..64607c159 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -87,6 +87,7 @@ nav:
- Processing: processing.md
- Video: video.md
- NSFW classification: nsfw.md
+ - Enhancing images: enhance.md
- Notifications: notifications.md
- Automations: automations.md
- Timeline: timeline.md
diff --git a/nextjs-frontend/__tests__/enhance.test.tsx b/nextjs-frontend/__tests__/enhance.test.tsx
new file mode 100644
index 000000000..b4fc78dd3
--- /dev/null
+++ b/nextjs-frontend/__tests__/enhance.test.tsx
@@ -0,0 +1,112 @@
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { fireEvent, render, screen, waitFor } from "@testing-library/react";
+import "@testing-library/jest-dom";
+
+import { Resources } from "@/app/openapi-client";
+import type { Enhanced } from "@/app/openapi-client/types.gen";
+import { EnhanceDetail } from "@/components/recording/enhance";
+import { engineName, enhanceText } from "@/components/recording/enhance-model";
+import { RecordingProvider, type RecordingCtx } from "@/components/recording/context";
+import { DESCRIBE, NEEDS, moveStep } from "@/components/pipelines/pipeline-model";
+import { FIELDS, SECTIONS, WORKER_STEPS } from "@/components/settings/model";
+import { SETTINGS_GROUPS } from "@/components/settings/find";
+import { STEP_LABEL } from "@/components/ui/loop";
+
+jest.mock("@/app/openapi-client", () => ({ Resources: { enhanceRecording: jest.fn() } }));
+jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) }));
+const push = jest.fn();
+jest.mock("next/navigation", () => ({ useRouter: () => ({ push }) }));
+const toast = jest.fn();
+jest.mock("@/components/ui/toast", () => ({ useToast: () => toast }));
+
+const MEASURED = { width: 240, height: 180, sharpness: 12, noise: 2, contrast: 40, brightness: 120 };
+const MADE: Enhanced = {
+ kind: "image",
+ low: ["small", "blurry"],
+ measured: MEASURED,
+ enhanced: true,
+ engine: "classic",
+ scale: 4,
+ width: 960,
+ height: 720,
+ forced: false,
+ at: "2026-10-17T01:00:00Z",
+ download: "/api/v1/recordings/9/enhanced?sig=abc",
+};
+
+const ok = (data: unknown) => Promise.resolve({ data, response: { ok: true, status: 200 } });
+const ctx = (enhanced: Enhanced | null, canEdit = true) =>
+ ({ id: 9, ns: "pods", rec: { id: 9, title: "Scan", enhanced }, paged: true, canEdit }) as unknown as RecordingCtx;
+const wrap = (value: RecordingCtx) => {
+ const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
+ return render(
+
+
+
+
+ ,
+ );
+};
+
+describe("enhance row text", () => {
+ it("says what was found and what was made", () => {
+ expect(enhanceText(null)).toBe("not run");
+ expect(enhanceText(MADE)).toBe("Small, blurry · 240×180 to 960×720 by the classic engine");
+ expect(enhanceText({ ...MADE, low: [], forced: true, engine: "realesr-general-x4v3" })).toBe(
+ "On request · 240×180 to 960×720 by Real-ESRGAN",
+ );
+ expect(enhanceText({ ...MADE, low: ["dull", "small_faces"] })).toMatch(/^Low contrast, faces too small/);
+ expect(enhanceText({ kind: "image", low: [], measured: MEASURED, enhanced: false })).toBe(
+ "good enough as it is (240×180)",
+ );
+ expect(engineName(null)).toBeNull();
+ });
+});
+
+describe("EnhanceDetail", () => {
+ beforeEach(() => jest.clearAllMocks());
+
+ it("links the copy and enhances again on request", async () => {
+ (Resources.enhanceRecording as jest.Mock).mockReturnValue(ok({ ok: true, job: 31 }));
+ wrap(ctx(MADE));
+ expect(screen.getByRole("link", { name: /Save the copy/ })).toHaveAttribute("href", MADE.download);
+ fireEvent.click(screen.getByRole("button", { name: "Enhance again" }));
+ await waitFor(() =>
+ expect(Resources.enhanceRecording).toHaveBeenCalledWith(expect.objectContaining({ path: { rid: 9 } })),
+ );
+ await waitFor(() => expect(toast).toHaveBeenCalledWith(expect.objectContaining({ title: "Enhancing" })));
+ });
+
+ it("offers nothing to save before a copy exists, and no button to viewers", () => {
+ wrap(ctx(null, false));
+ expect(screen.getByText("not run")).toBeInTheDocument();
+ expect(screen.queryByRole("link")).toBeNull();
+ expect(screen.queryByRole("button")).toBeNull();
+ });
+});
+
+describe("enhance in the web app", () => {
+ it("has a settings section with every backend key, off by default", () => {
+ const keys = FIELDS.filter((f) => f.section === "enhance").map((f) => f.key);
+ expect(keys).toEqual([
+ "images",
+ "image_engine",
+ "only_faces",
+ "min_side",
+ "min_sharpness",
+ "min_face_px",
+ "max_scale",
+ ]);
+ expect(SECTIONS.some((s) => s.id === "enhance")).toBe(true);
+ expect(SETTINGS_GROUPS.some((g) => g.sections.includes("enhance"))).toBe(true);
+ });
+
+ it("is a pipeline step that runs before read", () => {
+ expect(WORKER_STEPS[0]).toBe("enhance");
+ expect(STEP_LABEL.enhance).toBe("Enhance");
+ expect(DESCRIBE.enhance).toMatch(/Settings → Enhance/);
+ expect(NEEDS.read).toContain("enhance");
+ const steps = [{ type: "enhance" }, { type: "read" }];
+ expect(moveStep(steps, 1, 0).error).toBeTruthy();
+ });
+});
diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts
index b510f7043..a758ebf57 100644
--- a/nextjs-frontend/app/openapi-client/index.ts
+++ b/nextjs-frontend/app/openapi-client/index.ts
@@ -919,6 +919,13 @@ export type {
EnableAutomationResponse,
EnableAutomationResponses,
EncryptionProgress,
+ Enhanced,
+ EnhanceMeasured,
+ EnhanceRecordingData,
+ EnhanceRecordingError,
+ EnhanceRecordingErrors,
+ EnhanceRecordingResponse,
+ EnhanceRecordingResponses,
EnrichedDate,
EnrichedPlace,
Enrichment,
@@ -1152,6 +1159,10 @@ export type {
GetEncryptionData,
GetEncryptionResponse,
GetEncryptionResponses,
+ GetEnhancedData,
+ GetEnhancedError,
+ GetEnhancedErrors,
+ GetEnhancedResponses,
GetEntityConnectionsData,
GetEntityConnectionsError,
GetEntityConnectionsErrors,
diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts
index b17155e7b..a58c897cf 100644
--- a/nextjs-frontend/app/openapi-client/sdk.gen.ts
+++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts
@@ -433,6 +433,9 @@ import type {
EnableAutomationData,
EnableAutomationErrors,
EnableAutomationResponses,
+ EnhanceRecordingData,
+ EnhanceRecordingErrors,
+ EnhanceRecordingResponses,
EntityToTopicData,
EntityToTopicErrors,
EntityToTopicResponses,
@@ -535,6 +538,9 @@ import type {
GetEmbedLinkResponses,
GetEncryptionData,
GetEncryptionResponses,
+ GetEnhancedData,
+ GetEnhancedErrors,
+ GetEnhancedResponses,
GetEntityConnectionsData,
GetEntityConnectionsErrors,
GetEntityConnectionsResponses,
@@ -5564,6 +5570,22 @@ export class Resources {
});
}
+ /**
+ * Enhance Recording
+ *
+ * Enhance an image whatever the settings and its quality (docs/enhance.md): an enhanced copy is made, and its
+ * pipeline's steps after enhance (reading its text, faces, objects, …) run again on that copy. The original stays as
+ * it is. 400 for anything but an image. Audited as `recording.enhance`.
+ */
+ public static enhanceRecording(
+ options: Options,
+ ): RequestResult {
+ return (options.client ?? client).post({
+ url: "/api/v1/resources/{rid}/enhance",
+ ...options,
+ });
+ }
+
/**
* Revoke Shares
*
@@ -8347,6 +8369,22 @@ export class Video {
});
}
+ /**
+ * Get Enhanced
+ *
+ * The enhanced copy the enhance step made of a low quality image (docs/enhance.md), to save. Members of its
+ * namespace only (a bearer token, or a signed link the API gave one): visitors get its pages, with faces pixelated
+ * where the namespace does that.
+ */
+ public static getEnhanced(
+ options: Options,
+ ): RequestResult {
+ return (options.client ?? client).get({
+ url: "/api/v1/resources/{rid}/enhanced",
+ ...options,
+ });
+ }
+
/**
* Get Frame
*
diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts
index 5b94dccc9..b8d2f8b8d 100644
--- a/nextjs-frontend/app/openapi-client/types.gen.ts
+++ b/nextjs-frontend/app/openapi-client/types.gen.ts
@@ -5635,6 +5635,124 @@ export type EncryptionProgress = {
[key: string]: unknown;
};
+/**
+ * EnhanceMeasured
+ *
+ * How good a picture looked to the enhance step (docs/enhance.md).
+ */
+export type EnhanceMeasured = {
+ /**
+ * Width
+ */
+ width: number;
+ /**
+ * Height
+ */
+ height: number;
+ /**
+ * Sharpness
+ *
+ * the variance of its Laplacian: how sharply its tones change
+ */
+ sharpness: number;
+ /**
+ * Noise
+ *
+ * its estimated noise, in levels out of 255
+ */
+ noise: number;
+ /**
+ * Contrast
+ *
+ * the standard deviation of its tones (0 to 255)
+ */
+ contrast: number;
+ /**
+ * Brightness
+ *
+ * the mean of its tones (0 to 255)
+ */
+ brightness: number;
+ /**
+ * Faces
+ *
+ * how many faces it has, where they were looked for
+ */
+ faces?: number | null;
+ /**
+ * Smallest Face
+ *
+ * the height of its smallest face, in pixels
+ */
+ smallest_face?: number | null;
+ [key: string]: unknown;
+};
+
+/**
+ * Enhanced
+ *
+ * What the enhance step found and did (docs/enhance.md): whether the picture looked low quality and why, and the
+ * enhanced copy it's read from, if one was made. The original stays as it is.
+ */
+export type Enhanced = {
+ /**
+ * Kind
+ */
+ kind: "image";
+ /**
+ * Low
+ *
+ * why it looked low quality; empty when it looked good enough
+ */
+ low?: Array<"small" | "blurry" | "noisy" | "dull" | "small_faces">;
+ measured: EnhanceMeasured;
+ /**
+ * Enhanced
+ *
+ * an enhanced copy was made, and the steps after it read that
+ */
+ enhanced: boolean;
+ /**
+ * Engine
+ *
+ * what enhanced it: classic, or realesr-general-x4v3 (Real-ESRGAN)
+ */
+ engine?: string | null;
+ /**
+ * Scale
+ *
+ * how many times bigger the copy is
+ */
+ scale?: number | null;
+ /**
+ * Width
+ *
+ * the copy's width in pixels
+ */
+ width?: number | null;
+ /**
+ * Height
+ */
+ height?: number | null;
+ /**
+ * Forced
+ *
+ * enhanced on request, whatever the settings and its quality
+ */
+ forced?: boolean | null;
+ /**
+ * At
+ */
+ at?: string | null;
+ /**
+ * Download
+ *
+ * a signed link to save the enhanced copy
+ */
+ download?: string | null;
+ [key: string]: unknown;
+};
+
/**
* EnrichedDate
*
@@ -15755,6 +15873,10 @@ export type Recording = {
* NSFW classification, where it's on in its namespace and it was classified
*/
nsfw?: NsfwState | null;
+ /**
+ * an image the enhance step looked at: what it found, and its enhanced copy
+ */
+ enhanced?: Enhanced | null;
[key: string]: unknown;
};
@@ -29803,6 +29925,36 @@ export type ReprocessRecordingResponses = {
export type ReprocessRecordingResponse = ReprocessRecordingResponses[keyof ReprocessRecordingResponses];
+export type EnhanceRecordingData = {
+ body?: never;
+ path: {
+ /**
+ * Rid
+ */
+ rid: number;
+ };
+ query?: never;
+ url: "/api/v1/resources/{rid}/enhance";
+};
+
+export type EnhanceRecordingErrors = {
+ /**
+ * Validation Error
+ */
+ 422: HttpValidationError;
+};
+
+export type EnhanceRecordingError = EnhanceRecordingErrors[keyof EnhanceRecordingErrors];
+
+export type EnhanceRecordingResponses = {
+ /**
+ * Successful Response
+ */
+ 200: JobQueued;
+};
+
+export type EnhanceRecordingResponse = EnhanceRecordingResponses[keyof EnhanceRecordingResponses];
+
export type RevokeSharesData = {
body?: never;
path: {
@@ -35551,6 +35703,39 @@ export type GetPdfResponses = {
export type GetPdfResponse = GetPdfResponses[keyof GetPdfResponses];
+export type GetEnhancedData = {
+ body?: never;
+ path: {
+ /**
+ * Rid
+ */
+ rid: number;
+ };
+ query?: {
+ /**
+ * S
+ */
+ s?: string;
+ };
+ url: "/api/v1/resources/{rid}/enhanced";
+};
+
+export type GetEnhancedErrors = {
+ /**
+ * Validation Error
+ */
+ 422: HttpValidationError;
+};
+
+export type GetEnhancedError = GetEnhancedErrors[keyof GetEnhancedErrors];
+
+export type GetEnhancedResponses = {
+ /**
+ * Successful Response
+ */
+ 200: unknown;
+};
+
export type GetFrameData = {
body?: never;
path: {
diff --git a/nextjs-frontend/components/batches/format.ts b/nextjs-frontend/components/batches/format.ts
index 11b8ca80a..7a8f4281b 100644
--- a/nextjs-frontend/components/batches/format.ts
+++ b/nextjs-frontend/components/batches/format.ts
@@ -35,7 +35,17 @@ export function confirmNumber(expected: string | null | undefined): string {
}
/** What a run would need that a transcript-only recording can't give: audio and video steps. */
-export const MEDIA_STEPS = new Set(["transcribe", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe"]);
+export const MEDIA_STEPS = new Set([
+ "enhance",
+ "transcribe",
+ "diarize",
+ "shots",
+ "ocr",
+ "faces",
+ "objects",
+ "nsfw",
+ "describe",
+]);
/** True when every recording is transcript-only and every step needs media: nothing can run. */
export function nothingRunnable(est: Pick, steps: { type?: unknown }[]): boolean {
diff --git a/nextjs-frontend/components/pipelines/pipeline-canvas.tsx b/nextjs-frontend/components/pipelines/pipeline-canvas.tsx
index f126068b2..dbbf18052 100644
--- a/nextjs-frontend/components/pipelines/pipeline-canvas.tsx
+++ b/nextjs-frontend/components/pipelines/pipeline-canvas.tsx
@@ -17,6 +17,7 @@ import {
Shapes,
Sparkles,
TextSearch,
+ WandSparkles,
Workflow,
type LucideIcon,
} from "lucide-react";
@@ -52,6 +53,7 @@ import { useArchive } from "@/lib/hooks/session";
import { cn } from "@/lib/utils";
const ICON: Record = {
+ enhance: WandSparkles,
transcribe: Captions,
read: ScanText,
diarize: AudioLines,
@@ -70,7 +72,7 @@ const ICON: Record = {
workflow: Workflow,
};
const TONE = { intent: "blue", red: "red", green: "green", gate: "gold", neutral: "neutral" } as const;
-const ASSETS = ["transcribe", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe"];
+const ASSETS = ["enhance", "transcribe", "diarize", "shots", "ocr", "faces", "objects", "nsfw", "describe"];
const METADATA = ["analyze", "embed", "summarize", "llm", "report", "export"];
/** A pipeline drawn as a graph: asset steps, metadata steps and workflows, an edge saying what runs after what. */
diff --git a/nextjs-frontend/components/pipelines/pipeline-editor.tsx b/nextjs-frontend/components/pipelines/pipeline-editor.tsx
index bc9688c11..fc8057061 100644
--- a/nextjs-frontend/components/pipelines/pipeline-editor.tsx
+++ b/nextjs-frontend/components/pipelines/pipeline-editor.tsx
@@ -18,6 +18,7 @@ import {
Shapes,
Sparkles,
TextSearch,
+ WandSparkles,
Webhook,
Workflow,
X,
@@ -60,6 +61,7 @@ import { useArchive } from "@/lib/hooks/session";
import { cn } from "@/lib/utils";
const ICON: Record = {
+ enhance: WandSparkles,
transcribe: Captions,
read: ScanText,
diarize: AudioLines,
@@ -78,6 +80,7 @@ const ICON: Record = {
workflow: Workflow,
};
const LIB_ORDER = [
+ "enhance",
"transcribe",
"diarize",
"shots",
diff --git a/nextjs-frontend/components/pipelines/pipeline-model.ts b/nextjs-frontend/components/pipelines/pipeline-model.ts
index b4ff63817..77168af2b 100644
--- a/nextjs-frontend/components/pipelines/pipeline-model.ts
+++ b/nextjs-frontend/components/pipelines/pipeline-model.ts
@@ -70,6 +70,7 @@ export function cleanSpec(s: StepSpec): StepSpec | string {
/** Which earlier steps a step reads from, when they are in the same pipeline. */
export const NEEDS: Record = {
+ read: ["enhance"],
diarize: ["transcribe"],
ocr: ["shots"],
faces: ["shots"],
@@ -87,6 +88,7 @@ export const NEEDS: Record = {
/** What a step makes available to later steps and templates. */
export const PROVIDES: Record = {
+ enhance: "an enhanced copy that read uses",
transcribe: "transcript · segments[] · language",
read: "pages[] · segments[]",
diarize: "speakers[] · turns",
@@ -106,6 +108,8 @@ export const PROVIDES: Record = {
};
export const DESCRIBE: Record = {
+ enhance:
+ "Keeps a sharper copy of small, blurry, noisy or dull images for read to use; the original stays (Settings → Enhance)",
transcribe: "Speech to text with word timings (Settings → Transcription)",
read: "Draws a document’s or an image’s pages and reads their text (Settings → Documents)",
diarize: "Who spoke when (Settings → Diarization)",
diff --git a/nextjs-frontend/components/recording/details-tab.tsx b/nextjs-frontend/components/recording/details-tab.tsx
index 486a4d943..0587f337c 100644
--- a/nextjs-frontend/components/recording/details-tab.tsx
+++ b/nextjs-frontend/components/recording/details-tab.tsx
@@ -21,6 +21,7 @@ import { uploadKindName } from "@/components/import/files";
import { NsfwDetail } from "@/components/nsfw/nsfw";
import { ContentTypePicker } from "@/components/recording/content-type";
import { useRec } from "@/components/recording/context";
+import { EnhanceDetail } from "@/components/recording/enhance";
import {
describedRows,
emailRows,
@@ -104,6 +105,7 @@ export function DetailsTab() {
["Fingerprint", rec.fingerprint ?? null, true],
];
const processing: Row[] = [
+ ["Enhance", model.media.kind === "image" ? : null],
[
"Transcribe",
rec.transcribed_at
diff --git a/nextjs-frontend/components/recording/enhance-model.ts b/nextjs-frontend/components/recording/enhance-model.ts
new file mode 100644
index 000000000..1b38445c9
--- /dev/null
+++ b/nextjs-frontend/components/recording/enhance-model.ts
@@ -0,0 +1,30 @@
+import type { Enhanced } from "@/app/openapi-client/types.gen";
+
+/** Why the enhance step thought an image looked low quality (docs/enhance.md). */
+export const ENHANCE_REASONS: Record = {
+ small: "small",
+ blurry: "blurry",
+ noisy: "noisy",
+ dull: "low contrast",
+ small_faces: "faces too small",
+};
+
+/** The engine that made the copy, as people know it. */
+export function engineName(engine: string | null | undefined): string | null {
+ if (!engine) return null;
+ return engine === "classic" ? "the classic engine" : engine.startsWith("realesr") ? "Real-ESRGAN" : engine;
+}
+
+/** One line for the Details tab: what the enhance step found and did ("not run" before it ran). */
+export function enhanceText(e: Enhanced | null | undefined): string {
+ if (!e) return "not run";
+ const m = e.measured;
+ const was = `${m.width}×${m.height}`;
+ if (!e.enhanced) return `good enough as it is (${was})`;
+ const why = (e.low ?? []).map((r) => ENHANCE_REASONS[r] ?? r).join(", ");
+ const made = e.width && e.height ? `${was} to ${e.width}×${e.height}` : was;
+ return [
+ why ? `${why[0].toUpperCase()}${why.slice(1)}` : "On request",
+ [made, engineName(e.engine) && `by ${engineName(e.engine)}`].filter(Boolean).join(" "),
+ ].join(" · ");
+}
diff --git a/nextjs-frontend/components/recording/enhance.tsx b/nextjs-frontend/components/recording/enhance.tsx
new file mode 100644
index 000000000..79b065ffa
--- /dev/null
+++ b/nextjs-frontend/components/recording/enhance.tsx
@@ -0,0 +1,63 @@
+"use client";
+
+import { useMutation, useQueryClient } from "@tanstack/react-query";
+import { Download, WandSparkles } from "lucide-react";
+import { useRouter } from "next/navigation";
+
+import { Resources } from "@/app/openapi-client";
+import { useRec } from "@/components/recording/context";
+import { enhanceText } from "@/components/recording/enhance-model";
+import { rk } from "@/components/recording/hooks";
+import { Button } from "@/components/ui/button";
+import { useToast } from "@/components/ui/toast";
+import { data, useApiClient } from "@/lib/api/browser";
+import { absolute } from "@/lib/format";
+
+/**
+ * An image's Enhance row (docs/enhance.md): what the step found, a link to save the enhanced copy, and, for editors,
+ * a button that enhances it now whatever the settings, then reads it again.
+ */
+export function EnhanceDetail() {
+ const { id, rec, canEdit } = useRec();
+ const client = useApiClient();
+ const qc = useQueryClient();
+ const toast = useToast();
+ const router = useRouter();
+ const e = rec.enhanced;
+ const run = useMutation({
+ mutationFn: () => data(Resources.enhanceRecording({ client, path: { rid: id } })),
+ onSuccess: (r) => {
+ void qc.invalidateQueries({ queryKey: rk.jobs(id) });
+ toast({
+ title: "Enhancing",
+ body: "Its text is read again from the enhanced copy; the original stays as it is.",
+ tone: "intent",
+ action: { label: "View job", onClick: () => router.push(`/activity/${r.job}`) },
+ });
+ },
+ onError: (err) => toast({ title: "Couldn’t enhance it", body: (err as Error).message, tone: "red" }),
+ });
+ return (
+
+
+ {enhanceText(e)}
+ {e?.at ? ` · ${absolute(e.at)}` : ""}
+
+ {e?.enhanced && e.download && (
+
+
+ Save the copy
+
+ )}
+ {canEdit && (
+ } onClick={() => run.mutate()} disabled={run.isPending}>
+ {e?.enhanced ? "Enhance again" : "Enhance"}
+
+ )}
+
+ );
+}
diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx
index 9f2b7f1f9..cd055cde0 100644
--- a/nextjs-frontend/components/settings/bodies.tsx
+++ b/nextjs-frontend/components/settings/bodies.tsx
@@ -357,6 +357,28 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) {
>
);
+ case "enhance":
+ return (
+ <>
+
+
+
+
+
+
+
+
+
+
+
+
+ The enhanced copy is kept beside the original, encrypted like it, and is what OCR and descriptions read.
+ Real-ESRGAN general x4v3 (BSD-3-Clause) runs with ONNX Runtime on this server: pictures aren’t sent
+ anywhere. Any image can be enhanced on request from its Details tab, even with this off. Audio isn’t
+ enhanced yet.
+
+ >
+ );
case "workers":
return (
<>
diff --git a/nextjs-frontend/components/settings/find.ts b/nextjs-frontend/components/settings/find.ts
index 214be80ee..2c322f1d7 100644
--- a/nextjs-frontend/components/settings/find.ts
+++ b/nextjs-frontend/components/settings/find.ts
@@ -16,7 +16,7 @@ import {
export const SETTINGS_GROUPS: { label: string; sections: SectionId[] }[] = [
{ label: "AI and search", sections: ["llm", "local-model", "ai", "search", "analysis"] },
{ label: "Speech and voices", sections: ["transcription", "speaker-separation", "voice-ids", "speech-providers"] },
- { label: "Media and documents", sections: ["uploads", "documents", "video", "nsfw", "reports", "iiif"] },
+ { label: "Media and documents", sections: ["uploads", "documents", "video", "nsfw", "enhance", "reports", "iiif"] },
{ label: "Chat and email", sections: ["telegram", "slack", "bridge", "mailbox", "mail", "notifications"] },
{ label: "Security and access", sections: ["sign-in", "tokens", "encryption", "access", "remote-access", "https"] },
{ label: "Files and devices", sections: ["storage", "shares", "smb", "nfs", "fedora", "sensors"] },
diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts
index 05684aa54..20ec9f01c 100644
--- a/nextjs-frontend/components/settings/model.ts
+++ b/nextjs-frontend/components/settings/model.ts
@@ -71,6 +71,7 @@ export type SectionId =
| "reports"
| "video"
| "nsfw"
+ | "enhance"
| "workers"
| "automations"
| "components"
@@ -197,6 +198,13 @@ export const SECTIONS: SectionSpec[] = [
description:
"Flag videos, documents and images with NSFW pictures, with a model on this server, and blur or hide them. Off until you turn it on here or a namespace's owner does for theirs.",
},
+ {
+ id: "enhance",
+ label: "Enhance",
+ backend: ["enhance"],
+ description:
+ "Find small, blurry, noisy or dull images and keep a sharper copy that Lens reads instead. The original stays as it is. Off until you turn it on.",
+ },
{
id: "workers",
label: "Workers",
@@ -409,6 +417,7 @@ export const SECTIONS: SectionSpec[] = [
const pct = (min = 0, max = 1) => ({ kind: "number" as const, min, max });
export const WORKER_STEPS = [
+ "enhance",
"transcribe",
"diarize",
"shots",
@@ -2218,6 +2227,68 @@ export const FIELDS: FieldSpec[] = [
max: 5000,
hint: "Spread evenly over a long video or document",
},
+ // Enhance
+ {
+ section: "enhance",
+ key: "images",
+ label: "Enhance low quality images",
+ kind: "switch",
+ hint: "Keeps a sharper copy of small, blurry, noisy or dull images and reads that instead; the original stays",
+ },
+ {
+ section: "enhance",
+ key: "image_engine",
+ label: "Engine",
+ kind: "select",
+ options: [
+ { value: "classic", label: "Classic: denoise, resize, contrast, sharpen (light)" },
+ { value: "realesrgan", label: "Real-ESRGAN upscaler (about 6 MB, slower)" },
+ ],
+ hint: "Real-ESRGAN is fetched the first time and runs on this server; very large images use the classic engine",
+ },
+ {
+ section: "enhance",
+ key: "only_faces",
+ label: "Only images with faces",
+ kind: "switch",
+ hint: "Needs face detection on in the namespace",
+ },
+ {
+ section: "enhance",
+ key: "min_side",
+ label: "Small below",
+ kind: "int",
+ min: 64,
+ max: 4000,
+ hint: "Pixels on the shorter side; smaller images are enlarged",
+ },
+ {
+ section: "enhance",
+ key: "min_sharpness",
+ label: "Blurry below",
+ kind: "number",
+ min: 0,
+ max: 10000,
+ hint: "Sharpness score (variance of the Laplacian); 0 never counts an image as blurry",
+ },
+ {
+ section: "enhance",
+ key: "min_face_px",
+ label: "Faces smaller than",
+ kind: "int",
+ min: 0,
+ max: 1024,
+ hint: "Pixels tall; an image with a smaller face is enlarged so recognition has more to go on. 0 turns this off",
+ },
+ {
+ section: "enhance",
+ key: "max_scale",
+ label: "Enlarge at most",
+ kind: "number",
+ min: 1,
+ max: 4,
+ hint: "Times the original size, and never past 4000 pixels a side",
+ },
// Workers
{
section: "workers",
diff --git a/nextjs-frontend/components/settings/settings-shell.tsx b/nextjs-frontend/components/settings/settings-shell.tsx
index 919b349ef..23f9e7bb6 100644
--- a/nextjs-frontend/components/settings/settings-shell.tsx
+++ b/nextjs-frontend/components/settings/settings-shell.tsx
@@ -46,6 +46,7 @@ import {
RefreshCw,
Network,
ScrollText,
+ WandSparkles,
Workflow,
} from "lucide-react";
import Link from "next/link";
@@ -87,6 +88,7 @@ const ICON: Record = {
reports: ChartNoAxesColumn,
video: Clapperboard,
nsfw: EyeOff,
+ enhance: WandSparkles,
workers: Cpu,
automations: Workflow,
components: PackageCheck,
diff --git a/nextjs-frontend/components/ui/loop.tsx b/nextjs-frontend/components/ui/loop.tsx
index b55f1da8c..1cb76a534 100644
--- a/nextjs-frontend/components/ui/loop.tsx
+++ b/nextjs-frontend/components/ui/loop.tsx
@@ -9,6 +9,7 @@ import { cn } from "@/lib/utils";
* upcoming ones are pale tints.
*/
export const STEP_TONE: Record = {
+ enhance: "intent",
transcribe: "intent",
read: "intent",
shots: "intent",
@@ -28,6 +29,7 @@ export const STEP_TONE: Record = {
};
export const STEP_LABEL: Record = {
+ enhance: "Enhance",
transcribe: "Transcribe",
read: "Read pages",
diarize: "Diarize",
diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json
index ff39c888f..6ddaa73bd 100644
--- a/nextjs-frontend/openapi.json
+++ b/nextjs-frontend/openapi.json
@@ -10636,6 +10636,49 @@
}
}
},
+ "/api/v1/resources/{rid}/enhance": {
+ "post": {
+ "tags": [
+ "resources"
+ ],
+ "summary": "Enhance Recording",
+ "description": "Enhance an image whatever the settings and its quality (docs/enhance.md): an enhanced copy is made, and its\npipeline's steps after enhance (reading its text, faces, objects, \u2026) run again on that copy. The original stays as\nit is. 400 for anything but an image. Audited as `recording.enhance`.",
+ "operationId": "enhance_recording",
+ "parameters": [
+ {
+ "name": "rid",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "integer",
+ "title": "Rid"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/JobQueued"
+ }
+ }
+ }
+ },
+ "422": {
+ "description": "Validation Error",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/HTTPValidationError"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
"/api/v1/resources/{rid}/share": {
"post": {
"tags": [
@@ -20150,6 +20193,55 @@
}
}
},
+ "/api/v1/resources/{rid}/enhanced": {
+ "get": {
+ "tags": [
+ "video"
+ ],
+ "summary": "Get Enhanced",
+ "description": "The enhanced copy the enhance step made of a low quality image (docs/enhance.md), to save. Members of its\nnamespace only (a bearer token, or a signed link the API gave one): visitors get its pages, with faces pixelated\nwhere the namespace does that.",
+ "operationId": "get_enhanced",
+ "parameters": [
+ {
+ "name": "rid",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "integer",
+ "title": "Rid"
+ }
+ },
+ {
+ "name": "s",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "string",
+ "default": "",
+ "title": "S"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "image/jpeg": {}
+ }
+ },
+ "422": {
+ "description": "Validation Error",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/HTTPValidationError"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
"/api/v1/resources/{rid}/frames/{name}": {
"get": {
"tags": [
@@ -40140,6 +40232,197 @@
"title": "EncryptionProgress",
"description": "Converting the files already kept after encryption.files changed: `to` \"encrypted\" or \"plain\" (vaults stay\nencrypted), how many changed and how many were skipped (a locked vault, a damaged file), and why a run stopped\nearly."
},
+ "EnhanceMeasured": {
+ "properties": {
+ "width": {
+ "type": "integer",
+ "title": "Width"
+ },
+ "height": {
+ "type": "integer",
+ "title": "Height"
+ },
+ "sharpness": {
+ "type": "number",
+ "title": "Sharpness",
+ "description": "the variance of its Laplacian: how sharply its tones change"
+ },
+ "noise": {
+ "type": "number",
+ "title": "Noise",
+ "description": "its estimated noise, in levels out of 255"
+ },
+ "contrast": {
+ "type": "number",
+ "title": "Contrast",
+ "description": "the standard deviation of its tones (0 to 255)"
+ },
+ "brightness": {
+ "type": "number",
+ "title": "Brightness",
+ "description": "the mean of its tones (0 to 255)"
+ },
+ "faces": {
+ "anyOf": [
+ {
+ "type": "integer"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Faces",
+ "description": "how many faces it has, where they were looked for"
+ },
+ "smallest_face": {
+ "anyOf": [
+ {
+ "type": "integer"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Smallest Face",
+ "description": "the height of its smallest face, in pixels"
+ }
+ },
+ "additionalProperties": true,
+ "type": "object",
+ "required": [
+ "width",
+ "height",
+ "sharpness",
+ "noise",
+ "contrast",
+ "brightness"
+ ],
+ "title": "EnhanceMeasured",
+ "description": "How good a picture looked to the enhance step (docs/enhance.md)."
+ },
+ "Enhanced": {
+ "properties": {
+ "kind": {
+ "type": "string",
+ "const": "image",
+ "title": "Kind"
+ },
+ "low": {
+ "items": {
+ "type": "string",
+ "enum": [
+ "small",
+ "blurry",
+ "noisy",
+ "dull",
+ "small_faces"
+ ]
+ },
+ "type": "array",
+ "title": "Low",
+ "description": "why it looked low quality; empty when it looked good enough"
+ },
+ "measured": {
+ "$ref": "#/components/schemas/EnhanceMeasured"
+ },
+ "enhanced": {
+ "type": "boolean",
+ "title": "Enhanced",
+ "description": "an enhanced copy was made, and the steps after it read that"
+ },
+ "engine": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Engine",
+ "description": "what enhanced it: classic, or realesr-general-x4v3 (Real-ESRGAN)"
+ },
+ "scale": {
+ "anyOf": [
+ {
+ "type": "number"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Scale",
+ "description": "how many times bigger the copy is"
+ },
+ "width": {
+ "anyOf": [
+ {
+ "type": "integer"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Width",
+ "description": "the copy's width in pixels"
+ },
+ "height": {
+ "anyOf": [
+ {
+ "type": "integer"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Height"
+ },
+ "forced": {
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Forced",
+ "description": "enhanced on request, whatever the settings and its quality"
+ },
+ "at": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "At"
+ },
+ "download": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Download",
+ "description": "a signed link to save the enhanced copy"
+ }
+ },
+ "additionalProperties": true,
+ "type": "object",
+ "required": [
+ "kind",
+ "measured",
+ "enhanced"
+ ],
+ "title": "Enhanced",
+ "description": "What the enhance step found and did (docs/enhance.md): whether the picture looked low quality and why, and the\nenhanced copy it's read from, if one was made. The original stays as it is."
+ },
"EnrichedDate": {
"properties": {
"text": {
@@ -57282,6 +57565,17 @@
}
],
"description": "NSFW classification, where it's on in its namespace and it was classified"
+ },
+ "enhanced": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/Enhanced"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "an image the enhance step looked at: what it found, and its enhanced copy"
}
},
"additionalProperties": true,
From a0e02468aac998f0ef228e3122d78b263ff2ba3b Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sat, 17 Oct 2026 01:47:38 +0000
Subject: [PATCH 2/8] test(imports): a transcript's import skips enhance too
Co-Authored-By: Claude Opus 5.5 (1M context)
Claude-Session: https://claude.ai/code/session_012m72U9mPLCbuH8rSAZGSuZ
---
fastapi_backend/tests/api/test_imports.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fastapi_backend/tests/api/test_imports.py b/fastapi_backend/tests/api/test_imports.py
index 516388354..df291543f 100644
--- a/fastapi_backend/tests/api/test_imports.py
+++ b/fastapi_backend/tests/api/test_imports.py
@@ -78,5 +78,5 @@ def test_choosing_the_pipeline_that_runs_after(client, db, cfg, folder):
assert [s["type"] for s in row["steps"]] == ["analyze", "report"] and row["pipeline"]["name"] == "Quick look"
# without one, the namespace's (here the standard one), less what a transcript has nothing to do for
job = client.post("/api/v1/import", headers=h, json={"namespace": "pods", "text": text + "\n[00:09] Ben: Later."}).json()["job"]
- standard = [t for t in pipelines.STANDARD if t not in ("transcribe", "diarize")]
+ standard = [t for t in pipelines.STANDARD if t not in ("enhance", "transcribe", "diarize")]
assert [s["type"] for s in db.one("SELECT steps FROM $j", j=store.R("job", job))["steps"]] == standard
From 4365af5d915a84cd6ace1ab4497e313821d3322f Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sat, 17 Oct 2026 02:00:25 +0000
Subject: [PATCH 3/8] docs(threat-model): the Real-ESRGAN model the enhance
step can fetch
Co-Authored-By: Claude Opus 5.5 (1M context)
Claude-Session: https://claude.ai/code/session_012m72U9mPLCbuH8rSAZGSuZ
---
threat-model.md | 1 +
threat-model.yaml | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/threat-model.md b/threat-model.md
index 622e7219a..68dc7bb75 100644
--- a/threat-model.md
+++ b/threat-model.md
@@ -457,6 +457,7 @@ third-party source; dependencies come from `uv.lock`, the pnpm lockfile and cont
| cloudflared, llama.cpp, `hf:` models, pip packages | integrity only as far as HTTPS and the publisher | upstream; tampered downloads are disclaimed (N-DOWNLOADS) | *(maintainer, 2026-10)* |
| Catalog GGUF models | pinned revision and SHA-256 | in model (P-MODEL-CHECKSUM) | *(documented, docs/local-models.md "checked against their SHA-256")* |
| Prompt-injection classifier (ProtectAI deberta-v3-base-prompt-injection-v2, `injection` extra, opt-in) | a judgement on whether a tool's result reads like an injection; the model is fetched at a pinned revision and checked against its SHA-256 | upstream for the model's judgement: a miss or a false flag is N-PROMPT; a model file that passes unchecked is in model | *(documented, docs/assistant.md "fetched when you pick it")* |
+| Real-ESRGAN general x4v3 (`enhance.image_engine` `realesrgan`, opt-in) | enlarging and cleaning up low quality pictures before they're read; the ONNX model is fetched at a pinned revision and checked against its SHA-256, and runs in ONNX Runtime on the CPU in fixed tiles of pictures Lens has already opened and size-checked | upstream for ONNX Runtime and the model's output; a model file that passes unchecked is in model | *(documented, docs/enhance.md "fetched the first time the step runs with it chosen")* |
| Optional ML extras (whisper, pyannote, speechbrain, insightface, doctr) | model inference; some fetch weights on their own | upstream | *(inferred, Q19)* |
| StyleTTS2 service (`styletts2/`, opt-in Compose profile) and macOS `say` | text to speech; StyleTTS2 fetches its weights from Hugging Face and GitHub on first use and listens only on the Compose network; `say` gets the text on stdin, never as arguments | upstream for the model and synthesis; the argument handling in model | *(maintainer, 2026-10)* |
| MCP runner (`mcp-runner/`, opt-in Compose profile) and the stdio MCP servers admins add (usually `npx` or `uvx` packages) | running those servers away from Lens: a container with none of Lens's data, keys or database, a non-root user, every capability dropped, `no-new-privileges`, a read-only image, memory and process limits, a network shared only with the backend (plus the internet to fetch packages), an empty home and only its extension's environment per server, and a bearer token on a volume only the two share | upstream for the servers and the packages they fetch (an admin's choice, trusted input, §1.3); the runner's isolation and token check in model | *(documented, docs/assistant.md "The MCP runner")* |
diff --git a/threat-model.yaml b/threat-model.yaml
index 0af7b1773..8e7574141 100644
--- a/threat-model.yaml
+++ b/threat-model.yaml
@@ -2,7 +2,7 @@
# Authority order: prose > yaml > json. Regenerate whenever the prose changes.
schema: threat-model-sidecar/v2
project: lens
-prose_version: "threat-model.md@sha256:1a60d0746521aec2bb7513e87bca41a4be26c912cc815a6eb0a9e767d732f461"
+prose_version: "threat-model.md@sha256:e961adc317bb2dd7f4a2ab5ea6d3ed031e7df960c11d1a434df87641a2007ac7"
model_status: under-review
triage_policy: strict
confidence: {documented: 202, maintainer: 46, inferred: 91, assumption: 8}
From 9934c3fb8d44c1c9dc365591064aa2fd19544f8a Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sat, 17 Oct 2026 02:20:38 +0000
Subject: [PATCH 4/8] docs(configuration): rewrap the Settings paragraph
---
docs/configuration.md | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 20487e02a..7d42386f9 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -107,12 +107,13 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th
## Settings in the app
Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM
-provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word
-search and search by meaning), reports and graph, video, OCR, faces and objects, [NSFW classification](nsfw.md),
-[enhancing images](enhance.md), workers, sign-in, components (and the edition, full or lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript
-upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
-automations, telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys, and IIIF and metadata. The API
-refuses an allowed-host list that leaves out the address you are using.
+provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word search
+and search by meaning), reports and graph, video, OCR, faces and objects, [NSFW classification](nsfw.md), [enhancing
+images](enhance.md), workers, sign-in, components (and the edition, full or lean), access and embedding (whether other
+devices on the network reach Lens, embed frame ancestors, text and transcript upload limit, allowed hosts, trusted
+proxies, session length), remote access, email, chat rooms, notifications, automations, telemetry, the Fedora
+repository, sensors, file shares, uploads, storage, documents, API keys, and IIIF and metadata. The API refuses an
+allowed-host list that leaves out the address you are using.
The sections are in groups: AI and search, Speech and voices, Media and documents, Chat and email, Security and
access, Files and devices, and System. **Find a setting** at the top of the list (or `/`) finds a section by its name
From 39479729f1bdfe7c23abade3fe67516ff94ef65e Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sat, 17 Oct 2026 04:38:41 +0000
Subject: [PATCH 5/8] docs(readme): enhancing images is built, not in progress
---
README.md | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 50a4fa519..60095e18a 100644
--- a/README.md
+++ b/README.md
@@ -206,8 +206,7 @@ These are open pull requests, not yet part of Lens:
- **Notes:** download a note, or all of a namespace's notes, as Markdown
([#419](https://github.com/adeelahmad/lense/pull/419)).
- **Sources:** RSS and Atom news feeds ([#453](https://github.com/adeelahmad/lense/pull/453)).
-- **Pictures:** described on this machine with Florence-2 ([#477](https://github.com/adeelahmad/lense/pull/477)), and
- low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
+- **Pictures:** described on this machine with Florence-2 ([#477](https://github.com/adeelahmad/lense/pull/477)).
- **Entities:** names given their real types instead of Term ([#395](https://github.com/adeelahmad/lense/pull/395)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
From 3cc0f74d3d5d7a648be249cadfac6682958447bc Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sun, 18 Oct 2026 03:04:45 +0000
Subject: [PATCH 6/8] docs(threat-model): list Real-ESRGAN among the
dependencies
The threat model's YAML now changes with the prose, as the validator asks
since main's same-commit scheme.
Co-Authored-By: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_012m72U9mPLCbuH8rSAZGSuZ
---
threat-model.yaml | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/threat-model.yaml b/threat-model.yaml
index 061ada6d9..5da975471 100644
--- a/threat-model.yaml
+++ b/threat-model.yaml
@@ -8,7 +8,7 @@ project: lens
prose_version: "threat-model.md@same-commit"
model_status: under-review
triage_policy: strict
-confidence: {documented: 208, maintainer: 46, inferred: 91, assumption: 8}
+confidence: {documented: 209, maintainer: 46, inferred: 91, assumption: 8}
generation:
model: "Claude Opus 5.5 (claude-opus-5-5)"
@@ -1120,6 +1120,15 @@ dependencies:
covered_here: true
violation_disposition: "OUT-OF-MODEL: dependency-contract"
provenance: {kind: documented, source: 'docs/local-models.md "checked against their SHA-256"'}
+ - name: "Real-ESRGAN general x4v3 (ONNX, enhance.image_engine realesrgan, opt-in)"
+ relied_on_for: "enlarging and cleaning up low quality pictures before they're read; fetched at a pinned revision and checked against its SHA-256, run in ONNX Runtime on the CPU in fixed tiles of pictures Lens has already opened and size-checked; a model file that passes unchecked is in model"
+ relied_on_properties: [P-MODEL-CHECKSUM, model-inference]
+ caller_obligations_acknowledged: []
+ adversary_capabilities_forwarded: [supply-parsed-bytes]
+ outputs_consumed: []
+ covered_here: false
+ violation_disposition: "OUT-OF-MODEL: dependency-contract"
+ provenance: {kind: documented, source: 'docs/enhance.md "fetched the first time the step runs with it chosen"'}
- name: "Optional ML extras (whisper, pyannote, speechbrain, insightface, doctr)"
relied_on_for: "model inference; some fetch weights on their own"
relied_on_properties: [model-inference]
From df70dca93a2cfb3be511dd553cf89626800ad5f4 Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sun, 18 Oct 2026 07:33:29 +0000
Subject: [PATCH 7/8] docs(readme): answering about Lens itself is built, not
in progress
#400 is released, so the requirements check on main was red; its In
progress line goes, and its checks join the built list.
Co-Authored-By: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_012m72U9mPLCbuH8rSAZGSuZ
---
README.md | 8 ++++----
docs/requirements.md | 1 +
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index 36beb9496..b4c238e67 100644
--- a/README.md
+++ b/README.md
@@ -87,13 +87,15 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
nothing is lost. Each answer keeps a record of what produced it (the model, its tools, how long it took and what it
cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine
(opt-in), any message in one click. Long conversations are summarised before they fill the model's context, and
- `/clear` and `/compact` start afresh or summarise now. Conversations title themselves after their first answer, unless
+ `/clear` and `/compact` start afresh or summarise now. Questions about Lens itself (how do I back up, is Telegram
+ on) are answered from its docs and this server, cited. Conversations title themselves after their first answer, unless
you name them. Each namespace can have its own assistant with
its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in
Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you
email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages),
[mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records),
- [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md);
+ [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md),
+ [questions about Lens itself](docs/assistant.md#questions-about-lens-itself);
[checks](docs/requirements.md#ask-and-get-cited-answers))
- **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python),
skills, hooks and MCP servers, by code, on the canvas or just by asking. Remote MCP servers connect by address, and
@@ -236,8 +238,6 @@ still loading. ([Components](docs/components.md),
These are open pull requests, not yet part of Lens:
-- **The assistant** answering questions about Lens itself, from its docs and this server
- ([#400](https://github.com/adeelahmad/lense/pull/400)).
- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
diff --git a/docs/requirements.md b/docs/requirements.md
index 7c53bfeb7..1d5f052d8 100644
--- a/docs/requirements.md
+++ b/docs/requirements.md
@@ -90,6 +90,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` |
| Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` |
| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again`, `test_what_tools_returned_is_cut_to_fit_the_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app`; [chat-summary-mark.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-summary-mark.test.tsx) `says how many messages the summary stands for, and shows it on request` |
+| Questions about Lens itself answered from its docs and this server, cited, for people and agents | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_assistant_answers_about_lens_itself`, `test_admins_see_where_each_setting_is`, `test_agents_ask_lens_over_mcp` |
| A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` |
| Telegram and Slack built in, with voice notes and files | [test_telegram.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_telegram.py) `test_it_answers_private_chats_and_what_is_said_to_it_in_groups`, `test_voice_notes_are_heard`, `test_files_are_held_for_the_assistant_to_import`; [test_slack.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_slack.py) `test_it_answers_direct_messages_mentions_and_the_slash_command`, `test_voice_clips_are_heard_and_files_held` |
| Other chat rooms through Matterbridge | [test_bridge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_bridge.py) `test_it_answers_what_is_said_to_it_and_nothing_else`; [test_matterbridge_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_matterbridge_run.py) `test_chat_networks_are_set_in_the_app_and_become_matterbridges_config`, `test_the_bridge_talks_to_the_matterbridge_lens_runs`; [matterbridge.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/matterbridge.yml) `Lens writes a config, and Matterbridge starts with it` |
From 924b85fa7b258d90932e4f2dc9d1fd4cec79b6a3 Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sun, 18 Oct 2026 08:48:50 +0000
Subject: [PATCH 8/8] docs(readme): keep the enhance step in the built list
after merging main
Co-Authored-By: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_012m72U9mPLCbuH8rSAZGSuZ
---
README.md | 27 +++++++++++++--------------
docs/requirements.md | 1 +
2 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/README.md b/README.md
index e7863f539..bc7017eac 100644
--- a/README.md
+++ b/README.md
@@ -123,20 +123,20 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser))
- **Understand it.** Transcription, people recognised by voice (long recordings go to a speech provider in pieces that
fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
- server (off by default, per namespace), entities and topics in a knowledge graph you can explore, query with Cypher or
- SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
- (`/wiki//`), and entities, topics and people are edited and merged from their lists, the graph or an
- asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
- rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an
- asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos
- by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by
- Florence-2 or Moondream on your own server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing
- and, with a local model, punctuation put back), entities found by spaCy instead of the rules, and the dates people say
- read as days on the calendar, each off by default.
+ server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities and
+ topics in a knowledge graph you can explore, query with Cypher or SPARQL, and see as it was at any version. Every
+ entity has its own page at a wiki address (`/wiki//`), and entities, topics and people are edited and
+ merged from their lists, the graph or an asset, with Undo instead of a confirm. Names get their real types (people,
+ places, organisations, events, works) by rules, without a model, and a decision model types the ones the rules can't.
+ Find what is most like a passage, an asset, a topic or an entity, in a Similar tab on every asset or by selecting text
+ anywhere, and, with anytopdf, photos by what they show. Pages and shots are described in words by the model provider's
+ vision model or, off by default, by Florence-2 or Moondream on your own server. Transcripts can be cleaned up before
+ analysis (stutters, "um"s, spacing and, with a local model, punctuation put back), entities found by spaCy instead of
+ the rules, and the dates people say read as days on the calendar, each off by default.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
- [long recordings](docs/speech-providers.md#long-recordings),
+ [enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
@@ -244,7 +244,6 @@ still loading. ([Components](docs/components.md),
These are open pull requests, not yet part of Lens:
- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
-- **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
@@ -300,8 +299,8 @@ The guides are on the [docs site](https://adeelahmad.github.io/lense/) and in th
[the assistant by email](docs/assistant-email.md), [podcasts](docs/podcasts.md), [notes](docs/notes.md),
[web clipper](docs/web-clipper.md), [the assistant in your browser](docs/browser-assistant.md),
[sensors](docs/sensors.md), [processing](docs/processing.md), [video](docs/video.md),
- [NSFW classification](docs/nsfw.md), [notifications](docs/notifications.md), [automations](docs/automations.md),
- [timeline](docs/timeline.md)
+ [NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md), [notifications](docs/notifications.md),
+ [automations](docs/automations.md), [timeline](docs/timeline.md)
- **Knowledge:** [graph and Cypher](docs/graph.md), [graph history](docs/graph-history.md), [topics](docs/topics.md),
[the graph as files](docs/file-tree.md), [NFS share](docs/file-shares.md), [SMB share](docs/smb.md),
[linked data (RDF)](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md)
diff --git a/docs/requirements.md b/docs/requirements.md
index 46236ad24..39178d48a 100644
--- a/docs/requirements.md
+++ b/docs/requirements.md
@@ -147,6 +147,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
| Faces, off by default | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_people_on_screen`, `test_shots_text_search_chat_and_iiif`; [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_video.py) `test_face_modes` |
| NSFW pictures flagged by a model on your own server, off by default, per namespace | [test_nsfw.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_nsfw.py) `test_off_by_default_nothing_is_fetched_or_classified`, `test_a_namespace_classifies_flags_blurs_and_hides`; [nsfw.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/nsfw.test.tsx) |
+| Low quality images enhanced before they're read, off by default | [test_enhance.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_enhance.py) `test_off_by_default_the_original_is_read`, `test_a_low_quality_image_is_read_from_its_enhanced_copy`, `test_a_good_image_is_left_as_it_is`, `test_agents_and_automations_enhance_too`, `test_the_copy_is_encrypted_where_the_files_are`; [enhance.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/enhance.test.tsx) |
| Entities and topics in a knowledge graph to explore and query with Cypher or SPARQL | [test_graph.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph.py) `test_query`; [test_rdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rdf.py) `test_sparql`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `graph-find` |
| The graph as it was at any version | [test_graph_history.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph_history.py) `test_the_explorer_and_cypher_as_of_a_version`, `test_the_canvas_as_of_a_version`; [test_graph_history.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_graph_history.py) `test_every_curation_is_an_event_and_every_version_replays` |
| Every entity's own page at `/wiki//` | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_entities.py) `test_wiki_addresses`; [wiki-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/wiki-page.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-find` |