Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
fixed, by rules, on English lines. Every changed line is in Change history with Revert, and lines someone corrected
are left alone. Reprocess, pipelines, `POST /api/v1/recordings/<id>/reprocess` and `lens clean` run it
([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)).
- **Describe pictures on this machine with Florence-2.** Settings → LLM provider → Describing pictures can now have
Microsoft's Florence-2 (MIT) describe each page and shot instead of the provider's model that can see images
(`llm.describe_engine` `florence-2`). It runs on the CPU through ONNX Runtime, so no picture leaves the machine and
no provider is needed; its 8-bit export is fetched the first time a job waits to describe something, pinned and
checked against its SHA-256: base (about 280 MB, `llm.florence_model`) or large (about 830 MB). Each picture is a
`model.describe` row in the activity ledger. Agents read and change it with the MCP tools `picture_descriptions` and
`set_picture_descriptions`. The default stays the provider's model ([Descriptions](docs/configuration.md#descriptions)).
- **Automations run.** An automation saved over the API (`/api/v1/automations`) now reacts to new assets in its
namespaces: when its If passes, it runs its workflow as the person who made it, within its limits and budgets, and
each run shows what its actions did. Try one on an activity without keeping or sending anything, or run it now;
Expand Down
18 changes: 18 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,24 @@ from a LanguageTool server, spaCy fetched for named entities, and dates found an

Refine later: revert a whole cleanup at once; cleanup rules for other languages' fillers; per-namespace switch.

## 2026-10-17 · Describing pictures with Florence-2 on this machine

Adeel: use Moondream 3 and/or Microsoft Florence for visual descriptions. Florence-2 (MIT) is now a describe engine
(`llm.describe_engine` `florence-2`): its 8-bit ONNX export runs on ONNX Runtime on the CPU, fetched on first use,
pinned and hashed, base or large. It runs a Raspberry Pi's way (no PyTorch, about 280 MB for base) and needs no model
provider. Checked on two photos here: base takes about 3 s a picture on four CPU cores, large about 9 s, and both say
plainly what each shows.

Moondream 3 comes next as an opt-in engine. Its licences aren't open: Moondream 3 Preview's weights are BSL 1.1 (with
a production use grant; Apache-2.0 two years after release) and Moondream 3.1's are the Moondream Model License; 3.1
runs only on Photon, whose kernels (`kestrel-kernels`) are proprietary and licensed only under a separate agreement, so
Lens won't install Photon. Moondream 3 Preview runs through Hugging Face transformers (permissive) on an NVIDIA GPU or
a Mac with enough memory (about 19 GB of weights): not on a Pi.

Refine later: beam search (the model's own default is three beams; greedy with no repeated trigrams matches it on the
photos tried); Florence-2's OCR task beside the OCR step; a GPU (CUDA or CoreML execution provider) when there is one;
the Q4 exports for a smaller download; describing only the pictures NSFW classification didn't flag.

## 2026-10-15 · Entity types beyond Term

Adeel: "every entity comes as term? why!!". The default rules extractor typed a capitalised name PERSON only after a
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
rules, without a model. Find what is most like a passage, an asset, a topic or an entity, in a Similar
tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show.
tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show. Pages and shots are
described in words by the model provider's vision model or, off by default, by Florence-2 on your own server.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
Expand Down Expand Up @@ -207,8 +208,7 @@ These are open pull requests, not yet part of Lens:
([#400](https://github.com/adeelahmad/lense/pull/400)).
- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
- **Sources:** RSS and Atom news feeds ([#453](https://github.com/adeelahmad/lense/pull/453)).
- **Pictures:** described on this machine with Florence-2 ([#477](https://github.com/adeelahmad/lense/pull/477)), and
low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
- **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
Expand Down
6 changes: 3 additions & 3 deletions docs/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ its namespace, its pipeline and the routine. A request counts for the resources
/api/v1/routines/3` is `routine:3`), for what a `POST` made (from the `id` in its reply), and for who sent it.

Besides the resources, a row has its kind (`in`, `out` or `run`), what it was (`POST /api/v1/routines/{rid}`,
`model.chat`, `embeddings`, `decision`, `notify.webhook`, `tool.http`, `web.search`, `web.read`, `google.token`,
`gmail.list`, `job.succeeded`, `routine.done`), the model,
tokens in and out, the estimated cost in USD, how long it took, and how it ended (an error's type or HTTP status,
`model.chat`, `model.describe` (a picture Florence-2 described on this server), `embeddings`, `decision`,
`notify.webhook`, `tool.http`, `web.search`, `web.read`, `google.token`, `gmail.list`, `job.succeeded`,
`routine.done`), the model, tokens in and out, the estimated cost in USD, how long it took, and how it ended (an error's type or HTTP status,
never its message). It never holds prompt, reply, asset text or file content.

## Costs
Expand Down
16 changes: 9 additions & 7 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1364,14 +1364,16 @@ photo search results carry `nsfw: true` for the ones the web app blurs.

### Descriptions

The `describe` step has a model that can see images, the one an admin chose as `llm.vision_model`, say in a few
sentences what each page of a document or an image shows, and each shot of a video (from its keyframe): the setting,
the people (by what they do and wear, never who they might be), the things and any text in it. It comes after
`nsfw` in the standard pipeline, and describes at most `llm.describe_max` pages or shots of a resource. Without an
LLM, or with no `llm.vision_model`, it's skipped, saying why; a model that can't see makes it fail with what the server
said ([Configuration](configuration.md#descriptions)). The player's `descriptions` lists them in order:
The `describe` step has a model that can see images say in a few sentences what each page of a document or an image
shows, and each shot of a video (from its keyframe): the setting, the people (by what they do and wear, never who they
might be), the things and any text in it. The model is the one an admin chose as `llm.vision_model` on the LLM
provider, or, with `llm.describe_engine` `florence-2`, Florence-2 on the server itself, which needs no provider. It
comes after `nsfw` in the standard pipeline, and describes at most `llm.describe_max` pages or shots of a resource.
Without an LLM, with no `llm.vision_model`, or before Florence-2 is fetched, it's skipped, saying why; a model that
can't see makes it fail with what the server said ([Configuration](configuration.md#descriptions)). The player's `descriptions` lists them in order:
`{idx, t0, t1, text, model, frame, paged}`: the shot or page, where it starts and ends (ms; on pages, the page from 0
and the next), what the model said, which model, and a signed link to the picture it was shown. Search finds them
and the next), what the model said, which model (such as `Florence-2-base`), and a signed link to the picture it was
shown. Search finds them
(`source: described`, [search](#search)). Moving or deleting a resource moves or deletes them; the step run again
replaces them.

Expand Down
4 changes: 3 additions & 1 deletion docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ never fetch a speech or voice model.
| Face models | `video.face_engine` is `anytopdf` | SFace, for anytopdf to describe faces with (anytopdf too) |
| Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` |
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Florence-2 | `llm.describe_engine` is `florence-2` | Florence-2-base-ft or -large-ft (MIT, `llm.florence_model`; about 280 or 830 MB), their 8-bit ONNX exports, the tokenizers library and ONNX Runtime, to [describe pictures](configuration.md#descriptions) on this machine |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| spaCy | `analysis.entities` is `spacy` ([Entities](processing.md#entities-and-the-graph-explorer)) | spaCy (MIT) and the model in `analysis.spacy_model`: `en_core_web_sm`, `de_core_news_sm` or `xx_ent_wiki_sm` (MIT, 11 to 15 MB each), about 60 MB in all |
Expand All @@ -32,7 +33,8 @@ never fetch a speech or voice model.
- Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win.
- Models go into `data_dir/models`: the Hugging Face, ModelScope and PyTorch caches point there (unless `HF_HOME`,
`MODELSCOPE_CACHE` or `TORCH_HOME` say otherwise). They outlast container rebuilds and upgrades.
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model, spaCy's models) are checked against their SHA-256.
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model, Florence-2, spaCy's models) are checked against their
SHA-256.
- A job step that needs something still being fetched waits for it instead of falling back or failing. If a fetch fails,
the step runs as before (for transcription, falling back to another engine), and the fetch is tried again an hour
later, when the settings change, or on **Check again now**.
Expand Down
31 changes: 24 additions & 7 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,18 +219,35 @@ Settings → AI assistant → Web search; the setup chat always asks before it c

## Descriptions

The describe step ([API](api.md#descriptions)) sends each page of a document or an image, and each shot's keyframe of
a video (at most 1024 pixels on a side), to a model that can see images, and keeps what it says each shows, for
search and for people who can't see them. It uses the LLM provider's server (Settings → LLM provider) with a model you
choose there as one that can see images: OpenAI's, or a vision model on LM Studio, Ollama or vLLM (LLaVA, Qwen-VL,
Llama 3.2 Vision, Gemma 3 …). Lens can't tell which models see, so none is chosen until you choose one. Each page or
shot is a request, so a long document costs as many: `llm.describe_max` caps how many of a resource are described.
The describe step ([API](api.md#descriptions)) shows each page of a document or an image, and each shot's keyframe of
a video, to a model that can see images, and keeps what it says each shows, for search and for people who can't see
them. Settings → LLM provider → Describing pictures chooses the model (`llm.describe_engine`):

- **The provider's model that can see images** (`llm`, the default). Each picture (at most 1024 pixels on a side) is a
request to the LLM provider's server, with a model you choose as one that can see images: OpenAI's, or a vision
model on LM Studio, Ollama or vLLM (LLaVA, Qwen-VL, Llama 3.2 Vision, Gemma 3 …). Lens can't tell which models see,
so none is chosen until you choose one.
- **Florence-2 on this server** (`florence-2`). Microsoft's Florence-2 (MIT), fine-tuned, describes each picture in a
paragraph on the worker's CPU, through ONNX Runtime: no picture leaves the machine, no provider is needed and nothing
is billed. Its 8-bit ONNX export is fetched the first time a job waits to describe something
([Components](components.md)), pinned to a revision and checked against its SHA-256: base (0.23B parameters, about
280 MB; a few seconds a picture on a laptop's CPU, longer on a Raspberry Pi) or large (0.77B, about 830 MB, more
detail and about three times slower). It says what it sees in its own words: it reads no instructions, so it isn't
told what the picture is, and it describes, rather than transcribes, the text in it (the OCR step reads that).

Each page or shot is a call to the model, so a long document costs as many: `llm.describe_max` caps how many of a
resource are described. Each call is in the [activity ledger](activity.md) (`model.chat` for the provider's model,
`model.describe` for Florence-2, with the tokens read and written and the time it took).

| Setting | Default | |
|---|---|---|
| `llm.vision_model` | none | the model that describes pages and shots; none: the describe step is skipped |
| `llm.describe_engine` | `llm` | what describes pages and shots: `llm` (the provider's model below) or `florence-2` (on this server) |
| `llm.vision_model` | none | the provider's model that describes pages and shots; none (with `llm`): the describe step is skipped |
| `llm.florence_model` | `base` | Florence-2's size: `base` or `large` |
| `llm.describe_max` | 50 | pages or shots of a resource described at most, 1–1000 (in order: the rest aren't) |

Agents read and change these with the MCP tools `picture_descriptions` and `set_picture_descriptions` (admins).

## Uploads

Audio, video, documents (PDF, Office and OpenDocument files, text, Markdown, saved web pages and emails) and images
Expand Down
4 changes: 3 additions & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `preview_template` | what a template in a drafted automation gives before it's saved, against an activity or the trigger kind's sample; a prompt shows outside text marked as the model sees it; an error names the missing variable and its nearest name |
| `file_storage` | where Lens keeps its own files (notes' images and attachments) and the connections it could use; `check` tests it. Admins only |
| `set_file_storage` | keep new files on this machine or a storage connection (folder, rclone crypt); checked before it's saved. Admins only, with a write-scope token |
| `picture_descriptions` | what describes the pages and shots of resources ([Descriptions](configuration.md#descriptions)): the LLM provider's model that can see images or Florence-2 on this server, its model, whether it's ready and why not, and the engines there are. Admins only |
| `set_picture_descriptions` | choose what describes pictures (`engine` `llm` or `florence-2`), the provider's `vision_model`, Florence-2's size (`florence_model`) and `describe_max`. Admins only, with a write-scope token |
| `install_settings` | whether other devices on the network reach Lens and its edition (full or lean), whether the machine's upgrade agent can change them, and the install line to run when it can't. Admins only |
| `set_install_settings` | open Lens to the network or keep it to this machine, or change the edition: the machine sets itself up again and Lens restarts ([Updates](updates.md#setting-the-machine-up-again)). Admins only, with a write-scope token |
| `propose_graph_change` | ask for two entities to be merged or linked; it waits in Proposed changes unless `apply` makes it at once (it can be undone). Needs a write-scope token and editor access |
Expand All @@ -139,7 +141,7 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed
`merge_topics`, `delete_topic` and `undo_topic_change`, which curate entities and topics as an editor does in the web
app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which
queues an episode, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring`,
editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring`,
`set_sandbox_gateway` (the [code sandbox](assistant.md#code-sandbox)'s gateway) and `set_install_settings`, which have
the machine change how Lens runs (admins).
Importing stays in the web app and the API.
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Names get their real types (people, places, organisations, events, works) by rules, without a model; older Terms are typed on upgrade | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_names_get_the_type_their_words_give`, `test_analysis_types_people_places_and_events`, `test_the_upgrade_types_the_old_terms_and_keeps_what_people_chose` |
| What is most like a passage, an asset, a topic or an entity | [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities`, `test_like_a_topic_and_an_entity`; [similar.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/similar.test.tsx) |
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |

## Learn from it

Expand Down
Loading
Loading