Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
fixed, by rules, on English lines. Every changed line is in Change history with Revert, and lines someone corrected
are left alone. Reprocess, pipelines, `POST /api/v1/recordings/<id>/reprocess` and `lens clean` run it
([Cleaning up transcripts](docs/processing.md#cleaning-up-transcripts)).
- **Describe pictures on this machine with Moondream.** Settings → LLM provider → Describing pictures can also have
M87 Labs' Moondream caption each page and shot (`llm.describe_engine` `moondream`), through transformers and
PyTorch, so no picture leaves the machine: Moondream 3 Preview (`llm.moondream_model` `moondream3-preview`, about
19 GB, an NVIDIA GPU or a Mac with 32 GB; BSL 1.1, not open source) or Moondream 2 (`moondream2`, Apache-2.0, about
3.7 GB, on a CPU, slowly). It's fetched, pinned to a revision, the first time a job waits to describe something, and
only where the chosen one can run; elsewhere the step says why. Agents choose it with `set_picture_descriptions`
([Descriptions](docs/configuration.md#descriptions)).
- **Describe pictures on this machine with Florence-2.** Settings → LLM provider → Describing pictures can now have
Microsoft's Florence-2 (MIT) describe each page and shot instead of the provider's model that can see images
(`llm.describe_engine` `florence-2`). It runs on the CPU through ONNX Runtime, so no picture leaves the machine and
Expand Down
21 changes: 21 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,27 @@ from a LanguageTool server, spaCy fetched for named entities, and dates found an

Refine later: revert a whole cleanup at once; cleanup rules for other languages' fillers; per-namespace switch.

## 2026-10-17 · Describing pictures with Moondream on this machine

Moondream is the third describe engine (`llm.describe_engine` `moondream`), opt-in, through Hugging Face transformers
(Apache-2.0) and PyTorch (BSD), the `moondream` extra fetched on first use. Both models' code comes with their
weights (`trust_remote_code`), pinned to a revision; both caption with the same call (`caption(image,
length="normal")`). `llm.moondream_model` picks Moondream 3 Preview (the default when Moondream is chosen, since that's
what was asked for) or Moondream 2. `moondream.cannot_run` keeps 19 GB from being fetched where Moondream 3 can't run
(no NVIDIA GPU with 20 GB, no Mac with 32 GB) and says so in the step's log and in `picture_descriptions`.

Checked here: Moondream 2 at the pinned revision loads with the locked transformers and describes a photo well ("A
vintage light blue Volkswagen Beetle is parked parallel to a yellow building with a wooden door …"), but took 3 to 6
minutes a picture on four shared CPU cores in float32: Florence-2 base took 3 s. Moondream 3 Preview wasn't
run here (no GPU); its code at the pinned revision was read: the same caption call, with flex attention. The opt-in
test (`LENS_TEST_MOONDREAM=1`) fetches PyTorch's CPU build, transformers and Moondream 2 through the component and
describes a red dot. It first wrote "and, (, (, (…": with `OMP_THREAD_LIMIT` below PyTorch's thread count (the test
suite sets it to 1), PyTorch's CPU kernels write nonsense, so `_load` now caps PyTorch's threads at that limit.

Refine later: Moondream's query skill with what the picture is ("page 3 of …"), as the provider's model is told; its
`compile()` on a GPU; bfloat16 on CPUs that have it; Moondream 3 on a GPU in CI or on a maintainer's machine; a
quantised Moondream 2 for small machines.

## 2026-10-17 · Describing pictures with Florence-2 on this machine

Adeel: use Moondream 3 and/or Microsoft Florence for visual descriptions. Florence-2 (MIT) is now a describe engine
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
rules, without a model. Find what is most like a passage, an asset, a topic or an entity, in a Similar
tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show. Pages and shots are
described in words by the model provider's vision model or, off by default, by Florence-2 on your own server.
Transcripts can be cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation put
back), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each off
by default.
described in words by the model provider's vision model or, off by default, by Florence-2 or Moondream on your own
server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation
put back), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each
off by default.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
Expand Down
2 changes: 1 addition & 1 deletion docs/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ its namespace, its pipeline and the routine. A request counts for the resources
/api/v1/routines/3` is `routine:3`), for what a `POST` made (from the `id` in its reply), and for who sent it.

Besides the resources, a row has its kind (`in`, `out` or `run`), what it was (`POST /api/v1/routines/{rid}`,
`model.chat`, `model.describe` (a picture Florence-2 described on this server), `embeddings`, `decision`,
`model.chat`, `model.describe` (a picture Florence-2 or Moondream described on this server), `embeddings`, `decision`,
`notify.webhook`, `tool.http`, `web.search`, `web.read`, `google.token`, `gmail.list`, `job.succeeded`,
`routine.done`), the model, tokens in and out, the estimated cost in USD, how long it took, and how it ended (an error's type or HTTP status,
never its message). It never holds prompt, reply, asset text or file content.
Expand Down
9 changes: 6 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1381,12 +1381,15 @@ photo search results carry `nsfw: true` for the ones the web app blurs.
The `describe` step has a model that can see images say in a few sentences what each page of a document or an image
shows, and each shot of a video (from its keyframe): the setting, the people (by what they do and wear, never who they
might be), the things and any text in it. The model is the one an admin chose as `llm.vision_model` on the LLM
provider, or, with `llm.describe_engine` `florence-2`, Florence-2 on the server itself, which needs no provider. It
provider, or, with `llm.describe_engine` `florence-2` or `moondream`, Florence-2 or Moondream on the server itself,
which needs no provider. It
comes after `nsfw` in the standard pipeline, and describes at most `llm.describe_max` pages or shots of a resource.
Without an LLM, with no `llm.vision_model`, or before Florence-2 is fetched, it's skipped, saying why; a model that
Without an LLM, with no `llm.vision_model`, before Florence-2 or Moondream is fetched, or where the chosen Moondream
can't run, it's skipped, saying why; a model that
can't see makes it fail with what the server said ([Configuration](configuration.md#descriptions)). The player's `descriptions` lists them in order:
`{idx, t0, t1, text, model, frame, paged}`: the shot or page, where it starts and ends (ms; on pages, the page from 0
and the next), what the model said, which model (such as `Florence-2-base`), and a signed link to the picture it was
and the next), what the model said, which model (such as `Florence-2-base` or `Moondream-2`), and a signed link to
the picture it was
shown. Search finds them
(`source: described`, [search](#search)). Moving or deleting a resource moves or deletes them; the step run again
replaces them.
Expand Down
1 change: 1 addition & 0 deletions docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ never fetch a speech or voice model.
| Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` |
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Florence-2 | `llm.describe_engine` is `florence-2` | Florence-2-base-ft or -large-ft (MIT, `llm.florence_model`; about 280 or 830 MB), their 8-bit ONNX exports, the tokenizers library and ONNX Runtime, to [describe pictures](configuration.md#descriptions) on this machine |
| Moondream | `llm.describe_engine` is `moondream`, and this machine can run the chosen one | Moondream 3 Preview (BSL 1.1, about 19 GB, an NVIDIA GPU or a Mac with 32 GB) or Moondream 2 (Apache-2.0, about 3.7 GB), `llm.moondream_model`, at a pinned revision, with its tokenizer, PyTorch (the CPU build without an NVIDIA GPU) and transformers, to [describe pictures](configuration.md#descriptions) on this machine |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| spaCy | `analysis.entities` is `spacy` ([Entities](processing.md#entities-and-the-graph-explorer)) | spaCy (MIT) and the model in `analysis.spacy_model`: `en_core_web_sm`, `de_core_news_sm` or `xx_ent_wiki_sm` (MIT, 11 to 15 MB each), about 60 MB in all |
Expand Down
19 changes: 17 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,16 +234,31 @@ them. Settings → LLM provider → Describing pictures chooses the model (`llm.
280 MB; a few seconds a picture on a laptop's CPU, longer on a Raspberry Pi) or large (0.77B, about 830 MB, more
detail and about three times slower). It says what it sees in its own words: it reads no instructions, so it isn't
told what the picture is, and it describes, rather than transcribes, the text in it (the OCR step reads that).
- **Moondream on this server** (`moondream`). M87 Labs' Moondream captions each picture on the worker, through
Hugging Face transformers and PyTorch: no picture leaves the machine and no provider is needed. Which one is
`llm.moondream_model`:
- `moondream3-preview`, Moondream 3 (Preview): 9B parameters (2B active), about 19 GB. It needs an NVIDIA GPU with
20 GB or more, or a Mac with 32 GB of memory: not a Raspberry Pi. Its weights and code are under the Business
Source License 1.1 (production use is allowed, except offering it to others in competition with M87 Labs' own),
which isn't an open-source licence, so Lens fetches it only when you pick it.
- `moondream2`, Moondream 2 (Apache-2.0): 1.9B parameters, about 3.7 GB. It runs on a CPU with about 10 GB of
memory, slowly: minutes a picture on four cores, against seconds for Florence-2.

PyTorch (its CPU build without an NVIDIA GPU), transformers and the weights are fetched the first time a job waits
to describe something ([Components](components.md)), the weights and the model's own code pinned to a revision;
where the chosen one can't run, nothing is fetched and the step says why. Moondream 3.1 runs only on M87 Labs'
Photon engine, whose kernels are licensed only under a separate agreement with them, so Lens doesn't offer it.

Each page or shot is a call to the model, so a long document costs as many: `llm.describe_max` caps how many of a
resource are described. Each call is in the [activity ledger](activity.md) (`model.chat` for the provider's model,
`model.describe` for Florence-2, with the tokens read and written and the time it took).
`model.describe` for Florence-2 and Moondream, with the time it took and, for Florence-2, the tokens read and written).

| Setting | Default | |
|---|---|---|
| `llm.describe_engine` | `llm` | what describes pages and shots: `llm` (the provider's model below) or `florence-2` (on this server) |
| `llm.describe_engine` | `llm` | what describes pages and shots: `llm` (the provider's model below), or on this server `florence-2` or `moondream` |
| `llm.vision_model` | none | the provider's model that describes pages and shots; none (with `llm`): the describe step is skipped |
| `llm.florence_model` | `base` | Florence-2's size: `base` or `large` |
| `llm.moondream_model` | `moondream3-preview` | which Moondream: `moondream3-preview` (a GPU) or `moondream2` (a CPU will do) |
| `llm.describe_max` | 50 | pages or shots of a resource described at most, 1–1000 (in order: the rest aren't) |

Agents read and change these with the MCP tools `picture_descriptions` and `set_picture_descriptions` (admins).
Expand Down
4 changes: 2 additions & 2 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,8 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `preview_template` | what a template in a drafted automation gives before it's saved, against an activity or the trigger kind's sample; a prompt shows outside text marked as the model sees it; an error names the missing variable and its nearest name |
| `file_storage` | where Lens keeps its own files (notes' images and attachments) and the connections it could use; `check` tests it. Admins only |
| `set_file_storage` | keep new files on this machine or a storage connection (folder, rclone crypt); checked before it's saved. Admins only, with a write-scope token |
| `picture_descriptions` | what describes the pages and shots of resources ([Descriptions](configuration.md#descriptions)): the LLM provider's model that can see images or Florence-2 on this server, its model, whether it's ready and why not, and the engines there are. Admins only |
| `set_picture_descriptions` | choose what describes pictures (`engine` `llm` or `florence-2`), the provider's `vision_model`, Florence-2's size (`florence_model`) and `describe_max`. Admins only, with a write-scope token |
| `picture_descriptions` | what describes the pages and shots of resources ([Descriptions](configuration.md#descriptions)): the LLM provider's model that can see images, or Florence-2 or Moondream on this server, its model, whether it's ready and why not, and the engines there are. Admins only |
| `set_picture_descriptions` | choose what describes pictures (`engine` `llm`, `florence-2` or `moondream`), the provider's `vision_model`, Florence-2's size (`florence_model`), which Moondream (`moondream_model`) and `describe_max`. Admins only, with a write-scope token |
| `install_settings` | whether other devices on the network reach Lens and its edition (full or lean), whether the machine's upgrade agent can change them, and the install line to run when it can't. Admins only |
| `set_install_settings` | open Lens to the network or keep it to this machine, or change the edition: the machine sets itself up again and Lens restarts ([Updates](updates.md#setting-the-machine-up-again)). Admins only, with a write-scope token |
| `propose_graph_change` | ask for two entities to be merged or linked; it waits in Proposed changes unless `apply` makes it at once (it can be undone). Needs a write-scope token and editor access |
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ request, to another check that proves the same thing; loosening a requirement or
| What is most like a passage, an asset, a topic or an entity | [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities`, `test_like_a_topic_and_an_entity`; [similar.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/similar.test.tsx) |
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |
| Or by Moondream on this server, off by default, fetched at a pinned revision only where the chosen model can run, and saying why where it can't | [test_moondream.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_moondream.py) `test_moondream_describes_pictures_without_a_model_provider`, `test_where_each_moondream_can_run`, `test_on_a_cpu_moondream_3_is_never_fetched_and_the_step_says_why`, `test_moondream_is_fetched_pinned`, `test_admins_and_agents_choose_moondream` |
| Transcripts cleaned up before analysis (stutters, "um"s, spacing), off by default, each change in the edit history with Revert and people's corrections left alone | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_off_by_default_changes_nothing`, `test_clean_step_tidies_lines_and_keeps_them_in_the_history`, `test_people_s_corrections_outrank_the_rules` |
| Punctuation put back in transcripts that lack it by a local model, fetched only when picked | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_punctuation_put_back_where_it_is_missing`, `test_without_the_model_the_rules_run_alone`, `test_the_model_is_fetched_only_when_it_is_asked_for` |
| Entities found by spaCy, fetched the first time it's picked, with the defined entities first and the rules until it's there | [test_spacy_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_spacy_entities.py) `test_defined_entities_come_first_then_spacy_s_names`, `test_the_analyze_step_and_workflows_use_it`, `test_without_spacy_the_rules_find_them`, `test_spacy_is_fetched_only_when_it_is_picked` |
Expand Down
Loading
Loading