Repository navigation
feat(describe): describe pictures on this machine with Florence-2 - #477
Merged
Merged
Conversation
Settings > LLM provider > Describing pictures can now have Microsoft's Florence-2 (MIT) describe each page and shot, instead of the provider's model that can see images (llm.describe_engine "florence-2"; the default stays "llm"). It runs the model's 8-bit ONNX export on ONNX Runtime on the CPU, so no picture leaves the machine and no provider is needed; the files are a component fetched on first use, pinned to a revision and checked against their SHA-256: base (about 280 MB) or large (about 830 MB), llm.florence_model. Generation follows Florence2ForConditionalGeneration: 768 x 768 picture, the <MORE_DETAILED_CAPTION> prompt, forced BOS, greedy with no repeated trigrams, and the merged decoder's attention cache. Each picture is a model.describe row in the activity ledger. Agents read and change it with the MCP tools picture_descriptions and set_picture_descriptions (admins). Docs, changelog, devlog and threat model updated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Since #395 the entity-types upgrade step types Terms such as Monday as Dates, which the entities list leaves out unless asked for, so the check read them as gone. It now lists the quiet types too: an entity may change type in an upgrade, but none may go. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g
…ir checks Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
…-ogtvpr # Conflicts: # docs/mcp.md # threat-model.yaml
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: the describe step could only send each page and shot to the LLM provider's model that can see images (
llm.vision_model). Without a provider, or without a vision model, pictures were never described.After: Settings → LLM provider → Describing pictures has a "Describe pictures with" choice. Picking Florence-2 on this server has Microsoft's Florence-2 (MIT) describe each page and shot in a paragraph on the worker's CPU, through ONNX Runtime. No picture leaves the machine, no provider is needed and nothing is billed. The default stays the provider's model.
How:
florence.pyruns onnx-community's 8-bit ONNX export the wayFlorence2ForConditionalGenerationdoes: a 768 x 768 normalised picture, the<MORE_DETAILED_CAPTION>prompt, decoding from EOS with BOS forced, greedy with no repeated trigrams, and the merged decoder's attention cache (encoder part kept from the first step). The files are a component (florence) fetched the first time a job waits at the describe step, pinned to a revision and checked against their SHA-256: base (about 280 MB) or large (about 830 MB),llm.florence_model. Theflorenceextra is ONNX Runtime and tokenizers. Each picture is amodel.describerow in the activity ledger with tokens read and written. Agents use the new MCP toolspicture_descriptionsandset_picture_descriptions(admins). The README lists it as built, with its checks indocs/requirements.md.Checked with the real models through Lens's own code: base takes about 3 s a picture on four CPU cores, large about 9 s, and both describe test photos plainly. An opt-in test (
LENS_TEST_FLORENCE_DIR) runs the real base model.The upgrade check's fix from #512 was ported here while it landed; it is on main now, so it no-ops.
Moondream (the other half of the ask) follows in a second PR as an opt-in engine, since Moondream 3's weights are BSL 1.1, not open source.
Motivation and Context
Adeel asked to use Moondream 3 and/or Microsoft Florence for visual descriptions. Florence-2 is MIT, runs on a Raspberry Pi's CPU without PyTorch, and needs no provider.
Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md.Done questions (AGENTS.md)
picture_descriptionsandset_picture_descriptions; also the settings API.llm.describe_engineandllm.florence_modelin Settings with defaults; nothing in the wizard.model.describeledger row.make openapigave no changes); the README lists it as built, with its checks indocs/requirements.md.🤖 Generated with Claude Code
https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW