Skip to content

feat(describe): describe pictures on this machine with Florence-2 - #477

Merged
adeelahmad merged 12 commits into
mainfrom
claude/project-thread-ogtvpr
Oct 17, 2026
Merged

adeelahmad merged 12 commits into
mainfrom
claude/project-thread-ogtvpr

Conversation

@adeelahmad

@adeelahmad adeelahmad commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: the describe step could only send each page and shot to the LLM provider's model that can see images (llm.vision_model). Without a provider, or without a vision model, pictures were never described.

After: Settings → LLM provider → Describing pictures has a "Describe pictures with" choice. Picking Florence-2 on this server has Microsoft's Florence-2 (MIT) describe each page and shot in a paragraph on the worker's CPU, through ONNX Runtime. No picture leaves the machine, no provider is needed and nothing is billed. The default stays the provider's model.

How: florence.py runs onnx-community's 8-bit ONNX export the way Florence2ForConditionalGeneration does: a 768 x 768 normalised picture, the <MORE_DETAILED_CAPTION> prompt, decoding from EOS with BOS forced, greedy with no repeated trigrams, and the merged decoder's attention cache (encoder part kept from the first step). The files are a component (florence) fetched the first time a job waits at the describe step, pinned to a revision and checked against their SHA-256: base (about 280 MB) or large (about 830 MB), llm.florence_model. The florence extra is ONNX Runtime and tokenizers. Each picture is a model.describe row in the activity ledger with tokens read and written. Agents use the new MCP tools picture_descriptions and set_picture_descriptions (admins). The README lists it as built, with its checks in docs/requirements.md.

Checked with the real models through Lens's own code: base takes about 3 s a picture on four CPU cores, large about 9 s, and both describe test photos plainly. An opt-in test (LENS_TEST_FLORENCE_DIR) runs the real base model.

The upgrade check's fix from #512 was ported here while it landed; it is on main now, so it no-ops.

Moondream (the other half of the ask) follows in a second PR as an opt-in engine, since Moondream 3's weights are BSL 1.1, not open source.

Motivation and Context

Adeel asked to use Moondream 3 and/or Microsoft Florence for visual descriptions. Florence-2 is MIT, runs on a Raspberry Pi's CPU without PyTorch, and needs no provider.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: Florence-2 describes pictures on this machine; merged on green CI (local CI run before pushing).
  • Agents: MCP tools picture_descriptions and set_picture_descriptions; also the settings API.
  • Zero setup: the model and ONNX Runtime are fetched on first use; nothing to install or edit.
  • Fewest clicks: one select in Settings; the size defaults to base.
  • Configurable: llm.describe_engine and llm.florence_model in Settings with defaults; nothing in the wizard.
  • Light: off by default; base is about 280 MB on ONNX Runtime, no PyTorch, and runs on a Pi.
  • Private and secure: pictures stay on the machine; threat model row added (pinned, hashed model files).
  • Cost: no provider call; each picture is a model.describe ledger row.
  • User decides: the default engine stays the provider's model; Florence-2 only when an admin picks it.
  • Upgrades: new settings keys with defaults; no stored-data change.
  • Honest: docs give measured times and say what the model does and doesn't do (it reads no instructions).
  • Tests, docs and generated files updated (make openapi gave no changes); the README lists it as built, with its checks in docs/requirements.md.
  • Open standards; Florence-2 and the export are MIT, ONNX Runtime and tokenizers permissive.
  • Result reported with evidence; "refine later" items in DEVLOG.md.
  • Reuse: the describe step, components (fetch on first use, pinned and hashed), the activity ledger and the MCP tools are Lens's own; ONNX Runtime and tokenizers run the model, and only the short decoding loop is new, since the ONNX export comes without one.

🤖 Generated with Claude Code

https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW

Settings > LLM provider > Describing pictures can now have Microsoft's
Florence-2 (MIT) describe each page and shot, instead of the provider's
model that can see images (llm.describe_engine "florence-2"; the default
stays "llm"). It runs the model's 8-bit ONNX export on ONNX Runtime on the
CPU, so no picture leaves the machine and no provider is needed; the files
are a component fetched on first use, pinned to a revision and checked
against their SHA-256: base (about 280 MB) or large (about 830 MB),
llm.florence_model. Generation follows Florence2ForConditionalGeneration:
768 x 768 picture, the <MORE_DETAILED_CAPTION> prompt, forced BOS, greedy
with no repeated trigrams, and the merged decoder's attention cache.

Each picture is a model.describe row in the activity ledger. Agents read
and change it with the MCP tools picture_descriptions and
set_picture_descriptions (admins). Docs, changelog, devlog and threat
model updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
@adeelahmad adeelahmad self-assigned this Oct 17, 2026
claude added 11 commits October 17, 2026 02:24
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Since #395 the entity-types upgrade step types Terms such as Monday as
Dates, which the entities list leaves out unless asked for, so the check
read them as gone. It now lists the quiet types too: an entity may change
type in an upgrade, but none may go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013sC1p4dJXjteytbxDXnk5g
…ir checks

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
…-ogtvpr

# Conflicts:
#	docs/mcp.md
#	threat-model.yaml
@adeelahmad
adeelahmad merged commit 4c29d68 into main Oct 17, 2026
21 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-ogtvpr branch October 17, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants