Skip to content

feat(analysis): find entities with spaCy, fetched the first time it's picked - #492

Merged
adeelahmad merged 2 commits into
mainfrom
claude/project-thread-hyaa6h
Oct 17, 2026
Merged

adeelahmad merged 2 commits into
mainfrom
claude/project-thread-hyaa6h

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: Settings → Analysis → Entity extraction → spaCy only worked if someone had installed spaCy and a model by hand; otherwise analysis quietly used the rules. Importing app.domain.workflows on its own also failed with a circular import.

After: picking spaCy fetches it and its small model the first time an asset needs it (about 60 MB): English (en_core_web_sm), German (de_core_news_sm) or one for many languages (xx_ent_wiki_sm), all MIT. The defined entities still come first and keep their names and types where spaCy overlaps them; a lone word like "Okay" isn't taken for a name; German and multi-language labels (PER, MISC) map to people and terms. Until spaCy is there the rules find the entities and the job's log says so. Workflows' Extract entities (rules) node uses the same extractor.

How: a Spacy component installs the spacy extra (versions from uv.lock), then the model wheel pinned in components.FILES and checked against its SHA-256, with --no-deps. analyze.entity_finder loads only spaCy's tokenizer, a sentencizer and its NER. The first commit makes automations.provided a call-time lookup, which breaks the import cycle.

Motivation and Context

Adeel asked for "nlp libraries for extracting NER" in the cleanup series (#476 cleanup, #485 punctuation). Dates and grammar follow.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: NER with an NLP library, used by analysis and workflows; merged on green CI.
  • Agents: PUT /api/v1/settings/analysis (entities, spacy_model), the setup assistant's change_settings, the components API; entities reach agents through the existing graph, MCP and assistant tools.
  • Zero setup: fetched on first use.
  • Fewest clicks: the existing Entity extraction select; nothing new to fill.
  • Configurable: Settings → Analysis (entities, spaCy model), defaults unchanged (rules, en_core_web_sm).
  • Light: rules stay the default; spaCy is fetched only when picked.
  • Private and secure: runs locally; model wheels pinned by SHA-256 and installed without dependencies (threat-model row added).
  • Cost: local, no model calls.
  • User decides: off unless picked; defined entities win over spaCy's.
  • Upgrades: no stored-data change.
  • Honest: docs say what spaCy finds and what happens without it.
  • Tests in tests/domain/test_spacy_entities.py (with a fake pipeline, and the real en_core_web_sm when installed: passed locally with it); docs/processing.md, docs/components.md, CHANGELOG, DEVLOG updated; OpenAPI unchanged.
  • Open: spaCy and its models are MIT.
  • Reported in the project thread; refine later: medium and large models, more languages, spaCy's own dates feeding the date reader.
  • Reuse: spaCy and its published models; Lens's components fetcher, pip_install and fetch_file checks.

Local CI (ci/local-ci.sh): backend (embedded and server SurrealDB), Next.js, pre-commit and title passed.


Generated by Claude Code

automations took automation_nodes.provided at import time, but
automation_nodes imports automations (through actions), so importing
app.domain.workflows first failed with "cannot import name 'provided' from
partially initialized module". provided is now looked up when it's called.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
… picked

Settings > Analysis > Entity extraction > spaCy needed spaCy and a model
installed by hand. spaCy is now a component: the `spacy` extra at uv.lock's
versions and the model named in analysis.spacy_model as a pinned wheel
checked against its SHA-256 (en_core_web_sm, de_core_news_sm or
xx_ent_wiki_sm, all MIT), fetched the first time an asset needs it.

In spaCy mode the defined entities come first and keep their names and types
where they overlap spaCy's; a lone word like "Okay" isn't taken for a name;
PER and MISC (the German and multi-language models) map to people and terms.
Only spaCy's tokenizer, a sentencizer and its ner run. Until spaCy is there
the rules find the entities and the job log says so. The workflows' rules
node uses the same extractor.

Docs, changelog, dev log and threat model updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014z3r72SB2quWGKbgGt46mZ
@adeelahmad adeelahmad self-assigned this Oct 17, 2026
@adeelahmad
adeelahmad merged commit b3d45c3 into main Oct 17, 2026
18 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-hyaa6h branch October 17, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants