Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Entities found by spaCy, fetched when picked.** Settings → Analysis → Entity extraction → spaCy now fetches spaCy
(MIT) and its small model the first time an asset needs it (about 60 MB): English, German, or one for many
languages. The defined entities still come first, a lone word like "Okay" isn't taken for a name, and until spaCy is
there the rules find them and the job log says so. Workflows' Extract entities (rules) node uses it too
([Entities and the graph explorer](docs/processing.md#entities-and-the-graph-explorer)).
- **Punctuation put back in transcripts.** With transcripts cleaned up, Settings → Analysis → Put punctuation back
adds full stops, commas and question marks with a local model (kredor/punctuate-all, MIT, 12 languages), to
transcripts that have hardly any (imported captions) or to every one, keeping the marks they have. The model and
Expand Down
10 changes: 10 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

Plans and progress for work in flight. Newest first.

## 2026-10-17 · Entities found by spaCy

Adeel: NLP libraries for named entities. Settings → Analysis → Entity extraction → spaCy existed but needed spaCy
installed by hand; it's now a component fetched the first time an asset needs it (the `spacy` extra and a pinned,
checksummed model wheel: English, German or the multi-language one, all MIT). The defined entities come first, spaCy's
names fill in around them, and the rules stand in until spaCy is there. Workflows' rules node uses the same extractor.

Refine later: pick the model by the transcript's language; spaCy's medium and large English models (MIT too, 40 to
560 MB) as options; run spaCy over a recording's lines in one batch.

## 2026-10-17 · Cleaning up transcripts before analysis

Adeel: a text cleaning step before entity extraction and transcript processing, from a punctuation and grammar
Expand Down
3 changes: 2 additions & 1 deletion docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ never fetch a speech or voice model.
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| spaCy | `analysis.entities` is `spacy` ([Entities](processing.md#entities-and-the-graph-explorer)) | spaCy (MIT) and the model in `analysis.spacy_model`: `en_core_web_sm`, `de_core_news_sm` or `xx_ent_wiki_sm` (MIT, 11 to 15 MB each), about 60 MB in all |
| DuckDuckGo search | `web.search` is `duckduckgo` | ddgs (MIT, about 30 MB with lxml and primp), the client the assistant's [web search](assistant.md#web-search-and-reading-pages) reads DuckDuckGo's results with |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
Expand All @@ -31,7 +32,7 @@ never fetch a speech or voice model.
- Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win.
- Models go into `data_dir/models`: the Hugging Face, ModelScope and PyTorch caches point there (unless `HF_HOME`,
`MODELSCOPE_CACHE` or `TORCH_HOME` say otherwise). They outlast container rebuilds and upgrades.
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model) are checked against their SHA-256.
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model, spaCy's models) are checked against their SHA-256.
- A job step that needs something still being fetched waits for it instead of falling back or failing. If a fetch fails,
the step runs as before (for transcription, falling back to another engine), and the fetch is tried again an hour
later, when the settings change, or on **Check again now**.
Expand Down
12 changes: 11 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,16 @@ Nodes are people and named things. Edges are:
People, organisations, products, places, events, works and topics are extracted from every asset's text. Dates and
numbers are extracted too, but hidden unless asked for.

**How they're found.** By rules out of the box: capitalised names, titles, organisation endings, dates and numbers,
after the entities a namespace defines. With `analysis.entities` set to `spacy` (Settings → Analysis → Entity
extraction), spaCy's small language model finds them instead, after the defined entities, which keep their names and
types where the two overlap; a lone word like "Okay" isn't taken for a name. Lens fetches spaCy (MIT) and the model
named in `analysis.spacy_model` the first time an asset needs them ([Components](components.md)): `en_core_web_sm`
(English, the default), `de_core_news_sm` (German) or `xx_ent_wiki_sm` (many languages; people, organisations and
places). spaCy's other languages' models carry GPL, LGPL or Creative Commons licenses, so Lens doesn't fetch them; one
installed by hand works. Until spaCy is there the rules find the entities and the job log says so. The workflows'
Extract entities (rules) node uses the same extractor.

The **Entities** page in the web app lists a namespace's entities (search, type and collection filters, a Hidden tab).
An entity's name opens its own page, `/wiki/<namespace>/<Name>`, with its notes, every line that says it and what it
comes up with ([Entity pages](notes.md#entity-pages)); the rest of its row opens a side panel where editors rename,
Expand Down Expand Up @@ -520,7 +530,7 @@ A pipeline's steps make derived files (a transcript or a page's text, shots, tex
workflow is a versioned graph drawn on a canvas (Pipelines → Workflows): it starts from the Asset node, and each
node passes what it makes along its connections.

- **Entities:** Extract entities (rules) runs the built-in extractor analyze uses, plus your terms (`Name|TYPE`) and
- **Entities:** Extract entities (rules) runs the extractor analyze uses (the rules, or spaCy), plus your terms (`Name|TYPE`) and
regular expressions (`TYPE: pattern`); Extract entities (LLM) asks the model for structured entities and keeps,
corrects or adds to any passed in; Merge joins lists; Save entities makes them the asset's entities (people's
corrections kept) and redoes keywords and chapters.
Expand Down
61 changes: 54 additions & 7 deletions fastapi_backend/app/domain/analyze.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ def initial(k):
_NLP = {}
SPACY_MAP = {
"PERSON": "PERSON",
"PER": "PERSON",
"ORG": "ORG",
"PRODUCT": "PRODUCT",
"GPE": "PLACE",
Expand All @@ -214,15 +215,57 @@ def initial(k):
"QUANTITY": "NUMBER",
"NORP": "TERM",
"LANGUAGE": "TERM",
"MISC": "TERM",
}
SPACY_SKIP = ["parser", "lemmatizer", "tagger", "morphologizer", "attribute_ruler", "senter"] # names need only the ner


def spacy_entities(text, model):
def spacy_nlp(model):
"""spaCy's pipeline for `model`, only what finds names (and sentence ends, so a name doesn't run across two); None
while spaCy or the model isn't installed here (components.py fetches them the first time they're needed)."""
if model not in _NLP:
import spacy
try:
import spacy

nlp = spacy.load(model, exclude=SPACY_SKIP)
except (ImportError, OSError):
return None
if "sentencizer" not in nlp.pipe_names:
nlp.add_pipe("sentencizer", first=True)
_NLP[model] = nlp
return _NLP[model]


def spacy_entities(text, nlp, gaz=()):
"""What spaCy's model finds, after the defined entities (gaz), which keep their names and types where they
overlap; a lone word spaCy takes for a name ("Okay", "Yeah") is dropped."""
out, taken = [], []
for _, typ, rx in gaz:
for m in rx.finditer(text):
if not any(m.start() < b and m.end() > a for a, b in taken):
taken.append((m.start(), m.end()))
out.append((m.start(), m.group(0), typ))
for e in nlp(text).ents:
typ, name = SPACY_MAP.get(e.label_), e.text.strip()
if not typ or not name or (len(e) == 1 and _base(name) in STOPCAP):
continue
if not any(e.start_char < b and e.end_char > a for a, b in taken):
taken.append((e.start_char, e.end_char))
out.append((e.start_char, name, typ))
out.sort()
return [(t, typ) for _, t, typ in out]


_NLP[model] = spacy.load(model)
return [(e.text, SPACY_MAP[e.label_]) for e in _NLP[model](text).ents if e.label_ in SPACY_MAP]
def entity_finder(cfg, gaz=()):
"""The extractor Settings → Analysis picks, as a function of a line's text: the rules, or spaCy. While spaCy or its
model isn't installed here the rules find them, and `finder.fell_back` says so."""
nlp = spacy_nlp(cfg["analysis"].get("spacy_model") or "en_core_web_sm") if cfg["analysis"]["entities"] == "spacy" else None

def finder(text):
return spacy_entities(text, nlp, gaz) if nlp else extract_entities(text, gaz)

finder.fell_back = cfg["analysis"]["entities"] == "spacy" and nlp is None
return finder


# ---------- sections (TextTiling) ----------
Expand Down Expand Up @@ -383,7 +426,7 @@ def _place(db, cfg, nid, rec, setup, listed, first, segs, ch):
return known


def analyze_recording(db, cfg, rid, seg_ents=None):
def analyze_recording(db, cfg, rid, seg_ents=None, say=None):
"""Entities, keywords, sections and talk statistics. `seg_ents` (per line, [(name, type)]) replaces the extractor:
a workflow's save entities node passes what its nodes found."""
rec = db.one("SELECT space, collection FROM $r", r=R("recording", rid))
Expand All @@ -396,13 +439,17 @@ def analyze_recording(db, cfg, rid, seg_ents=None):
)
extra = entity_map.terms(db, nid, entity_map.chain_of(db, rec.get("collection"))) if listed else []
gaz = parse_gazetteer(list(cfg["analysis"].get("gazetteer") or []) + extra) # the defined entities, however written
use_spacy = cfg["analysis"]["entities"] == "spacy"
find = entity_finder(cfg, gaz) if seg_ents is None else None
if find and find.fell_back and say:
say(
f"spaCy or its model {cfg['analysis'].get('spacy_model')} isn't installed here (Settings → Components): the rules found the entities"
)
given, seg_ents, toks, wc, tn, ts = seg_ents, [], [], [], Counter(), defaultdict(Counter)
for k, s in enumerate(segs):
if given is not None:
seg_ents.append(list(given[k]) if k < len(given) else [])
else:
seg_ents.append(spacy_entities(s["text"], cfg["analysis"]["spacy_model"]) if use_spacy else extract_entities(s["text"], gaz))
seg_ents.append(find(s["text"]))
ws = words(s["text"])
toks.append([w for w, _ in ws])
wc.append(len(s["text"].split()))
Expand Down
9 changes: 8 additions & 1 deletion fastapi_backend/app/domain/automations.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,16 @@
workflows,
)
from . import activity as activity_mod
from .automation_nodes import provided

R = store.R


def provided(module, name):
"""automation_nodes.provided, looked up when it's called: automation_nodes imports this module (through actions),
so its names aren't there yet while either is being imported."""
return automation_nodes.provided(module, name)


FIELDS = (
"record::id(id) AS id, name, description, enabled, enabled_at, spaces, run_as, triggers, join, condition, workflow, "
"version, policy, limits, created_by, created_at, updated_at, last_run_at, last_status, last_run, counters, removed_at"
Expand Down
70 changes: 70 additions & 0 deletions fastapi_backend/app/domain/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,24 @@
"0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79",
39,
),
# spaCy's MIT-licensed small models (analysis.spacy_model), as the wheels spaCy publishes for 3.8; installed with
# no dependencies into the packages folder. Its other languages' models carry GPL, LGPL or Creative Commons
# licenses, so they're for people to install themselves.
"en_core_web_sm-3.8.0-py3-none-any.whl": (
"https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl",
"1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85",
13,
),
"de_core_news_sm-3.8.0-py3-none-any.whl": (
"https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.8.0/de_core_news_sm-3.8.0-py3-none-any.whl",
"fec69fec52b1780f2d269d5af7582a5e28028738bd3190532459aeb473bfa3e7",
15,
),
"xx_ent_wiki_sm-3.8.0-py3-none-any.whl": (
"https://github.com/explosion/spacy-models/releases/download/xx_ent_wiki_sm-3.8.0/xx_ent_wiki_sm-3.8.0-py3-none-any.whl",
"6f3c4b853852ea9e9d2dc76cc950dddb10a7e4c42d813308caefe6c5e8be2f0a",
11,
),
# AdamCodd/vit-base-nsfw-detector (Apache-2.0), its author's 8-bit ONNX export, at a fixed revision (nsfw.py)
"vit-base-nsfw-detector-int8.onnx": (
"https://huggingface.co/AdamCodd/vit-base-nsfw-detector/resolve/8587de998f441aac03fdd57a85d2e4cb808c7d64/onnx/model_quantized.onnx",
Expand Down Expand Up @@ -496,6 +514,50 @@ def fetch(self, cfg, m, say):
fetch_file(cfg, n, say)


SPACY_MODELS = {"en_core_web_sm": "en_core_web_sm-3.8.0-py3-none-any.whl", "de_core_news_sm": "de_core_news_sm-3.8.0-py3-none-any.whl", "xx_ent_wiki_sm": "xx_ent_wiki_sm-3.8.0-py3-none-any.whl"} # fmt: skip


class Spacy(Component):
"""spaCy and its model (analysis.spacy_model), when Settings → Analysis finds entities with spaCy."""

kind = "model"

def _model(self, cfg):
return (cfg.get("analysis") or {}).get("spacy_model") or "en_core_web_sm"

def needed(self, cfg, m):
return (cfg.get("analysis") or {}).get("entities") == "spacy"

def present(self, cfg):
return importable("spacy", self._model(cfg))

def fetch(self, cfg, m, say):
if not importable("spacy"):
pip_install(cfg, "spacy", say)
model = self._model(cfg)
if importable(model):
return
if model not in SPACY_MODELS:
raise Unavailable(f"{model} isn't one Lens fetches ({', '.join(SPACY_MODELS)}); install it with pip")
wheel = fetch_file(cfg, SPACY_MODELS[model], say)
if say:
say(f"installing the spaCy model {model}")
target = packages_dir(cfg)
target.mkdir(parents=True, exist_ok=True)
uv = shutil.which("uv")
if not uv:
raise Unavailable(f"needs uv to install {model}; install it yourself with: pip install {wheel}")
_run(
[uv, "pip", "install", "--python", sys.executable, "--target", str(target), "--no-deps", "--quiet", wheel],
env={**os.environ, "UV_NO_CACHE": "1"},
)
pathlib.Path(wheel).unlink(missing_ok=True) # installed: the wheel would only take up the disk
activate(cfg)

def detail(self, cfg):
return self._model(cfg)


class Punctuation(Component):
"""PyTorch, Transformers and the punctuation model (cleanup.py), when Settings → Analysis cleans up transcripts and
restores their punctuation."""
Expand Down Expand Up @@ -801,6 +863,14 @@ def detail(self, cfg):
license="Apache-2.0",
),
LAYA,
Spacy(
"spacy",
"spaCy",
"finds people, organisations, places, dates and more in text with a small language model (analysis.entities spacy)",
steps={"analyze", "workflow"},
size_mb=60,
license="MIT",
),
Punctuation(
"punctuation",
"Punctuation model",
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ def _clean(db, cfg, rid, say, spec=None):


def _analyze(db, cfg, rid, say, spec=None):
analyze.analyze_recording(db, cfg, rid)
analyze.analyze_recording(db, cfg, rid, say=say)
say("analysed")


Expand Down
8 changes: 3 additions & 5 deletions fastapi_backend/app/domain/workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,19 +237,17 @@ def extract_rules(db, cfg, rid, c):
list(c.get("terms") or []) + (list(cfg["analysis"].get("gazetteer") or []) if c.get("builtin", True) else [])
)
pats = [(re.compile(p["pattern"]), p["type"]) for p in c.get("patterns") or []]
spacy = c.get("builtin", True) and cfg["analysis"]["entities"] == "spacy"
find = analyze.entity_finder(cfg, gaz) if c.get("builtin", True) else None
types = set(c.get("types") or [])
out = []
for k, s in enumerate(_segments(db, rid)):
text, found = s["text"] or "", []
for rx, typ in pats:
found += [(m.group(0), typ) for m in rx.finditer(text) if m.group(0).strip()]
if c.get("builtin", True) and not spacy:
found += analyze.extract_entities(text, gaz) # as analyze finds them, your terms included
if find:
found += find(text) # as analyze finds them (the rules, or spaCy), your terms included
else:
found += [(m.group(0), typ) for _, typ, rx in gaz for m in rx.finditer(text)]
if spacy:
found += analyze.spacy_entities(text, cfg["analysis"]["spacy_model"])
out += [{"name": n.strip(), "type": t, "seg": k} for n, t in found if not types or t in types]
return merge([out])

Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ insightface = ["insightface", "onnxruntime"]
objects = ["onnxruntime"]
# Punctuation put back in transcripts (analysis.cleanup_punctuation): kredor/punctuate-all on Transformers and PyTorch
punctuation = ["transformers", "torch"]
# Named entities found by spaCy (analysis.entities "spacy"); its MIT-licensed models are fetched as pinned wheels
spacy = ["spacy>=3.8,<3.9"]
# A local prompt-injection classifier (ai.injection_check "classifier"): ProtectAI's DeBERTa model on ONNX Runtime
injection = ["onnxruntime", "tokenizers"]
# Checking IIIF manifests against the official schema
Expand Down
Loading
Loading