Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
stream could wait for a database connection its own thread already held, and every request after it waited too.
A query made inside another on the same thread now runs on that connection
([Concurrency](docs/database.md#concurrency)).
- **The decision model types the names the rules can't.** When Jev or Laya is set up, analysis asks it what each new
Term of a recording is (a person, an organisation, a product...), with the lines it's said in, in one call per
recording, and keeps the types it is sure of. Settings › Analysis › Names the rules can't type (`analysis.typing`)
can also let the language model answer, or turn it off; the language model is never asked by default
([Names the rules can't type](docs/processing.md#entities-and-the-graph-explorer)).

## 0.14.0 <small>October 17, 2026</small> {id="0.14.0"}

Expand Down
8 changes: 6 additions & 2 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,12 @@ organisations and products, holidays), the words a name starts or ends with, rol
verbs of saying after it and the namespace's named speakers; a recording gives each name one type. Upgrade step
`entity-types` retypes old analysis-made Terms, skipping types people or agents set (graph history and audit log).

Refine later: the decision model (Jev or Laya) for the names the rules leave as Terms, in analysis and as an action
on existing Terms (next PR); place, name and brand lists for more languages and regions (Custom vocabulary in
Then the decision model: `analysis.typing` (`auto` by default: Jev or Laya when set up, never the language model;
`model` lets the language model answer; `rules` none) asks about a recording's new Terms in one `decide.choose_all`
call, with two lines each, and keeps answers above `decisions.act_above`.

Refine later: an action that asks the model about existing Terms (API, MCP, a button on the Entities page; next PR);
offer the namespace's own types to the model too; ask about names in the hybrid mode's new entities; place, name and brand lists for more languages and regions (Custom vocabulary in
Settings covers one install's own names); type existing entities again when a later recording gives better evidence
(needs a marker for types people set, kept out of the versioned rows); "Google Cloud"-like names (a known company,
then a word) as products.
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,12 +120,12 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
rules, without a model. Find what is most like a passage, an asset, a topic or an entity, in a Similar
tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show. Pages and shots are
described in words by the model provider's vision model or, off by default, by Florence-2 or Moondream on your own
server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation
put back), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each
off by default.
rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an
asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos
by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by
Florence-2 or Moondream on your own server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing
and, with a local model, punctuation put back), entities found by spaCy instead of the rules, and the dates people say
read as days on the calendar, each off by default.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
Expand Down
2 changes: 2 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -804,6 +804,8 @@ The assistant makes routine choices for you instead of asking. The first is wher
`import_files` without a namespace picks one from the namespaces you can add to, judging from their descriptions,
what's in them lately, the file names and your message. You're asked only when it isn't sure: below
`decisions.act_above` (0.8) confidence it lists the namespaces, its best guess first.
Analysis takes routine choices too: what a name the rules can't type is (a person, an organisation, a product...;
[Names the rules can't type](processing.md#entities-and-the-graph-explorer)).

A **decision model** answers these: Jev, typesafe.ai's System One, which takes a few hundred milliseconds and costs a
fraction of an LLM call. Set its key in **Settings → AI assistant → Routine choices**, or `TYPESAFE_API_KEY` in `.env`.
Expand Down
15 changes: 15 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,21 @@ reads each name and the words around it (`app/domain/name_types.py`):
topic, or retyped on the Entities page (one at a time or ticked together). Names listed in Settings › Analysis ›
Custom vocabulary (`Name | TYPE`) always get the type given there.

**Names the rules can't type.** When a [decision model](assistant.md#routine-choices) is set up (Jev with its key, or
Laya), analysis asks it what each new Term of a recording is, with the first two lines it's said in, and keeps the type
when it is sure (`decisions.act_above`, 0.8). Names that are already entities aren't asked again, and the short form of
a longer name in the same recording ("Johnson" of "Sarah Johnson") takes that name's type. The names of a recording go
in one call (up to 20 a call and 40 a recording; Laya, which reads less, one at a time). Each call is in Activity and
counts toward budgets, and none is made once the budget's hard limit is reached. Settings › Analysis › **Names the rules
can't type** (`analysis.typing`) chooses:

- `auto` (the default): the decision model when one is set up, else nothing; the language model is never asked.
- `model`: the decision model, else the language model, in one call per recording too.
- `rules`: no model; the names stay Terms.

A name the model isn't sure of stays a Term, and so do all of them when it can't be reached. Names a workflow's nodes
typed, and names mapped onto a fixed list ([Entity setup](#entity-setup)), aren't asked.

A name gets one type per recording, the one most of its lines give it, so "Lee" after "Dr Lee" is a person too. "And"
no longer joins two names into one ("Kenya and Nigeria" are two places). The given names are mostly the US Census
Bureau's frequent first names of 1990 (public domain); the other lists are written for Lens. With
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Every entity's own page at `/wiki/<namespace>/<Name>` | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_entities.py) `test_wiki_addresses`; [wiki-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/wiki-page.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-find` |
| Entities, topics and people edited and merged from their lists, the graph or an asset, with Undo instead of a confirm | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_entities.py) `test_suggestions_and_merge_undo`; [entities-curate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/entities-curate.test.tsx); [topics-curate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/topics-curate.test.tsx); [recording-entities-edit.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-entities-edit.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-merge`, `graph-merge`, `speaker-rename` |
| Names get their real types (people, places, organisations, events, works) by rules, without a model; older Terms are typed on upgrade | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_names_get_the_type_their_words_give`, `test_analysis_types_people_places_and_events`, `test_the_upgrade_types_the_old_terms_and_keeps_what_people_chose` |
| A decision model (Jev or Laya) types the names the rules leave as Terms, in one call, keeping only types it is sure of; Settings › Analysis can leave them Terms | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_the_decision_model_types_what_the_rules_cant`, `test_a_type_the_model_isnt_sure_of_stays_a_term`, `test_by_default_the_language_model_isnt_asked`, `test_with_typing_rules_nothing_is_asked` |
| What is most like a passage, an asset, a topic or an entity | [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities`, `test_like_a_topic_and_an_entity`; [similar.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/similar.test.tsx) |
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |
Expand Down
28 changes: 28 additions & 0 deletions fastapi_backend/app/domain/analyze.py
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,32 @@ def one_type(seg_ents):
return [[(n, best.get(ent_key(n), t)) for n, t in es] for es in seg_ents]


def model_types(db, cfg, nid, segs, seg_ents, gaz=()):
"""The terms of a recording that aren't entities yet, given the type a model is sure of (name_types.ask). A short
form of a longer name said here ("Johnson" of "Sarah Johnson") takes that name's type later, and a name the custom
vocabulary (`gaz`) lists is the type it says, a term too; neither is asked."""
if not name_types.asking(cfg):
return seg_ents
listed = {ent_key(g[0]) for g in gaz}
terms = {}
for es in seg_ents:
for n, t in es:
k = ent_key(n)
if k and t == "TERM" and k not in listed:
terms.setdefault(k, n)
keys = {ent_key(n) for es in seg_ents for n, _ in es}
terms = {k: n for k, n in terms.items() if not any(L.endswith(" " + k) for L in keys)}
if terms:
have = set(db.values("SELECT VALUE key FROM entity WHERE space = $s AND key IN $k", s=nid, k=sorted(terms)))
have |= set(db.values("SELECT VALUE key FROM entity_alias WHERE space = $s AND key IN $k", s=nid, k=sorted(terms)))
terms = {k: n for k, n in terms.items() if k not in have}
if not terms:
return seg_ents
texts = [s["text"] for s in segs]
typed = name_types.ask(cfg, {k: (n, name_types.said_in(n, texts)) for k, n in terms.items()})
return [[(n, typed.get(ent_key(n), t) if t == "TERM" else t) for n, t in es] for es in seg_ents] if typed else seg_ents


def analyze_recording(db, cfg, rid, seg_ents=None, say=None):
"""Entities, keywords, sections and talk statistics. `seg_ents` (per line, [(name, type)]) replaces the extractor:
a workflow's save entities node passes what its nodes found."""
Expand Down Expand Up @@ -486,6 +512,8 @@ def analyze_recording(db, cfg, rid, seg_ents=None, say=None):
have = {ent_key(n) for n, _ in es}
es.extend(h for h in hints if ent_key(h[0]) not in have)
seg_ents = one_type(seg_ents)
if given is None and not listed: # names the rules can't tell: a model may (Settings › Analysis)
seg_ents = model_types(db, cfg, nid, segs, seg_ents, gaz)
if setup["types"] and not listed: # the namespace (or collection) keeps only some types
seg_ents = [[(n, t) for n, t in es if entity_setup.keeps(setup, t)] for es in seg_ents]
surface = {t: c.most_common(1)[0][0] for t, c in ts.items()}
Expand Down
63 changes: 63 additions & 0 deletions fastapi_backend/app/domain/decide.py
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,69 @@ def choose(cfg, question, options, state):
}


def choose_all(cfg, asks, state, language_model=True):
"""Several choices about one `state` in one call: `asks` is {id: (question, options)}; returns {id: decision} as
`choose` gives them, for the questions answered. The language model answers when no decision model can, unless
`language_model` is False. Raises Undecided when nothing can answer."""
asks = {q: (text, dict(o)) for q, (text, o) in asks.items() if o}
if not asks:
return {}
ids = {f"q{i}": q for i, q in enumerate(asks)} # plain names for the questions, whatever the caller's ids
how = engine(cfg)
if how in ENGINES:
questions = {i: {"type": "choice", "instructions": asks[q][0], "criteria": asks[q][1]} for i, q in ids.items()}
try:
answers = ENGINES[how](cfg, questions, state)
out = {}
for i, q in ids.items():
a, o = answers.get(i) or {}, asks[q][1]
if a.get("choice") in o:
probs = a.get("probabilities") or {a["choice"]: a.get("confidence", 0)}
out[q] = {"choice": a["choice"], "confidence": float(a.get("confidence") or 0), "ranked": _ranked(probs), "by": how}
return out
except Undecided as e:
if not language_model:
raise
log.warning("decision model (%s) failed, asking the language model: %s", how, e)
how = "llm" if llm.configured(cfg) else None
if how != "llm" or not language_model:
raise Undecided("no decision model is set up" if not language_model else "no decision model or language model is set up")
one = {
i: {
"type": "object",
"properties": {"choice": {"type": "string", "enum": list(asks[q][1])}, "confidence": {"type": "number"}},
"required": ["choice", "confidence"],
}
for i, q in ids.items()
}
schema = {"type": "object", "properties": {"answers": {"type": "object", "properties": one}}, "required": ["answers"]}
listed = "\n\n".join(f"{i}: {asks[q][0]}\n" + "\n".join(f"- {k}: {v}" for k, v in asks[q][1].items()) for i, q in ids.items())
try:
reply = llm.json_out(
cfg,
"You take routine decisions. For each question pick the one option that fits, by its name, and say how sure "
"you are from 0 to 1: below 0.8 when the evidence is thin or two options fit.",
f"Questions:\n\n{listed}\n\nState:\n{_state(state)}",
schema,
)
except llm.LLMError as e:
raise Undecided(str(e)) from None
out = {}
for i, q in ids.items():
a, options = (reply.get("answers") or {}).get(i) or {}, asks[q][1]
if a.get("choice") not in options:
continue
conf = min(1.0, max(0.0, float(a.get("confidence") or 0)))
rest = (1 - conf) / max(1, len(options) - 1)
out[q] = {
"choice": a["choice"],
"confidence": conf,
"ranked": _ranked({k: conf if k == a["choice"] else rest for k in options}),
"by": "llm",
}
return out


def sure(cfg, decision):
"""Whether a decision is sure enough to act on without asking."""
return decision["confidence"] >= float((cfg.get("decisions") or {}).get("act_above") or 0.8)
Expand Down
Loading
Loading