diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a402b114..dc0304704 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version stream could wait for a database connection its own thread already held, and every request after it waited too. A query made inside another on the same thread now runs on that connection ([Concurrency](docs/database.md#concurrency)). +- **The decision model types the names the rules can't.** When Jev or Laya is set up, analysis asks it what each new + Term of a recording is (a person, an organisation, a product...), with the lines it's said in, in one call per + recording, and keeps the types it is sure of. Settings › Analysis › Names the rules can't type (`analysis.typing`) + can also let the language model answer, or turn it off; the language model is never asked by default + ([Names the rules can't type](docs/processing.md#entities-and-the-graph-explorer)). ## 0.14.0 October 17, 2026 {id="0.14.0"} diff --git a/DEVLOG.md b/DEVLOG.md index 3572e7187..86127258c 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -95,8 +95,12 @@ organisations and products, holidays), the words a name starts or ends with, rol verbs of saying after it and the namespace's named speakers; a recording gives each name one type. Upgrade step `entity-types` retypes old analysis-made Terms, skipping types people or agents set (graph history and audit log). -Refine later: the decision model (Jev or Laya) for the names the rules leave as Terms, in analysis and as an action -on existing Terms (next PR); place, name and brand lists for more languages and regions (Custom vocabulary in +Then the decision model: `analysis.typing` (`auto` by default: Jev or Laya when set up, never the language model; +`model` lets the language model answer; `rules` none) asks about a recording's new Terms in one `decide.choose_all` +call, with two lines each, and keeps answers above `decisions.act_above`. + +Refine later: an action that asks the model about existing Terms (API, MCP, a button on the Entities page; next PR); +offer the namespace's own types to the model too; ask about names in the hybrid mode's new entities; place, name and brand lists for more languages and regions (Custom vocabulary in Settings covers one install's own names); type existing entities again when a later recording gives better evidence (needs a marker for types people set, kept out of the versioned rows); "Google Cloud"-like names (a known company, then a word) as products. diff --git a/README.md b/README.md index 8bbcc7c9f..ecfc6de52 100644 --- a/README.md +++ b/README.md @@ -120,12 +120,12 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p SPARQL, and see as it was at any version. Every entity has its own page at a wiki address (`/wiki//`), and entities, topics and people are edited and merged from their lists, the graph or an asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by - rules, without a model. Find what is most like a passage, an asset, a topic or an entity, in a Similar - tab on every asset or by selecting text anywhere, and, with anytopdf, photos by what they show. Pages and shots are - described in words by the model provider's vision model or, off by default, by Florence-2 or Moondream on your own - server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation - put back), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each - off by default. + rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an + asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos + by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by + Florence-2 or Moondream on your own server. Transcripts can be cleaned up before analysis (stutters, "um"s, spacing + and, with a local model, punctuation put back), entities found by spaCy instead of the rules, and the dates people say + read as days on the calendar, each off by default. ([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages), [similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md), [topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md), diff --git a/docs/assistant.md b/docs/assistant.md index 1fcac9860..5f6fdc482 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -804,6 +804,8 @@ The assistant makes routine choices for you instead of asking. The first is wher `import_files` without a namespace picks one from the namespaces you can add to, judging from their descriptions, what's in them lately, the file names and your message. You're asked only when it isn't sure: below `decisions.act_above` (0.8) confidence it lists the namespaces, its best guess first. +Analysis takes routine choices too: what a name the rules can't type is (a person, an organisation, a product...; +[Names the rules can't type](processing.md#entities-and-the-graph-explorer)). A **decision model** answers these: Jev, typesafe.ai's System One, which takes a few hundred milliseconds and costs a fraction of an LLM call. Set its key in **Settings → AI assistant → Routine choices**, or `TYPESAFE_API_KEY` in `.env`. diff --git a/docs/processing.md b/docs/processing.md index 5c1cf5e19..b9f469e35 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -171,6 +171,21 @@ reads each name and the words around it (`app/domain/name_types.py`): topic, or retyped on the Entities page (one at a time or ticked together). Names listed in Settings › Analysis › Custom vocabulary (`Name | TYPE`) always get the type given there. +**Names the rules can't type.** When a [decision model](assistant.md#routine-choices) is set up (Jev with its key, or +Laya), analysis asks it what each new Term of a recording is, with the first two lines it's said in, and keeps the type +when it is sure (`decisions.act_above`, 0.8). Names that are already entities aren't asked again, and the short form of +a longer name in the same recording ("Johnson" of "Sarah Johnson") takes that name's type. The names of a recording go +in one call (up to 20 a call and 40 a recording; Laya, which reads less, one at a time). Each call is in Activity and +counts toward budgets, and none is made once the budget's hard limit is reached. Settings › Analysis › **Names the rules +can't type** (`analysis.typing`) chooses: + +- `auto` (the default): the decision model when one is set up, else nothing; the language model is never asked. +- `model`: the decision model, else the language model, in one call per recording too. +- `rules`: no model; the names stay Terms. + +A name the model isn't sure of stays a Term, and so do all of them when it can't be reached. Names a workflow's nodes +typed, and names mapped onto a fixed list ([Entity setup](#entity-setup)), aren't asked. + A name gets one type per recording, the one most of its lines give it, so "Lee" after "Dr Lee" is a person too. "And" no longer joins two names into one ("Kenya and Nigeria" are two places). The given names are mostly the US Census Bureau's frequent first names of 1990 (public domain); the other lists are written for Lens. With diff --git a/docs/requirements.md b/docs/requirements.md index b737234a2..5667ca092 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -147,6 +147,7 @@ request, to another check that proves the same thing; loosening a requirement or | Every entity's own page at `/wiki//` | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_entities.py) `test_wiki_addresses`; [wiki-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/wiki-page.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-find` | | Entities, topics and people edited and merged from their lists, the graph or an asset, with Undo instead of a confirm | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_entities.py) `test_suggestions_and_merge_undo`; [entities-curate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/entities-curate.test.tsx); [topics-curate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/topics-curate.test.tsx); [recording-entities-edit.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-entities-edit.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-merge`, `graph-merge`, `speaker-rename` | | Names get their real types (people, places, organisations, events, works) by rules, without a model; older Terms are typed on upgrade | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_names_get_the_type_their_words_give`, `test_analysis_types_people_places_and_events`, `test_the_upgrade_types_the_old_terms_and_keeps_what_people_chose` | +| A decision model (Jev or Laya) types the names the rules leave as Terms, in one call, keeping only types it is sure of; Settings › Analysis can leave them Terms | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_the_decision_model_types_what_the_rules_cant`, `test_a_type_the_model_isnt_sure_of_stays_a_term`, `test_by_default_the_language_model_isnt_asked`, `test_with_typing_rules_nothing_is_asked` | | What is most like a passage, an asset, a topic or an entity | [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities`, `test_like_a_topic_and_an_entity`; [similar.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/similar.test.tsx) | | With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` | | Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` | diff --git a/fastapi_backend/app/domain/analyze.py b/fastapi_backend/app/domain/analyze.py index 6f79629ee..ec2260e92 100644 --- a/fastapi_backend/app/domain/analyze.py +++ b/fastapi_backend/app/domain/analyze.py @@ -446,6 +446,32 @@ def one_type(seg_ents): return [[(n, best.get(ent_key(n), t)) for n, t in es] for es in seg_ents] +def model_types(db, cfg, nid, segs, seg_ents, gaz=()): + """The terms of a recording that aren't entities yet, given the type a model is sure of (name_types.ask). A short + form of a longer name said here ("Johnson" of "Sarah Johnson") takes that name's type later, and a name the custom + vocabulary (`gaz`) lists is the type it says, a term too; neither is asked.""" + if not name_types.asking(cfg): + return seg_ents + listed = {ent_key(g[0]) for g in gaz} + terms = {} + for es in seg_ents: + for n, t in es: + k = ent_key(n) + if k and t == "TERM" and k not in listed: + terms.setdefault(k, n) + keys = {ent_key(n) for es in seg_ents for n, _ in es} + terms = {k: n for k, n in terms.items() if not any(L.endswith(" " + k) for L in keys)} + if terms: + have = set(db.values("SELECT VALUE key FROM entity WHERE space = $s AND key IN $k", s=nid, k=sorted(terms))) + have |= set(db.values("SELECT VALUE key FROM entity_alias WHERE space = $s AND key IN $k", s=nid, k=sorted(terms))) + terms = {k: n for k, n in terms.items() if k not in have} + if not terms: + return seg_ents + texts = [s["text"] for s in segs] + typed = name_types.ask(cfg, {k: (n, name_types.said_in(n, texts)) for k, n in terms.items()}) + return [[(n, typed.get(ent_key(n), t) if t == "TERM" else t) for n, t in es] for es in seg_ents] if typed else seg_ents + + def analyze_recording(db, cfg, rid, seg_ents=None, say=None): """Entities, keywords, sections and talk statistics. `seg_ents` (per line, [(name, type)]) replaces the extractor: a workflow's save entities node passes what its nodes found.""" @@ -486,6 +512,8 @@ def analyze_recording(db, cfg, rid, seg_ents=None, say=None): have = {ent_key(n) for n, _ in es} es.extend(h for h in hints if ent_key(h[0]) not in have) seg_ents = one_type(seg_ents) + if given is None and not listed: # names the rules can't tell: a model may (Settings › Analysis) + seg_ents = model_types(db, cfg, nid, segs, seg_ents, gaz) if setup["types"] and not listed: # the namespace (or collection) keeps only some types seg_ents = [[(n, t) for n, t in es if entity_setup.keeps(setup, t)] for es in seg_ents] surface = {t: c.most_common(1)[0][0] for t, c in ts.items()} diff --git a/fastapi_backend/app/domain/decide.py b/fastapi_backend/app/domain/decide.py index e1c6a9621..d91b1795c 100644 --- a/fastapi_backend/app/domain/decide.py +++ b/fastapi_backend/app/domain/decide.py @@ -212,6 +212,69 @@ def choose(cfg, question, options, state): } +def choose_all(cfg, asks, state, language_model=True): + """Several choices about one `state` in one call: `asks` is {id: (question, options)}; returns {id: decision} as + `choose` gives them, for the questions answered. The language model answers when no decision model can, unless + `language_model` is False. Raises Undecided when nothing can answer.""" + asks = {q: (text, dict(o)) for q, (text, o) in asks.items() if o} + if not asks: + return {} + ids = {f"q{i}": q for i, q in enumerate(asks)} # plain names for the questions, whatever the caller's ids + how = engine(cfg) + if how in ENGINES: + questions = {i: {"type": "choice", "instructions": asks[q][0], "criteria": asks[q][1]} for i, q in ids.items()} + try: + answers = ENGINES[how](cfg, questions, state) + out = {} + for i, q in ids.items(): + a, o = answers.get(i) or {}, asks[q][1] + if a.get("choice") in o: + probs = a.get("probabilities") or {a["choice"]: a.get("confidence", 0)} + out[q] = {"choice": a["choice"], "confidence": float(a.get("confidence") or 0), "ranked": _ranked(probs), "by": how} + return out + except Undecided as e: + if not language_model: + raise + log.warning("decision model (%s) failed, asking the language model: %s", how, e) + how = "llm" if llm.configured(cfg) else None + if how != "llm" or not language_model: + raise Undecided("no decision model is set up" if not language_model else "no decision model or language model is set up") + one = { + i: { + "type": "object", + "properties": {"choice": {"type": "string", "enum": list(asks[q][1])}, "confidence": {"type": "number"}}, + "required": ["choice", "confidence"], + } + for i, q in ids.items() + } + schema = {"type": "object", "properties": {"answers": {"type": "object", "properties": one}}, "required": ["answers"]} + listed = "\n\n".join(f"{i}: {asks[q][0]}\n" + "\n".join(f"- {k}: {v}" for k, v in asks[q][1].items()) for i, q in ids.items()) + try: + reply = llm.json_out( + cfg, + "You take routine decisions. For each question pick the one option that fits, by its name, and say how sure " + "you are from 0 to 1: below 0.8 when the evidence is thin or two options fit.", + f"Questions:\n\n{listed}\n\nState:\n{_state(state)}", + schema, + ) + except llm.LLMError as e: + raise Undecided(str(e)) from None + out = {} + for i, q in ids.items(): + a, options = (reply.get("answers") or {}).get(i) or {}, asks[q][1] + if a.get("choice") not in options: + continue + conf = min(1.0, max(0.0, float(a.get("confidence") or 0))) + rest = (1 - conf) / max(1, len(options) - 1) + out[q] = { + "choice": a["choice"], + "confidence": conf, + "ranked": _ranked({k: conf if k == a["choice"] else rest for k in options}), + "by": "llm", + } + return out + + def sure(cfg, decision): """Whether a decision is sure enough to act on without asking.""" return decision["confidence"] >= float((cfg.get("decisions") or {}).get("act_above") or 0.8) diff --git a/fastapi_backend/app/domain/name_types.py b/fastapi_backend/app/domain/name_types.py index a31c77fe5..ce41e5f56 100644 --- a/fastapi_backend/app/domain/name_types.py +++ b/fastapi_backend/app/domain/name_types.py @@ -9,7 +9,9 @@ - PLACE: countries, regions, states and large cities (lexicon/places.txt); EVENT: holidays; DATE: days and months. - ORG, PRODUCT: well-known organisations and products (lexicon/organisations.txt, products.txt), and for PRODUCT a name with a number in it (iPhone 16). -A name none of these place is a TERM: a single unknown word (a brand, a project) needs a model to tell. +A name none of these place is a TERM: a single unknown word (a brand, a project) needs a model to tell. `ask` gives +such names to the decision model, with the lines they're said in (Settings › Analysis › Types the rules can't tell, +`analysis.typing`): by default only when a decision model is set up (Jev or Laya), never the language model. The given names are mostly the US Census Bureau's frequent first names of 1990 (public domain), with others added by hand; the places, organisations and products are written by hand (lexicon/). Words that are names but mostly something else on their own (Will, @@ -19,10 +21,14 @@ from __future__ import annotations import functools +import logging import pathlib import re +from . import decide + HERE = pathlib.Path(__file__).parent / "lexicon" +log = logging.getLogger("lens") TITLES = { "Dr", "Mr", @@ -298,3 +304,61 @@ def guess(name, lines=(), people=frozenset()): t = kind(words, people=people) return t return max(votes, key=lambda t: (votes[t], t == "PERSON")) if votes else None + + +# ---------- what the rules can't tell, a model may ---------- +ASKED = ("PERSON", "ORG", "PRODUCT", "PLACE", "EVENT", "WORK", "TERM") # the types a model picks from +TYPINGS = ("auto", "model", "rules") +BATCH = 20 # names in one call +MOST = 40 # names asked about at once (one recording, or one go of type_terms) +LINES = 2 # lines a name is said in, given with it + + +def _typing(cfg): + return (cfg.get("analysis") or {}).get("typing") or "auto" + + +def asking(cfg): + """Which model gives types to the names the rules leave terms (`analysis.typing`), or None: with "auto" (the + default) the decision model when one is set up (Jev or Laya), with "model" the language model too when none is, + with "rules" none. Decisions turned off (decisions.engine: off) is none either.""" + want = _typing(cfg) + how = decide.engine(cfg) + if want == "rules" or how is None: + return None + return how if how in decide.ENGINES or want == "model" else None + + +def said_in(name, lines, most=LINES): + """The first `most` lines `name` is said in, each cut to a sentence's length.""" + pat = re.compile(rf"(? option, sure of it; None (or no option) falls back to `answer` status = 200 def log_message(self, *a): @@ -22,6 +23,10 @@ def do_POST(self): answers = {} for name, q in body["questions"].items(): first = next(iter(q["criteria"])) + got = Handler.pick(q["instructions"], body["state"]) if Handler.pick else None + if got: + answers[name] = {"type": "choice", "choice": got, "confidence": 0.95, "probabilities": {got: 0.95}} + continue answers[name] = Handler.answer or {"type": "choice", "choice": first, "confidence": 1.0, "probabilities": {first: 1.0}} data = json.dumps({"model": "jev-1.13.0", "answers": answers, "usage": {"input_tokens": 400, "output_tokens": 40}}).encode() self.send_response(Handler.status) @@ -32,7 +37,7 @@ def do_POST(self): def start(cfg, key="jev-key"): - Handler.seen, Handler.answer, Handler.status = [], None, 200 + Handler.seen, Handler.answer, Handler.pick, Handler.status = [], None, None, 200 srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) threading.Thread(target=srv.serve_forever, daemon=True).start() cfg["decisions"] = {**cfg.get("decisions", {}), "base_url": f"http://127.0.0.1:{srv.server_address[1]}/v1", "api_key": key} diff --git a/fastapi_backend/tests/fake_llm.py b/fastapi_backend/tests/fake_llm.py index 4d3be7a79..62a0160b2 100644 --- a/fastapi_backend/tests/fake_llm.py +++ b/fastapi_backend/tests/fake_llm.py @@ -15,6 +15,7 @@ class Handler(http.server.BaseHTTPRequestHandler): blind = False # behave like a server whose model can't see images names = None # namespace names suggested for a question (auto_scope.py) decision = None # the answer to a decision (decide.py) when no decision model is set up; else the first option + pick = None # a function (question) -> option, for several decisions in one call (decide.choose_all); else the first cypher_script = [] # Cypher to answer graph questions with (graph_ask.py), in order podcast_script = None # the lines a podcast script comes back with (podcasts.write), else a built-in one spec = None # a spec's answer (specs.write), else one quoting the transcript's lines in turn (see _spec) @@ -249,6 +250,15 @@ def do_POST(self): title = re.search(r"^Title: (.*)$", text, re.M).group(1) first = text.split("Note:\n", 1)[1].strip().splitlines()[0] content = json.dumps({"title": "Capsid plan" if title == "Untitled" else title, "summary": f"About {first}"}) + elif set(schema.get("properties", {})) == {"answers"}: + # several decisions at once (decide.choose_all): Handler.pick's option for each question, else the first + asked = dict(re.findall(r"^(q\d+): (.*)$", body["messages"][-1]["content"], re.M)) + answers = {} + for i, a in schema["properties"]["answers"]["properties"].items(): + options = a["properties"]["choice"]["enum"] + got = Handler.pick(asked.get(i, "")) if Handler.pick else None + answers[i] = {"choice": got or options[0], "confidence": 0.9} + content = json.dumps({"answers": answers}) elif set(schema.get("properties", {})) == {"choice", "confidence"}: content = json.dumps(Handler.decision or {"choice": schema["properties"]["choice"]["enum"][0], "confidence": 0.9}) elif str(body["messages"][0].get("content") or "").startswith("You write research reports"): diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index ba5002530..0c40ed89e 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -252,6 +252,7 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) { )} {raw("analysis.entities") === "spacy" && } +