Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
(`POST /find`, the MCP tool `find`, the assistant's `find_resources`; [Finding by any
attribute](docs/processing.md#finding-by-any-attribute)). The Library list takes faces, entities, topics, content
types and emotions as filters too.
- **Classify an asset or a text.** Agents and the API can ask which of some labels fits, how sure that is and every
label ranked: labels they give, Lens's content types or a namespace's topics. The decision model answers, else the
language model, logged in the activity ledger; with `apply`, an editor's sure answer sets the content type, suggests
the topic or adds the tag (`POST /classify`, the MCP and assistant tool `classify`;
[Classifying](docs/processing.md#classifying)).

## 0.13.0 <small>October 16, 2026</small> {id="0.13.0"}

Expand Down
11 changes: 11 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1111,6 +1111,7 @@ asset later leaves its uploaded file in place, like any other media file.
GET /api/v1/search
GET /api/v1/search/terms
POST /api/v1/find
POST /api/v1/classify
GET /api/v1/similar
GET /api/v1/graph
GET /api/v1/mentions
Expand Down Expand Up @@ -1165,6 +1166,16 @@ per filter, and `narrower_topics`) and `text` (how the words were matched, as `G
moments looked through were `capped`). A name that matches nothing in the namespaces you can read is a 400 that names
the closest ones; an unknown field is a 422.

`POST /classify` says which label fits an asset or a text best ([classifying](processing.md#classifying)). The
body takes `recording` (an asset you can read) or `text` (at most 20,000 characters), and `labels` (a list, with
`label: what it means` items, or `{label: what it means}`) or `vocabulary` (`content_type` or `topic`; a text's topics
need `namespace`); `question` (at most 500 characters) asks it your way, and `apply` acts on a sure answer (an editor
of the asset). The reply has `choice`, `confidence` (0 to 1), `sure`, `ranked` (`[{label, p}]`, best first, with
the content type's `key` or the topic's `id`), `by` (`jev`, `laya`, `llm`, or `only option` when there is one), `vocabulary`, `question` and `applied`
(`content_type`, `topic_suggested`, `tag`, or null when nothing changed), plus the chosen content type's `key` or
topic's `id`. Bad labels or a missing choice are a 400, an asset you can't read a 404, `apply` without editing a 403,
and no model set up to answer a 409.

Search has no prefix search (`interp*` looks for the word "interp"). `GET /search/terms?prefix=interp` lists whole
words found in the namespaces you can read (`ns` for one) that start with it, the most frequent first, with how often and
in how many assets (`limit`, default 8, at most 20); the web app offers them as "Try …".
Expand Down
4 changes: 3 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,9 @@ When the configured model supports function calling, chat becomes an agent.
[stream of activities](automations.md#timeline-reads); `unreviewed` for what the person hasn't marked reviewed),
and assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates and the rest,
each with the moments that show why it matched (`find_resources`,
[Finding by any attribute](processing.md#finding-by-any-attribute)).
[Finding by any attribute](processing.md#finding-by-any-attribute)). It can also ask which of some labels, content
types or topics fits an asset or a text, and how sure that is (`classify`, [Classifying](processing.md#classifying));
it changes nothing.
- **The person's review marks:** `mark_activities` marks activities of the timeline reviewed, important or hidden for
the person ([Timeline](timeline.md#review-marks)). After it has read text from outside Lens, hiding or marking
reviewed waits for their yes; marking important or bringing things back acts at once.
Expand Down
1 change: 1 addition & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `search` | moments where words appear (written, spoken or shown on screen), best first: the asset, person, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, person or asset |
| `find_similar` | the assets (recordings, documents and images) most like a text, an asset, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links |
| `find` | assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates (a day, month or year), tags, language, objects, emotions, length, collection and words in them; ids or names. Each says why it matched, with moments and links to cite ([Finding by any attribute](processing.md#finding-by-any-attribute)) |
| `classify` | which of some labels fits an asset or a text, how sure, and every label ranked: labels you give, or Lens's content types or the namespace's topics; the decision model answers, else the language model. Read only: it changes nothing ([Classifying](processing.md#classifying)) |
| `list_namespaces` | the namespaces you can read, with their assets, hours, people and your role |
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_speakers` | the people of your namespaces, the ones with the most talk time first |
Expand Down
27 changes: 27 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,33 @@ narrower topics), `content_type` and `emotion`, besides its other filters ([API]
Planned, not built yet: these filters in the Library's filter bar and in saved collections, finding by what a photo
shows (`GET /search/photos`) inside the same filter, and ranking by more than the best matching moment.

### Classifying

The API (`POST /classify`), the assistant (`classify`) and the MCP server (`classify`) say which of some labels fits an
asset, or a piece of text, how sure that is, and every label ranked. The labels are:

* given, as a list (`["science", "sport: games and teams"]`, what a label means after `: `) or as
`{label: what it means}`, two to 60 of them;
* or a vocabulary Lens keeps: `content_type`, the [content types](#content-types) of the asset's base type (all of
them for text), or `topic`, its namespace's topics with their definitions and other labels (a text names the
namespace), the most used first.

The decision model answers when it is set up (Settings → AI assistant: Jev, typesafe.ai's System One, or Laya on
the machine), else the language model, asked for JSON; `by` says which, and the call is in the
[activity ledger](activity.md), counted for the asset and its namespace. The model reads the asset's kind, file
name, length, language, tags and accepted topics, and its title, summary, the entities it mentions most and the start
of its text, marked as data so instructions in it aren't followed
([prompt injection](assistant.md#third-party-content-and-prompt-injection)). `question`
asks something of your own ("Which desk should this go to?").

An answer is `sure` at `decisions.act_above` confidence or more (0.8 by default). Nothing changes unless `apply` is
set, by an editor, and the answer is sure: then the asset's content type is set (as choosing it in its Details tab
does), the topic is suggested for it, waiting for someone to accept it (one it already has, waits for or was dismissed
from stays as it is), or the label given is added as a tag (at most 40 characters). `applied` says which, and it is
in the audit log. The MCP and assistant tools never apply; the assistant can then suggest the topic (`suggest_topic`).

Planned, not built yet: a classify step in workflows, classifying many assets at once, and labels with examples.

## Background work

Imports, pipeline runs and folder scans return at once and run as jobs stored in SurrealDB. The server runs
Expand Down
44 changes: 44 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
approvals,
auth,
blobs,
decide,
events,
feeds,
filetree,
Expand All @@ -61,6 +62,7 @@
updates,
variables,
)
from app.domain import classify as classifymod
from app.domain import entities as ents
from app.domain import find as findmod
from app.domain import speakers as spk
Expand Down Expand Up @@ -750,6 +752,48 @@ def find(
return out


@tool(
"classify",
"Classify an asset or a text",
"Which label fits an asset (or a text) best, how sure that is, and every label ranked: labels you give "
'("name" or "name: what it means"), Lens\'s content types (vocabulary content_type: podcast, interview, scan, '
"email …, of the asset's kind) or its namespace's topics (vocabulary topic). A decision model answers when "
"one is set up, else the language model; `sure` says whether it is sure enough to act on without asking. "
"It changes nothing: to act, set the content type in the web app, suggest_topic, or tag it.",
Arg("recording_id", "integer", "the asset to classify"),
Arg("text", "string", "a text to classify instead", max_length=20000),
Arg("labels", "string[]", 'the labels to choose from (2 to 60): "name", or "name: what it means"'),
Arg("vocabulary", "string", "choose from Lens's content types or the namespace's topics instead", enum=("content_type", "topic")),
Arg("namespace", "string", "whose topics, when classifying a text by topic"),
Arg("question", "string", "what to ask (default: which label fits best)", max_length=500),
)
def classify(
ctx: Context,
recording_id: int | None,
text: str | None,
labels: list[str] | None,
vocabulary: str | None,
namespace: str | None,
question: str | None,
) -> dict[str, Any]:
space = None
if recording_id is not None:
_recording(ctx, recording_id)
elif namespace:
space = ctx.acl.namespace(namespace)
try:
res = classifymod.classify(
ctx.db, ctx.cfg, rid=recording_id, text=text, labels=labels, vocab=vocabulary, question=question, space=space
)
except decide.Undecided as e:
raise ToolError(f"nothing could answer: {e}") from None
out = {k: res[k] for k in ("choice", "confidence", "sure", "ranked", "by")}
for k in ("key", "id"):
if res.get(k) is not None:
out[k] = res[k]
return out


@tool(
"list_speakers",
"List people",
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
browser,
budgets,
chats,
classify,
collections,
comments,
completions,
Expand Down Expand Up @@ -109,6 +110,7 @@
uploads,
search,
find,
classify,
searches,
speakers,
entities,
Expand Down
47 changes: 47 additions & 0 deletions fastapi_backend/app/api/v1/routes/classify.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
"""Classify an asset or a text with the decision model (domain/classify.py)."""

from __future__ import annotations

from fastapi import APIRouter, HTTPException

from app.api.deps import Acl, Cfg, CurrentUser, Db, domain_errors
from app.domain import auth, decide
from app.domain import classify as classifymod
from app.schemas.classify import ClassifyRequest, ClassifyResult

router = APIRouter(tags=["search"])


@router.post("/classify")
def classify(body: ClassifyRequest, acl: Acl, user: CurrentUser, db: Db, cfg: Cfg) -> ClassifyResult:
"""Which label fits an asset (or a text) best, how sure that is, and every label ranked: the labels given, Lens's
content types (of the asset's kind) or the namespace's topics. The decision model answers when it's set up
(Settings → AI assistant), else the language model; the call is in the activity ledger. With `apply` (editors), a
sure answer is acted on: the content type is set, the topic is suggested for someone to accept, or the label is
added as a tag; one that isn't sure changes nothing. 409 when no model is set up to answer."""
space = None
if body.recording is not None:
acl.recording(body.recording, "edit" if body.apply else "viewer")
elif body.namespace:
space = acl.namespace(body.namespace)
with domain_errors():
try:
res = classifymod.classify(
db,
cfg,
rid=body.recording,
text=body.text,
labels=body.labels,
vocab=body.vocabulary,
question=body.question,
apply=body.apply,
user=user.email,
space=space,
)
except decide.Undecided as e:
raise HTTPException(409, f"nothing could answer: {e}") from None
if res["applied"]:
auth.audit(
db, user.as_audit(), "recording.classify", f"recording:{body.recording}", {"applied": res["applied"], "choice": res["choice"]}
)
return ClassifyResult.model_validate(res)
34 changes: 34 additions & 0 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
from . import (
auth,
batches,
classify as classifymod,
cypher,
decide,
entities,
event_marks,
extensions,
Expand Down Expand Up @@ -122,6 +124,23 @@
[],
False,
),
(
"classify",
"Classify an asset (or a text): which of some labels fits best, how sure, and every label ranked. Give "
'labels (names, or "name: what it means"), or a vocabulary: content_type (Lens\'s content types, such '
"as podcast, interview, scan, email) or topic (the namespace's topics). It changes nothing; sure says whether "
"the answer is sure enough to act on.",
{
"recording_id": _I,
"text": _S,
"labels": {"type": "array", "items": _S},
"vocabulary": {"type": "string", "enum": list(classifymod.VOCABULARIES)},
"namespace": _S,
"question": _S,
},
[],
False,
),
(
"mark_activities",
"Mark activities of the timeline (by their ids from timeline) reviewed, important or hidden, or clear those "
Expand Down Expand Up @@ -1408,6 +1427,21 @@ def t_find_resources(self, namespace=None, limit=10, **spec):
)
return {"total": res["total"], "resolved": res["resolved"] or None, "results": out}, f"Found {res['total']} asset(s)"

def t_classify(self, recording_id=None, text=None, labels=None, vocabulary=None, namespace=None, question=None):
space = None
if recording_id is not None:
self._ok(recording_id)
elif namespace:
space = self._space(namespace)
try:
res = classifymod.classify(
self.db, self.cfg, rid=recording_id, text=text, labels=labels, vocab=vocabulary, question=question, space=space
)
except decide.Undecided as e:
raise ValueError(f"nothing could answer: {e}") from None
out = {k: res[k] for k in ("choice", "confidence", "sure", "ranked", "by", "key", "id") if res.get(k) is not None}
return out, f"Classified as {res['choice']} ({round(res['confidence'] * 100)}% sure)"

def t_read_transcript(self, recording_id, start_seconds=0, end_seconds=None, max_lines=60):
cap = self.cfg["ai"].get("max_transcript_reads") or 20
if self.reads >= cap:
Expand Down
Loading
Loading