Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,10 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
language model, logged in the activity ledger; with `apply`, an editor's sure answer sets the content type, suggests
the topic or adds the tag (`POST /classify`, the MCP and assistant tool `classify`;
[Classifying](docs/processing.md#classifying)).
- **Classify in workflows.** A Classify node on the workflow canvas picks one of your labels, a content type or a topic
for the asset (or for what came in) and passes on the choice, how sure and every label ranked, for a Condition
or Switch to branch on; set to act, a sure answer sets the content type, suggests the topic or adds the tag
([Workflows and the canvas](docs/processing.md#workflows-and-the-canvas)).

## 0.13.0 <small>October 16, 2026</small> {id="0.13.0"}

Expand Down
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,16 @@ Everything below is built and merged, and each item links to the checks that pro
measured numbers ([requirements and their checks](docs/requirements.md)). Work that is still in a pull request is
listed under [In progress](#in-progress), and what comes next under [Planned](#planned).

- **Connect your agents.** Lens is an MCP server (`/mcp`): agents search, read assets' text and notes, cite exact
moments, find similar content, explore the graph with Cypher or SPARQL or browse it as folders and files, suggest
topics and graph changes, edit and merge entities and topics (with undo), write notes and make podcasts. They sign in
with OAuth and see only what their person may read; the account menu copies the MCP address in one click. A namespace
can also have an MCP server of its own (`/mcp/<namespace>`) and answer OpenAI's chat completions API for any app,
cited from that namespace alone, each off until its owner turns it on, with API keys that open that one namespace and
nothing else. The same folders and files can be mounted read-only over S3, WebDAV, NFS or SMB, so file managers and
scripts read Lens too. The web app runs on the same REST API, so an agent can do what a person can.
- **Connect your agents.** Lens is an MCP server (`/mcp`): agents search, find assets by face, voice, entity, topic,
type, content type or date in one call, classify an asset or a text with the decision model, read assets' text and
notes, cite exact moments, find similar content, explore the graph with Cypher or SPARQL or browse it as folders and
files, suggest topics and graph changes, edit and merge entities and topics (with undo), write notes and make
podcasts. They sign in with OAuth and see only what their person may read; the account menu copies the MCP address in
one click. A namespace can also have an MCP server of its own (`/mcp/<namespace>`) and answer OpenAI's chat
completions API for any app, cited from that namespace alone, each off until its owner turns it on, with API keys that
open that one namespace and nothing else. The same folders and files can be mounted read-only over S3, WebDAV, NFS or
SMB, so file managers and scripts read Lens too. The web app runs on the same REST API, so an agent can do what a
person can.
([MCP](docs/mcp.md), [one namespace's MCP server](docs/mcp.md#one-namespaces-mcp-server),
[chat completions](docs/api.md#chat-completions-for-any-app), [API](docs/api.md),
[the graph as files](docs/file-tree.md), [authentication](docs/authentication.md);
Expand Down Expand Up @@ -134,7 +136,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
people review on the Timeline page (and Latest on Home), marking what they've seen. Automations act on that stream, on
a schedule, at a time or when their own webhook is called: they test conditions, then tell you in your own chat app,
ask the model or the agent, use a tool, call a web address (its credentials kept sealed), run a workflow, pipeline or
routine, or change assets and notes. Anything that reaches outside Lens or changes more than what set it off asks
routine, or change assets and notes. A Classify node in a workflow sorts an asset into labels, content types or
topics, acting only on a sure answer. Anything that reaches outside Lens or changes more than what set it off asks
first, until you let it run on its own: on the Approvals page or, for a change inside Lens, with a yes in the chat it
told you in; agents can decline an approval but never give one. Each automation has limits and budgets, and agents
make, try and run them over MCP and in the assistant, or draft one from a sentence; the studio to draw them in the web
Expand Down Expand Up @@ -196,8 +199,6 @@ options most people leave as they are stay folded. ([Components](docs/components

These are open pull requests, not yet part of Lens:

- **Finding:** resources found by face, voice, entity, topic, type, content type and date
([#398](https://github.com/adeelahmad/lense/pull/398)).
- **The assistant** answering questions about Lens itself, from its docs and this server
([#400](https://github.com/adeelahmad/lense/pull/400)).
- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
Expand Down
17 changes: 11 additions & 6 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,9 @@ does), the topic is suggested for it, waiting for someone to accept it (one it a
from stays as it is), or the label given is added as a tag (at most 40 characters). `applied` says which, and it is
in the audit log. The MCP and assistant tools never apply; the assistant can then suggest the topic (`suggest_topic`).

Planned, not built yet: a classify step in workflows, classifying many assets at once, and labels with examples.
Workflows classify too, with a Classify node ([Workflows and the canvas](#workflows-and-the-canvas)), so a pipeline
can classify every asset it runs on. Planned, not built yet: classifying many assets at once from the API, and labels
with examples.

## Background work

Expand Down Expand Up @@ -593,11 +595,14 @@ node passes what it makes along its connections.
regular expressions (`TYPE: pattern`); Extract entities (LLM) asks the model for structured entities and keeps,
corrects or adds to any passed in; Merge joins lists; Save entities makes them the asset's entities (people's
corrections kept) and redoes keywords and chapters.
- **AI and logic:** LLM prompt (a prompt template, with what came in as `{{ input }}`), Pick (a path like
`action_items.0.text`), Condition (yes and no branches; one test, or a [match tree](automations.md#match-trees) of
several) and Merge.
- **Keeping results:** Save output (`outputs.<name>`), Set field (a custom field, kept in the metadata history) and
Save entities.
- **AI and logic:** LLM prompt (a prompt template, with what came in as `{{ input }}`), Classify (which of your
labels, the content types or the namespace's topics fits the asset, or what came in as text; it passes on `choice`,
`confidence`, `sure` and `ranked`, so a Condition or Switch can branch on them; [classifying](#classifying)), Pick
(a path like `action_items.0.text`), Condition (yes and no branches; one test, or a
[match tree](automations.md#match-trees) of several) and Merge.
- **Keeping results:** Save output (`outputs.<name>`), Set field (a custom field, kept in the metadata history), Save
entities, and a Classify node set to act on a sure answer (it sets the content type, suggests the topic or adds the
tag; a try on the canvas only says what it would do).

A pipeline runs a workflow as a Workflow step, pinned to the workflow's published version when the run is queued
(`POST /api/v1/workflows/{id}/run` runs one on an asset). Pipelines can be drawn on the canvas too: a connection
Expand Down
3 changes: 3 additions & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ request, to another check that proves the same thing; loosening a requirement or
| Requirement | Checks |
|---|---|
| An MCP server at `/mcp`: agents search, read assets' text and notes, and cite exact moments | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_search_and_cite`, `test_reading_recordings`, `test_note_tools` |
| Agents find assets by face, voice, entity, topic, type, content type and date in one call, each with the moments that match | [test_find.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_find.py) `test_by_face_voice_entity_and_date`, `test_by_topic_type_content_type_and_text`, `test_find_for_agents` |
| Agents classify an asset or a text against labels, content types or topics with the decision model, acting only on a sure answer | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_labels_given_and_what_the_model_reads`, `test_vocabularies_and_applying_a_sure_answer`, `test_agents_classify` |
| Agents find similar content | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_search_and_cite`; [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities` |
| Agents explore the graph with Cypher or SPARQL, or browse it as folders and files | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_graph_tools`, `test_sparql`; [test_file_tree.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_tree.py) `test_agents_browse_it_over_mcp` |
| Agents suggest topics and graph changes, and edit and merge entities and topics | [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_topic_tools`, `test_editing_topics_and_entities` |
Expand Down Expand Up @@ -155,6 +157,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Requirement | Checks |
|---|---|
| Pipelines and workflows on a canvas | [test_workflows.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_workflows.py) `test_workflows_run_in_pipelines_drawn_as_graphs`; [workflow-canvas.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-canvas.test.ts); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `pipeline-create`, `workflow-create` |
| A Classify node in workflows chooses from labels, content types or topics, and acting on a sure answer keeps something | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_a_workflow_classifies`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `chooses from labels, content types or topics, and acting on a sure answer keeps something` |
| Routines on a schedule | [test_routines.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_routines.py) `test_routines_are_checked_and_run_when_due`, `test_schedules`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `routine-create`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `routine-create` ≤ 2 clicks |
| Batch runs over a collection | [test_batches.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_batches.py) `test_collections_and_batches` |
| Notifications to webhooks, Slack, Discord and Matterbridge | [test_notifications.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_notifications.py) `test_owners_manage_targets_and_a_test_is_signed`, `test_runs_that_end_are_sent_once_to_matterbridge` |
Expand Down
58 changes: 56 additions & 2 deletions fastapi_backend/app/domain/workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
- `field`: sets a custom field of the recording to the value, checked like an edit (and kept in its history).
- `save_entities`: makes the entities passed in the recording's entities (in place of what analyze found), with
people's corrections kept, then redoes keywords and sections as analyze does.
- `classify`: which of some labels (or the content types, or the namespace's topics) fits the asset, or what came
in as text, as classify.py says: passes on {choice, confidence, sure, ranked, by}. With `apply` it acts on a sure
answer (sets the content type, suggests the topic, adds the tag), so it keeps something.

Positions are only for the canvas.

Expand All @@ -26,11 +29,14 @@

from __future__ import annotations

import json
import re

from . import (
analyze,
automation_nodes,
classify as classifymod,
decide,
fields as fieldmod,
flow,
llm,
Expand All @@ -44,7 +50,7 @@
)

R = store.R
OWN_NODES = ("llm", "extract_rules", "extract_llm", "output", "field", "save_entities")
OWN_NODES = ("llm", "extract_rules", "extract_llm", "classify", "output", "field", "save_entities")
TERMINAL = {"output", "field", "save_entities", "apply_changes"}
OPS = flow.OPS
OWN_CONFIG = {
Expand All @@ -54,6 +60,7 @@
"output": {"key"},
"field": {"field"},
"save_entities": set(),
"classify": {"labels", "vocabulary", "question", "of", "apply"},
}
ENTITY_TYPES = ("PERSON", "ORG", "PRODUCT", "PLACE", "EVENT", "WORK", "TERM", "DATE", "NUMBER")
PATH_RX = flow.PATH_RX
Expand Down Expand Up @@ -131,6 +138,29 @@ def _check_config(db, nid, t, cfg):
fieldmod.get(db, int(cfg.get("field") or 0))
except (KeyError, TypeError, ValueError):
raise ValueError(f"field node {nid}: choose a custom field") from None
elif t == "classify":
_check_classify(nid, cfg)


def _check_classify(nid, cfg):
labels, vocab = cfg.get("labels"), cfg.get("vocabulary")
if (labels is None) == (vocab is None):
raise ValueError(f"classify node {nid}: give labels, or choose content types or topics")
try:
if vocab is not None and vocab not in classifymod.VOCABULARIES:
raise ValueError(f"choose from {' or '.join(classifymod.VOCABULARIES)}")
if labels is not None:
if not isinstance(labels, list):
raise ValueError("labels are lines like science: research and medicine")
classifymod._labels(labels)
except ValueError as e:
raise ValueError(f"classify node {nid}: {e}") from None
if cfg.get("question") is not None and not isinstance(cfg["question"], str):
raise ValueError(f"classify node {nid}: the question is text")
if cfg.get("of", "resource") not in ("resource", "input"):
raise ValueError(f"classify node {nid}: it classifies the asset or what comes in")
if cfg.get("apply") and cfg.get("of") == "input":
raise ValueError(f"classify node {nid}: only the asset's classification can be acted on")


def _check_types(nid, cfg):
Expand Down Expand Up @@ -366,6 +396,8 @@ def _run_node(r, f, node, value, vals):
if after != (row.get("fields") or {}):
metadata.save_fields(db, cfg, rid, after, r.extra["by"])
r.say(f"set {fd['label']}")
elif t == "classify":
return {"out": _classify(r, c, upstream, name)}
elif t == "save_entities":
if not isinstance(value, list):
raise ValueError(f"{name}: needs a list of entities")
Expand All @@ -376,6 +408,27 @@ def _run_node(r, f, node, value, vals):
return {}


def _classify(r, c, upstream, name):
"""A classify node: the asset, or what came in as text."""
rid, text, space = r.extra["rid"], None, None
if c.get("of") == "input": # its topics are the recording's namespace's
text = upstream if isinstance(upstream, str) else json.dumps(upstream, ensure_ascii=False, default=str)
rid, space = None, r.db.values("SELECT VALUE space FROM $r", r=R("recording", int(rid)))[0]
try:
res = classifymod.classify(
r.db, r.cfg, rid=rid, text=text, labels=c.get("labels"), vocab=c.get("vocabulary"), question=c.get("question"),
apply=bool(c.get("apply")) and not r.dry, user=r.extra.get("by"), space=space,
) # fmt: skip
except decide.Undecided as e:
raise ValueError(f"{name}: nothing could answer ({e})") from None
sure = f"{round(res['confidence'] * 100)}% sure"
if c.get("apply") and r.dry and res["sure"]:
r.say(f"{name}: {res['choice']} ({sure}); would act on it")
else:
r.say(f"{name}: {res['choice']} ({sure}){'; ' + res['applied'].replace('_', ' ') if res['applied'] else ''}")
return {k: v for k, v in res.items() if k not in ("vocabulary", "question", "applied") and v is not None}


RUN_BY = {"graph": "organises the graph; a routine runs it", "event": "is what an automation does; its automation runs it"}


Expand Down Expand Up @@ -441,7 +494,8 @@ def _fit(f, v):
KITS = {
"recording": flow.Kit(
"recording", OWN_NODES, OWN_CONFIG, _check_config, _run_node, TERMINAL - {"apply_changes"},
"a workflow needs an output, field or save entities node, or it keeps nothing",
"a workflow needs an output, field or save entities node (or a classify node that acts), or it keeps nothing",
keeps=lambda n: n["type"] == "classify" and bool((n.get("config") or {}).get("apply")),
),
"graph": flow.Kit(
"graph", organize.OWN_NODES, organize.CONFIG, _check_config, organize.run_node, {"apply_changes"},
Expand Down
Loading
Loading