Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,12 +167,14 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
workflow, pipeline or routine, or change assets and notes. The Schedules page is Lens's crontab: every routine,
automation schedule and the backup in one list with when each runs next and last, to switch, re-time or run now, and
agents do the same over the API and MCP. A Classify node in a workflow sorts an asset into labels, content types or
topics, acting only on a sure answer, and an Entity filter node keeps the entities you want. Anything that reaches
outside Lens or changes more than what set it off asks first, until you let it run on its own: on the Approvals page
or, for a change inside Lens, with a yes in the chat it told you in; agents can decline an approval but never give
one. Each automation has limits and budgets, and agents make, try and run them over MCP and in the assistant, or
draft one from a sentence. An automation's workflow is drawn and tried on the workflow canvas, with its actions in
the palette; the studio's builder is planned. ([Processing](docs/processing.md#workflows-and-the-canvas),
topics, acting only on a sure answer, and an Entity filter node keeps the entities you want. Graph workflows tidy the
entities on a routine: look-alikes to merge or link, types the rules now read differently, and noise to hide, making
the changes they're sure of and proposing the rest, each one undoable. Anything that reaches outside Lens or changes
more than what set it off asks first, until you let it run on its own: on the Approvals page or, for a change inside
Lens, with a yes in the chat it told you in; agents can decline an approval but never give one. Each automation has
limits and budgets, and agents make, try and run them over MCP and in the assistant, or draft one from a sentence. An
automation's workflow is drawn and tried on the workflow canvas, with its actions in the palette; the studio's
builder is planned. ([Processing](docs/processing.md#workflows-and-the-canvas),
[converting assets](docs/processing.md#converting-assets), [notifications](docs/notifications.md),
[automations](docs/automations.md), [approvals](docs/automations.md#approvals), [schedules](docs/schedules.md),
[timeline](docs/timeline.md); [checks](docs/requirements.md#automate-it))
Expand Down
40 changes: 28 additions & 12 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -800,23 +800,39 @@ the backup, to switch, re-time and run each from one place; the Routines page an
A workflow's scope is assets (above), the graph, or an event: an automation's workflow, run by its automation
([The event workflow scope](automations.md#the-event-workflow-scope)). Graph workflows have their own nodes:

- **Candidates** finds pairs of entities that may be one thing, by the same rules as the merge suggestions (same
letters, acronym, spelling, sounds alike, one name inside the other): inside each namespace (merge) or across
namespaces whose graph is shared (link). Pairs someone marked as different, and pairs already proposed, are skipped.
- **Ask the model** (`llm_judge`) shows the model each pair with lines where the names appear and asks whether
they are the same thing, how sure it is, and which name to keep.
- **Filter** keeps the pairs that pass a test, e.g. `verdict.same` equals true.
- **Apply changes** merges or links the pairs whose confidence is at least *apply above* (at most *max apply* a run)
and proposes the rest. Without *apply above* it only proposes.
- **Find what to tidy** (`candidates`) finds, by rules and without a model, what *Look for* says:
- **Look-alikes** (merge, link): pairs of entities that may be one thing, by the same rules as the merge
suggestions (same letters, acronym, spelling, sounds alike, one name inside the other), inside each namespace
(merge) or across namespaces whose graph is shared (link). Pairs someone marked as different are skipped.
- **Wrong types** (retype): entities whose type the naming rules read differently now, from the name itself (a
title, an ending like *Labs* or *Inc*, a known place or organisation: 90% for a term, 80% for another type) or from
most of the lines that say it, at least two (up to 90%). An entity is typed by its first asset; this catches a
term that later lines show is a person. Types a person or an agent set, entities on a fixed list and hidden ones
are left alone, and so are types the namespace doesn't keep.
- **Noise** (hide): entities with no letters or one letter, made of filler words (*Okay so*; not acronyms like
*US*), or a phrase rather than a name (95%; filler words are 70% when the entity has a type other than Term), and
terms said only once (50%, below the default 70%). Entities someone showed again are left alone.
Anything already proposed, dismissed or undone is skipped too, so a run never redoes what someone took back.
- **Ask the model** (`llm_judge`) shows the model each candidate with lines where the names appear and asks, a kind
at a time: whether a pair is the same thing and which name to keep, what type a name is (from the namespace's
types), or whether it is noise; and how sure it is. Each gets a `verdict`, with `agree` true when the change
should be made (and, for pairs, `same`).
- **Filter** keeps the candidates that pass a test, e.g. `verdict.agree` (or `verdict.same`) equals true.
- **Apply changes** merges, links, retypes or hides the candidates whose confidence (the model's, else the rules') is
at least *apply above* (at most *max apply* a run) and proposes the rest. Without *apply above* it only proposes. A
candidate the model said no to is skipped.

A fresh archive has the workflow *Organise the entity graph* (both kinds of candidates, the model, then merges at 95%
or more, 25 a run) and the routine *Organise the graph every night* that runs it at 03:00 UTC, switched off. Turn it
on, or Run now with *propose only* first to see what it would do.

Every change is recorded (`GET /api/v1/graph-changes`): editors of the namespaces involved accept or dismiss
proposals (dismissing says the two are different, so the pair isn't suggested again) and undo applied changes; an
admin can undo everything a run applied at once (`POST /api/v1/routine-runs/{id}/undo`). Merges are undone exactly as
from an entity's page.
Every change is recorded (`GET /api/v1/graph-changes`), and shows in Routines › Proposed changes: editors of the
namespaces involved accept or dismiss proposals (dismissing a pair says the two are different; dismissing a type or a
hide leaves the entity as it is; neither is suggested again), choose another type when accepting a retype (`to`), and
undo applied changes; an admin can undo everything a run applied at once (`POST /api/v1/routine-runs/{id}/undo`).
Merges are undone exactly as from an entity's page; a type or a hide someone has changed since stays as they left it.
Agents do the same through the API, and `undo_graph_change` over MCP. To tidy types and noise, add a workflow that
looks for wrong types and noise (no built-in workflow does, so nothing changes until you add one).

### Publishing from the graph

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@ request, to another check that proves the same thing; loosening a requirement or
| A Classify node in workflows chooses from labels, content types or topics, and acting on a sure answer keeps something | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_a_workflow_classifies`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `chooses from labels, content types or topics, and acting on a sure answer keeps something` |
| An Entity filter node keeps a list's entities by type, stop list, length, pattern or how often they're said | [test_flow.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_flow.py) `test_entity_filter_keeps_the_entities_asked_for`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `an entity filter says what it keeps and refuses lowercase types` |
| Routines on a schedule | [test_routines.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_routines.py) `test_routines_are_checked_and_run_when_due`, `test_schedules`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `routine-create`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `routine-create` ≤ 2 clicks |
| Graph workflows find look-alikes to merge or link, wrong types and noise, make the sure changes and propose the rest, each undoable and not made again once taken back | [test_routines.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_routines.py) `test_graph_workflows_organise_entities`, `test_rules_read_types_and_noise`, `test_graph_workflows_retype_and_hide`; [graph-changes.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/graph-changes.test.tsx) `say a type change in words and make it` |
| Everything that runs on a clock (routines, automations' schedule and at triggers, the backup) in one Schedules list, switched, re-timed and run from it, by people and by agents over the API and MCP | [test_schedules.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_schedules.py) `test_one_list_holds_every_routine_automation_schedule_and_the_backup`, `test_schedules_are_changed_switched_and_run_from_the_list`, `test_agents_list_change_and_run_schedules_over_mcp`; [schedules-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/schedules-page.test.tsx) `lists every routine, automation schedule and the backup, each linking to its own page`, `switches one off, runs one now and changes when one runs, from the list` |
| Batch runs over a collection | [test_batches.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_batches.py) `test_collections_and_batches` |
| Notifications to webhooks, Slack, Discord and Matterbridge | [test_notifications.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_notifications.py) `test_owners_manage_targets_and_a_test_is_signed`, `test_runs_that_end_are_sent_once_to_matterbridge` |
Expand Down
3 changes: 2 additions & 1 deletion fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -1590,7 +1590,8 @@ def propose_graph_change(ctx: Context, kind: str, a: str, b: str, reason: str |
@tool(
"undo_graph_change",
"Undo a graph change",
"Take back a merge or link made by propose_graph_change (with apply, or once accepted): its id is the change's. "
"Take back a graph change that was made: a merge or link from propose_graph_change (with apply, or once accepted), "
"or a merge, link, type change or hide a graph workflow made (GET /api/v1/graph-changes). Its id is the change's. "
"Needs a token with the write scope and editor access.",
Arg("change_id", "integer", "the change (propose_graph_change's id)", required=True),
writes=True,
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/v1/routes/routines.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ def list_graph_changes(
def accept_graph_change(cid: int, body: GraphChangeAccept, user: Writer, acl: Acl, db: Db) -> Ok:
_change(db, acl, cid, "curate")
with domain_errors():
organize.accept(db, cid, user.email, body.keep)
organize.accept(db, cid, user.email, body.keep, body.to)
auth.audit(db, user.as_audit(), "graph_change.accept", f"graph_change:{cid}")
return Ok()

Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/graph_history.py
Original file line number Diff line number Diff line change
Expand Up @@ -526,7 +526,7 @@ def resolve(db, ref):

# ---------- rolling back ----------
MERGES = ("entity.merge", "merge.apply", "merge.accept")
APPLIED = ("merge.apply", "link.apply", "merge.accept", "link.accept")
APPLIED = tuple(f"{k}.{how}" for k in ("merge", "link", "retype", "hide") for how in ("apply", "accept"))
KEPT = ("analysis",) # what analysis found stays: it comes from the transcripts, and mentions point at it


Expand Down
31 changes: 20 additions & 11 deletions fastapi_backend/app/domain/name_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -285,25 +285,34 @@ def kind(words, before="", after="", people=frozenset()):
return None


def guess(name, lines=(), people=frozenset()):
"""The type rules give `name` from what it is and the lines it's said in: the type most of them agree on, else
None."""
def votes(name, lines=(), people=frozenset()):
"""({type: how many of the lines `name` is said in give it that type}, how many lines say it)."""
words = WORD.findall(name) or [name]
votes = {}
found = False
out, said = {}, 0
for line in lines:
text = line or ""
m = re.search(rf"(?<!\w){re.escape(name)}(?!\w)", text, re.I)
if not m:
continue
found = True
said += 1
t = kind(words, text[: m.start()], text[m.end() :], people)
if t:
votes[t] = votes.get(t, 0) + 1
if not found:
t = kind(words, people=people)
return t
return max(votes, key=lambda t: (votes[t], t == "PERSON")) if votes else None
out[t] = out.get(t, 0) + 1
return out, said


def best(counts):
"""The type most lines give, a person when it's a tie with one; None for no votes."""
return max(counts, key=lambda t: (counts[t], t == "PERSON")) if counts else None


def guess(name, lines=(), people=frozenset()):
"""The type rules give `name` from what it is and the lines it's said in: the type most of them agree on, else
None."""
counts, said = votes(name, lines, people)
if not said:
return kind(WORD.findall(name) or [name], people=people)
return best(counts)


# ---------- what the rules can't tell, a model may ----------
Expand Down
Loading
Loading