Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
recognised by voice (long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities
and topics in a knowledge graph you can explore, query with Cypher or
SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
SPARQL, and see as it was at any version, where every source, sensor, person and agent is a node linked to what it
brought in. Every entity has its own page at a wiki address
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an
Expand Down
6 changes: 6 additions & 0 deletions changelog.d/graph-actors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
**Sources, sensors, people and agents in the graph.** Whoever brought an asset in (the actor of its activity on the
timeline: a storage source, a sensor, a person, an automation, a routine, an import hook, or Lens itself) is now a
Source node in the graph with an ADDED link to the asset, so `(Source)-[:ADDED]->(Recording)-[:MENTIONS]->(Entity)`
says where Lens learns about each person, place or organisation. The canvas draws sources linked to what their assets
mention, the explorer lists an asset's source among its parents, and the Timeline page's Graph button opens the canvas
on the sources alone ([Actors in the graph](docs/timeline.md#actors-in-the-graph)).
13 changes: 9 additions & 4 deletions docs/graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ to itself and has one. On a computer the Find in graph box has the focus, so typ
| Speaker | `s<id>` | a person: name, seconds (talk time), namespace |
| Entity | `e<id>`, or `e:<key>` in the global scope | name, type, key, mentions, namespaces, ids |
| Topic | `t<id>`, or `t:<key>` in the global scope | name (the label), alt, definition, recordings, namespaces, ids |
| Source | `a:<actor>`, such as `a:source/12` or `a:account/1` | who brought assets in, as the [timeline](timeline.md)'s actors: name, actor (the activity's actor id), type (Service, Person or Application), kind (source, sensor, person, hook or agent), recordings, namespaces |

An entity has its type as a second label: `Person`, `Organisation`, `Product`, `Place`, `Event`, `Work`, `Term`, or
a namespace's own type (`MY_TYPE` becomes `MyType`). Every node also has `id`, `name` and `namespace`. Hidden entities
Expand All @@ -65,10 +66,14 @@ and quiet types (dates and numbers) are left out, as in the overview graph.
| ABOUT | Recording → Topic | source, weight |
| NARROWER | Topic → Topic (from the broader one) | |
| RELATED | Topic → Topic | |

CONTAINS, HAS_SPEAKER, MENTIONS, SAID, ABOUT and NARROWER are the hierarchy: an entity's parents are the assets that
mention it and the people who said or wrote it; its ancestors go on up to collections and the namespace. A topic's parents
are its broader topics and the assets about it. Topics are each namespace's vocabulary ([Topics](topics.md)).
| ADDED | Source → Recording (the source brought the asset in) | |

CONTAINS, HAS_SPEAKER, MENTIONS, SAID, ABOUT, NARROWER and ADDED are the hierarchy: an entity's parents are the assets
that mention it and the people who said or wrote it; its ancestors go on up to collections and the namespace. A topic's
parents are its broader topics and the assets about it. Topics are each namespace's vocabulary ([Topics](topics.md)).
An asset's parents are its collection and the source that added it, so `MATCH (s:Source)-[:ADDED]->(:Recording)-[:MENTIONS]->(e:Organisation) RETURN s.name, count(DISTINCT e)`
says where Lens learns about each organisation. On the canvas, a source links to what its assets mention (Brought in),
and the Timeline page's Graph button opens the canvas on the sources alone.

## Scope and rights

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Low quality images enhanced before they're read, off by default | [test_enhance.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_enhance.py) `test_off_by_default_the_original_is_read`, `test_a_low_quality_image_is_read_from_its_enhanced_copy`, `test_a_good_image_is_left_as_it_is`, `test_agents_and_automations_enhance_too`, `test_the_copy_is_encrypted_where_the_files_are`; [enhance.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/enhance.test.tsx) |
| Entities and topics in a knowledge graph to explore and query with Cypher or SPARQL | [test_graph.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph.py) `test_query`; [test_rdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rdf.py) `test_sparql`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `graph-find` |
| The graph as it was at any version | [test_graph_history.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph_history.py) `test_the_explorer_and_cypher_as_of_a_version`, `test_the_canvas_as_of_a_version`; [test_graph_history.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_graph_history.py) `test_every_curation_is_an_event_and_every_version_replays` |
| Sources, sensors, people and agents as actors in the graph, with ADDED links to what they brought in, on the canvas, in the explorer and in Cypher | [test_graph.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph.py) `test_sources_are_actors_in_the_graph`; jest [graph-explorer.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/graph-explorer.test.ts) `shows a source only while something it brought in is visible` |
| Every entity's own page at `/wiki/<namespace>/<Name>` | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_entities.py) `test_wiki_addresses`; [wiki-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/wiki-page.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-find` |
| Entities, topics and people edited and merged from their lists, the graph or an asset, with Undo instead of a confirm | [test_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_entities.py) `test_suggestions_and_merge_undo`; [entities-curate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/entities-curate.test.tsx); [topics-curate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/topics-curate.test.tsx); [recording-entities-edit.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-entities-edit.test.tsx); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `entity-merge`, `graph-merge`, `speaker-rename` |
| Names get their real types (people, places, organisations, events, works) by rules, without a model; older Terms are typed on upgrade | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_names_get_the_type_their_words_give`, `test_analysis_types_people_places_and_events`, `test_the_upgrade_types_the_old_terms_and_keeps_what_people_chose` |
Expand Down
14 changes: 13 additions & 1 deletion docs/timeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ Status: in progress. The studio's timeline is built in steps; only what is merge
| Ask about an item: talk to the assistant about it from the item, in the chat beside the page | built |
| What each item caused (automation runs, approvals), and Approve and Decline on waiting approvals | planned |
| Tagging the agent in a comment, so it answers in the thread | built |
| Sources, sensors, people and agents as actors in the graph; a For you order learned from what each person reacts to, comments on and hides | planned |
| Sources, sensors, people and agents as actors in the graph: a Source node with ADDED links, on the canvas, in the explorer and in Cypher | built |
| A For you order learned from what each person reacts to, comments on and hides | planned |

## What shows

Expand Down Expand Up @@ -109,6 +110,17 @@ ledger against their conversation and the item's namespace; when no model is set
used up, the reply says so instead. **Settings → AI assistant → Answer when tagged in a comment**
(`ai.answer_comments`, on) turns it off; without a tag nothing is asked of the model.

## Actors in the graph

Whoever brought an asset in, the actor of its `asset.added` activity (a storage source, a sensor, a person, an
automation, a routine, an import hook, or Lens itself), is a **Source** node in the [graph](graph.md) with an **ADDED**
relationship to the asset, so the graph says where Lens learns about each person, place or organisation:
`(Source)-[:ADDED]->(Recording)-[:MENTIONS]->(Entity)`. The explorer's parents of an asset are its collection and its
source; on the canvas a source links to what its assets mention (a *Brought in* edge, one per source and entity,
weighted by mentions), and the Timeline page's **Graph** button opens the canvas on the sources alone
(`/graph?show=actors`). Agents read the same through `GET /api/v1/graph/related`, `POST /api/v1/graph/query` and the
MCP graph tools. Nothing is stored for it: the nodes come from each asset's own record.

## Review marks

Each person marks activities for themselves alone; nobody else sees or sets them, admins included.
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/v1/routes/graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ def graph_related(
user: CurrentUser,
acl: Acl,
db: Db,
node: str = Query(description="a node id: n<id>, c<id>, r<id>, s<id>, e<id> or e:<name key>"),
node: str = Query(description="a node id: n<id>, c<id>, r<id>, s<id>, e<id> or e:<name key>, t<id>, or a:<actor> for a source"),
relation: Literal["children", "parents", "ancestors", "descendants", "neighbours"] = "neighbours",
depth: int = Query(1, ge=1, le=8),
types: str | None = Query(
Expand Down
19 changes: 15 additions & 4 deletions fastapi_backend/app/domain/events_content.py
Original file line number Diff line number Diff line change
Expand Up @@ -407,16 +407,27 @@ def _asset(db, r, types):
}


def _activities(db, rows):
"""The rows as activities of their kinds (a row whose created_at isn't a time is skipped)."""
def added_by(db, rows):
"""{recording id: the actor that brought it in}, for recording rows carrying id, space, remote, created_by and
sensor: the asset.added activity's actor (an automation, the storage source, whoever made it, the upload's account,
a sensor, whoever queued its first job, else Lens). The graph's Source nodes come from it (graph_model.py)."""
rows = [r for r in rows if r]
ids = [r["id"] for r in rows]
if not ids:
return []
return {}
first = _first_jobs(db, ids)
uploads = {u["recording"]: u for u in db.rows("SELECT recording, account, email FROM upload WHERE recording IN $ids", ids=ids)}
labels = {r["id"]: (_label_of(r, first.get(r["id"]), uploads.get(r["id"])), r.get("space")) for r in rows}
who = actors(db, labels.values())
return {rid: who[label] for rid, label in labels.items()}


def _activities(db, rows):
"""The rows as activities of their kinds (a row whose created_at isn't a time is skipped)."""
rows = [r for r in rows if r]
if not rows:
return []
who = added_by(db, rows)
srcs = _lookup(
db,
"storage_source",
Expand All @@ -430,7 +441,7 @@ def _activities(db, rows):
published = events.stamp(r["created_at"])
except (TypeError, ValueError):
continue
actor = who[labels[r["id"]]]
actor = who[r["id"]]
obj = _asset(db, r, types)
kind = _kind(r, obj["content_type"], (srcs.get((r.get("remote") or {}).get("source")) or {}).get("type"), actor)
if kind == "email.received":
Expand Down
37 changes: 35 additions & 2 deletions fastapi_backend/app/domain/graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,35 @@ def build(db, cfg, scope="global", allowed=None, as_of=None):
"recordings": recs_per[r["id"]],
"refs": [r["id"]],
}
# who brought each asset in, as the activity stream's actors: a Source node each, linked to what they mention
from . import events_content, graph_model

recs = db.rows("SELECT record::id(id) AS id, space, remote, created_by, sensor FROM recording WHERE space IN $s", s=nids)
src_of = {}
who = events_content.added_by(db, recs)
for r in recs:
a = who.get(r["id"])
if not a:
continue
key = graph_model.source_node_id(a["id"])
src_of[r["id"]] = key
n = nodes.setdefault(
key,
{
"id": key,
"kind": "source",
"label": a["name"],
"type": graph_model.actor_kind(a["id"]),
"ns": set(),
"weight": 0,
"refs": [],
},
)
n["weight"] += 1
n["ns"].add(names[r["space"]])
for n in nodes.values():
if n["kind"] == "source":
n["ns"] = sorted(n["ns"])
counts = {r["entity"]: r["n"] for r in db.rows("SELECT entity, count() AS n FROM mentions WHERE space IN $s GROUP BY entity", s=nids)}
shown = {}
for r in db.rows("SELECT entity, text, count() AS n FROM mentions WHERE space IN $s GROUP BY entity, text", s=nids):
Expand Down Expand Up @@ -123,18 +152,22 @@ def edge(a, b, k):
for i in range(len(g)):
for j in range(i + 1, len(g)):
edge(g[i], g[j], "together")
seg_ents, seg_spk = defaultdict(set), {}
for r in db.rows("SELECT record::id(in) AS seg, entity, speaker FROM mentions WHERE space IN $s", s=nids):
seg_ents, seg_spk, seg_src = defaultdict(set), {}, {}
for r in db.rows("SELECT record::id(in) AS seg, entity, speaker, recording FROM mentions WHERE space IN $s", s=nids):
n = node_of.get(r["entity"])
if n in nodes:
seg_ents[r["seg"]].add(n)
seg_spk[r["seg"]] = r.get("speaker")
seg_src[r["seg"]] = src_of.get(r.get("recording"))
for sid, es in seg_ents.items():
who = f"s{seg_spk[sid]}" if seg_spk.get(sid) else None
src = seg_src.get(sid)
es = sorted(es)
for i, e in enumerate(es):
if who in nodes:
edge(who, e, "mentions")
if src in nodes:
edge(src, e, "added")
for f in es[i + 1 :]:
edge(e, f, "mentioned together")
for ln in db.rows("SELECT record::id(in) AS a, record::id(out) AS b FROM same_as"):
Expand Down
Loading
Loading