diff --git a/docs/automations.md b/docs/automations.md index e63585db4..367224abc 100644 --- a/docs/automations.md +++ b/docs/automations.md @@ -14,7 +14,8 @@ merged is called built. | New assets as `asset.added` (the `recordings` source) | built | | New assets by kind: emails received, calendar events, files from storage, uploads, pages saved from the web, imports | built | | Runs that finished or failed, deletions and calendar events that moved | built | -| More kinds: comments, notes, graph changes, sensors, chat | planned | +| Comments, notes written and saved, and changes to the entity graph | built | +| More kinds: sensors, chat, webhooks, system events | planned | | The timeline API (`GET /api/v1/activities`), the MCP tools `timeline` and `event_kinds`, the assistant's `timeline` | built | | Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built | | The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built | @@ -94,6 +95,10 @@ and the paths holding outside text are in `GET /api/v1/activities/kinds` (`event | `asset.failed` | | `jobs` | as `asset.processed`, with `step` (where it failed) and `error` | Processing “Call with the school” failed at transcribe | | `asset.deleted` | | `log` | `type`, `id`, `name`, `media` | Sam deleted “Old minutes” | | `calendar_event.changed` | | `log` | `startTime`, `endTime`, `location`, `calendar`, and `before` (`startTime`, `endTime`) | Family calendar moved “Parents' evening” to 2026-10-21 18:00 | +| `comment.added` | | `comments` | a `Note`: `content`, `inReplyTo` (the comment it answers), `quote` (the passage), `t0`, `t1` (the moment, ms); `target` is the asset | Sam commented on “Call with the school” | +| `note.created` | | `note_pages` | a `Page`: `summary`, `about` (what it's the page of), `place`, `author` (person or assistant) | Assistant wrote the note “St Mary's School” | +| `note.saved` | | `note_versions` | as `note.created`, with `version` (the page as it was before) and `why` | Sam edited the note “School trip plans” | +| `entity.changed` | | `graph` | `op` (created, renamed, merged, split, linked, unlinked, deleted, rolled back, changed), `change` (the graph's own op), `entityType`, `before` (a renamed entity's name), `version`, `count`; tagged with its entities | Sam renamed “St Marys” to “St Mary's School” | A new asset is one activity, of the most specific kind its row tells, decided by its content type and where it came from, never by reading its file: an email (content type email, or from an IMAP or Gmail mailbox), a calendar event @@ -109,6 +114,16 @@ Mary's is understood". A run tried again finishes again, as another activity. A `asset.deleted` where the asset is deleted (once per asset); a calendar event read again that starts at another time is logged as `calendar_event.changed` (once per event and new start), with the times it had before. +A comment on an asset, or a reply on its thread, is `comment.added`, by whoever wrote it, with the asset as its +`target`. A note page is `note.created` when it is made (the assistant's own notes too) and `note.saved` when it +changes: at most one activity per page in each ten minutes of the clock (`note--`), read from the +versions the notebook keeps of each change. A follower hears of a window's first change, as soon as it is made; the +timeline shows its newest. Every change to the entity graph (`graph_event`, [graph history](graph-history.md)) is one +`entity.changed`, about its first entity that is still there, tagged with up to twenty of its entities and in the +first of its namespaces. Its `op` says what happened in a word; its actor is the automation, the assistant or the +agent over MCP it was made through, else the person, the routine, or Lens (analysis). A change that spans namespaces +(a link between two) names entities of each, so only admins see it, with every namespace in `lens.spaces`. + Fields come from what the row keeps, so reading them costs nothing on the ingest path. A message's addressing is kept as it is listed (`recording.mail`: who sent it to whom, its subject, list, mailbox, and Gmail's snippet), an event's times, place and people as it is imported (`recording.event`), and an email's attachments and the start of @@ -142,6 +157,9 @@ A kind belongs to the source named in its `source`, and a source may list more i - `jobs` (`app/domain/events_content.py`) reads `job` by `finished_at`, the id breaking ties, as `asset.processed` and `asset.failed`; their ids carry the time they finished (`processed-job-51-1792081450`), so a run tried again is a new activity. +- `comments` reads `comment` by `created_at`; `note_pages` reads `note_page` by `created_at`; `note_versions` reads + `note_version` by `at` (to the millisecond) and keeps one version for each page and window; `graph` reads + `graph_event` by `at`, the event number breaking ties (numbers are given out in order, so the two agree). - `log` reads `event_log`: activities no row of their own records (a deletion, a webhook call), written by `events.emit(db, kind, *, actor, obj, space, summary, target=None, tag=None, key=None, published=None, spaces=None, audience=None, owner=None, untrusted=None)`, which gives the id (`log-`). Emitting a `key` again changes @@ -376,6 +394,16 @@ vault stops the request until it is unlocked. - Folder scans of a namespace's own folders as `file.added`; uploaded `.ics` files with their event's fields; a snippet for IMAP messages before they are read (it needs the message body). - Per-recipient threading of email conversations; entity tags on new assets once extraction runs at ingest. +- `note.saved` comes as a window's first change is made, while the person may still be typing; an automation can + wait a little before it reads the page (M6). A page made and then written on shows both `note.created` and + `note.saved`. +- A graph change across namespaces is for admins; show it to whoever reads all of its namespaces, as the graph's + history does. +- Every processed asset's analysis is an `entity.changed` (op created or changed); the timeline page could fold them + into the asset's `asset.processed`. +- SurrealDB 3.2.4 gives a range's first rows, not its last, for `ORDER BY ... DESC LIMIT n` over an index, so + sources write a page's time bounds so that no index range is used, and its cursor (`events.after_sql`) reads the + table in full. Paging is right; give it one ordered index read per page once SurrealDB does that right. - Telling a person: buttons on chat approvals (Telegram inline keyboards, Slack Block Kit); email as a `notify` channel; web push; private chats through Matterbridge (it can't tell them from rooms); with several people let in, "me" is whoever last wrote privately, until each chat user has a Lens account of their own; telling which links in a diff --git a/fastapi_backend/app/domain/events_content.py b/fastapi_backend/app/domain/events_content.py index 0eedb69a8..4dad95f0e 100644 --- a/fastapi_backend/app/domain/events_content.py +++ b/fastapi_backend/app/domain/events_content.py @@ -21,6 +21,7 @@ from __future__ import annotations +import datetime as dt import mimetypes import pathlib import re @@ -437,6 +438,43 @@ def _sec(t): return events._parse(t).replace(microsecond=0).isoformat(timespec="seconds") +def _ms(t): + """A time as note versions write it (to the millisecond, +00:00).""" + return events._parse(t).isoformat(timespec="milliseconds") + + +def _bound(col, op, name): + """`col op $name` for a page, newest first. Written so SurrealDB doesn't read it as a range of an index: 3.2.4 + gives a range's first rows, not its last, for ORDER BY ... DESC LIMIT n.""" + return f"({col} {op} ${name} OR {col} = NONE)" + + +def _paging(col, table, before, who, flt, p, at=_sec, many=False): + """The WHERE parts every source's page shares: what `who` may read, before the cursor, the namespace and time + filters (looser bounds; the stream checks each filter exactly again). `many`: the rows list their namespaces in + `spaces`.""" + mine = None if who.get("admin") else (who.get("spaces") or set()) + where = [f"{col} != NONE"] + if mine is not None: + p["sp"] = sorted(mine) + where.append("spaces CONTAINSANY $sp" if many else "space IN $sp") + if before is not None and before[1] is None: + p["pt"] = before[0] + where.append(_bound(col, "<", "pt")) + elif before is not None: + where.append(events.after_sql(col, before, p, "<", table=table)) + if flt.get("spaces") is not None: + p["fs"] = sorted(flt["spaces"]) + where.append("spaces CONTAINSANY $fs" if many else "space IN $fs") + if flt.get("since"): + p["since"] = at(flt["since"]) + where.append(_bound(col, ">", "since")) + if flt.get("until"): + p["until"] = at(flt["until"]) + where.append(_bound(col, "<=", "until")) + return where + + class RecordingsSource(events.Source): """recording rows as asset.added and its children, read by (created_at, id).""" @@ -451,18 +489,7 @@ def read(self, db, cfg, pos, spaces, limit, hint=None): def page(self, db, cfg, before, who, flt, limit): p: dict = {} - where = [events.space_sql(None if who.get("admin") else (who.get("spaces") or set()), p), "created_at != NONE"] - if before is not None: - where.append(events.after_sql("created_at", before, p, "<", table="recording")) - if flt.get("spaces") is not None: - p["fs"] = sorted(flt["spaces"]) - where.append("space IN $fs") - if flt.get("since"): # a looser bound, to the second; the stream checks each filter exactly again - p["since"] = _sec(flt["since"]) - where.append("created_at > $since") - if flt.get("until"): - p["until"] = _sec(flt["until"]) - where.append("created_at <= $until") + where = _paging("created_at", "recording", before, who, flt, p) rows = db.rows( f"SELECT {FIELDS} FROM recording WHERE {' AND '.join(where)} ORDER BY created_at DESC, rid DESC LIMIT {int(limit)}", **p ) @@ -758,22 +785,7 @@ def read(self, db, cfg, pos, spaces, limit, hint=None): def page(self, db, cfg, before, who, flt, limit): p: dict = {} - where = [ - self._statuses(flt.get("kinds"), p), - events.space_sql(None if who.get("admin") else (who.get("spaces") or set()), p), - "finished_at != NONE", - ] - if before is not None: - where.append(events.after_sql("finished_at", before, p, "<", table="job")) - if flt.get("spaces") is not None: - p["fs"] = sorted(flt["spaces"]) - where.append("space IN $fs") - if flt.get("since"): - p["since"] = _sec(flt["since"]) - where.append("finished_at > $since") - if flt.get("until"): - p["until"] = _sec(flt["until"]) - where.append("finished_at <= $until") + where = [self._statuses(flt.get("kinds"), p), *_paging("finished_at", "job", before, who, flt, p)] rows = db.rows( f"SELECT {JOB_FIELDS} FROM job WHERE {' AND '.join(where)} ORDER BY finished_at DESC, rid DESC LIMIT {int(limit)}", **p ) @@ -1013,3 +1025,661 @@ def _sample(aid, kind, typ, source, pos, actor, obj, summary, untrusted, tag=()) ("object.name", "object.location"), ), ) + + +def _and_more(names, most=1): + """“A”, “A” and “B”, or “A” and 3 more.""" + names = [f"“{str(n)[:200]}”" for n in names if n] + if not names: + return "an entity" + if len(names) <= most + 1: + return " and ".join(names) + return f"{', '.join(names[:most])} and {len(names) - most} more" + + +# ---------- comments ---------- +COMMENT_FIELDS = "id AS rid, record::id(id) AS id, recording, space, account, parent, text, t0, t1, quote, created_at" +CONTENT_MAX = 1000 # characters of a comment an activity carries + + +def _comment_activities(db, rows): + """New comments as comment.added: the comment (a Note) on its asset, the target.""" + rows = [r for r in rows if r] + if not rows: + return [] + rids = list({r["recording"] for r in rows if _num(r.get("recording")) is not None}) + assets = {a["id"]: a for a in db.rows(f"SELECT {ASSET_FIELDS} FROM recording WHERE id IN $r", r=[R("recording", i) for i in rids])} + who = actors(db, [(r.get("account"), r.get("space")) for r in rows]) + types = _content_types(db) + out = [] + for r in rows: + try: + published = events.stamp(r["created_at"]) + except (TypeError, ValueError): + continue + rec = assets.get(r.get("recording")) + if rec: + target = _asset(db, rec, types) + else: + target = {"type": "Document", "id": f"lens:recording/{r.get('recording')}", "name": "A deleted asset"} + reply = _num(r.get("parent")) is not None + obj = store.clean( + { + "type": "Note", + "id": f"lens:comment/{r['id']}", + "content": _text(r.get("text"), CONTENT_MAX) or "", + "url": f"/resources/{r.get('recording')}", + "inReplyTo": f"lens:comment/{r['parent']}" if reply else None, + "quote": _text(r.get("quote")), + "t0": _num(r.get("t0")), + "t1": _num(r.get("t1")), + } + ) + actor = who[(r.get("account"), r.get("space"))] + out.append( + events.build( + f"comment-{r['id']}", + "comment.added", + actor=actor, + obj=obj, + target=target, + summary=f"{actor['name']} {'replied on' if reply else 'commented on'} “{target['name'][:200]}”", + space=r.get("space"), + source="comments", + pos=[r["created_at"], r["id"]], + published=published, + ) + ) + return out + + +class CommentsSource(events.Source): + """comment rows (on assets, threaded) as comment.added, read by (created_at, id).""" + + name = "comments" + + def read(self, db, cfg, pos, spaces, limit, hint=None): + p: dict = {} + where = f"{events.after_sql('created_at', pos, p, table='comment')} AND {events.space_sql(spaces, p)}" + rows = db.rows(f"SELECT {COMMENT_FIELDS} FROM comment WHERE {where} ORDER BY created_at, rid LIMIT {int(limit)}", **p) + return _comment_activities(db, rows) + + def page(self, db, cfg, before, who, flt, limit): + p: dict = {} + where = _paging("created_at", "comment", before, who, flt, p) + rows = db.rows( + f"SELECT {COMMENT_FIELDS} FROM comment WHERE {' AND '.join(where)} ORDER BY created_at DESC, rid DESC LIMIT {int(limit)}", **p + ) + return _comment_activities(db, rows) + + def get(self, db, cfg, activity_id): + m = re.match(r"^comment-(\d+)$", activity_id) + if not m: + return None + r = db.one(f"SELECT {COMMENT_FIELDS} FROM $r", r=R("comment", int(m.group(1)))) + got = _comment_activities(db, [r]) if r else [] + return got[0] if got else None + + def get_many(self, db, cfg, ids): + ns = sorted({int(m.group(1)) for m in (re.match(r"^comment-(\d+)$", i) for i in ids) if m}) + if not ns: + return [] + return _comment_activities(db, db.rows(f"SELECT {COMMENT_FIELDS} FROM comment WHERE id IN $rs", rs=[R("comment", n) for n in ns])) + + +events.register_source(CommentsSource()) + + +# ---------- notes ---------- +NOTE_WINDOW = 600 # seconds: a page's versions in one window of the clock this long are one note.saved +NOTE_ROUNDS = 5 # reads of versions a look makes at most, while windows already told of thin them out +VERSION_FIELDS = "id AS rid, record::id(id) AS id, page, space, at, by, author, title, why" +PAGE_FIELDS = "id AS rid, record::id(id) AS id, space, title, summary, about, place, author, created_by, created_at" +NOTE_ID_RX = re.compile(r"^note-(\d+)-(\d+)$") + + +def _slot(t): + return _epoch(t) // NOTE_WINDOW + + +def _slot_time(n): + return _ms(dt.datetime.fromtimestamp(n * NOTE_WINDOW, dt.timezone.utc)) + + +def _note_label(r, by="by"): + return "assistant" if r.get("author") == "assistant" else r.get(by) + + +def _about(about): + m = re.match(r"^([a-z_]+):(\d+)$", str(about or "")) + return f"lens:{m.group(1)}/{m.group(2)}" if m else None + + +def _page_object(pid, pg): + return store.clean( + { + "type": "Page", + "id": f"lens:note_page/{pid}", + "name": _text(pg.get("title"), 200) or "Untitled", + "url": f"/notes/{pid}", + "summary": _text(pg.get("summary")), + "about": _about(pg.get("about")), + "place": pg.get("place"), + } + ) + + +def _slot_of(r): + try: + return _slot(r["at"]) + except (TypeError, ValueError): + return None + + +def _told(db, rows, newest): + """The versions that stand for their page's window: its first (its newest, when paging back), one query for all.""" + slots = {r["id"]: _slot_of(r) for r in rows if r and _num(r.get("page")) is not None} + rows = [r for r in rows if r and slots.get(r["id"]) is not None] + if not rows: + return [] + near: dict = {} + for v in db.rows( + "SELECT page, at, record::id(id) AS id FROM note_version WHERE page IN $p AND at >= $lo AND at < $hi", + p=sorted({r["page"] for r in rows}), + lo=_slot_time(min(slots[r["id"]] for r in rows)), + hi=_slot_time(max(slots[r["id"]] for r in rows) + 1), + ): + if _slot_of(v) is not None: + near.setdefault((v["page"], _slot_of(v)), []).append((str(v["at"]), v["id"])) + pick = max if newest else min + return [r for r in rows if pick(near.get((r["page"], slots[r["id"]]), []) + [(str(r["at"]), r["id"])]) == (str(r["at"]), r["id"])] + + +def _note_activities(db, rows): + """Versions as note.saved: someone (or the assistant) changed the page at the version's time.""" + if not rows: + return [] + pages = { + x["id"]: x + for x in db.rows( + "SELECT record::id(id) AS id, title, summary, about, place FROM note_page WHERE id IN $p", + p=[R("note_page", i) for i in {r["page"] for r in rows}], + ) + } + who = actors(db, [(_note_label(r), r.get("space")) for r in rows]) + out = [] + for r in rows: + try: + published, slot = events.stamp(r["at"]), _slot(r["at"]) + except (TypeError, ValueError): + continue + obj = _page_object(r["page"], pages.get(r["page"]) or {"title": r.get("title")}) + obj.update(store.clean({"version": f"lens:note_version/{r['id']}", "author": r.get("author"), "why": _text(r.get("why"))})) + actor = who[(_note_label(r), r.get("space"))] + out.append( + events.build( + f"note-{r['page']}-{slot}", + "note.saved", + actor=actor, + obj=obj, + summary=f"{actor['name']} edited the note “{obj['name'][:200]}”", + space=r.get("space"), + source="note_versions", + pos=[r["at"], r["id"]], + published=published, + ) + ) + return out + + +class NoteVersionsSource(events.Source): + """note_version rows (a page as it was before each change: notebook.snapshot) as note.saved, read by (at, id): one + activity per page per NOTE_WINDOW, `note--`. Followers hear of a window's first change, the timeline + shows its newest.""" + + name = "note_versions" + + def _rounds(self, db, where, p, cursor, limit, newest): + """Versions from `cursor` (None: the newest), a batch at a time, until `limit` of them stand for their window + or none are left.""" + op, order = ("<", "at DESC, rid DESC") if newest else (">", "at, rid") + batch, out = max(int(limit), 50), [] + for _ in range(NOTE_ROUNDS): + q = dict(p) + parts = where + ([events.after_sql("at", cursor, q, op, table="note_version")] if cursor is not None else []) + rows = db.rows(f"SELECT {VERSION_FIELDS} FROM note_version WHERE {' AND '.join(parts)} ORDER BY {order} LIMIT {batch}", **q) + out += _told(db, rows, newest) + if len(rows) < batch or len(out) >= limit: + break + cursor = [rows[-1]["at"], rows[-1]["id"]] + return out[: int(limit)] + + def read(self, db, cfg, pos, spaces, limit, hint=None): + p: dict = {} + return _note_activities(db, self._rounds(db, [events.space_sql(spaces, p)], p, pos, limit, newest=False)) + + def page(self, db, cfg, before, who, flt, limit): + p: dict = {} + where = _paging("at", None, None, who, flt, p, at=_ms) + return _note_activities(db, self._rounds(db, where, p, before, limit, newest=True)) + + def get(self, db, cfg, activity_id): + m = NOTE_ID_RX.match(activity_id) + if not m: + return None + pid, slot = int(m.group(1)), int(m.group(2)) + r = db.one( + f"SELECT {VERSION_FIELDS} FROM note_version WHERE page = $p AND {_bound('at', '>=', 'lo')} AND {_bound('at', '<', 'hi')} " + "ORDER BY at DESC, rid DESC LIMIT 1", + p=pid, + lo=_slot_time(slot), + hi=_slot_time(slot + 1), + ) + got = _note_activities(db, [r]) if r else [] + return got[0] if got and got[0]["id"] == activity_id else None + + def get_many(self, db, cfg, ids): + want = {(int(m.group(1)), int(m.group(2))) for m in (NOTE_ID_RX.match(i) for i in ids) if m} + if not want: + return [] + rows = db.rows( + f"SELECT {VERSION_FIELDS} FROM note_version WHERE page IN $p AND at >= $lo AND at < $hi", + p=sorted({p for p, _ in want}), + lo=_slot_time(min(n for _, n in want)), + hi=_slot_time(max(n for _, n in want) + 1), + ) + newest: dict = {} + for r in rows: + k = (r.get("page"), _slot_of(r)) + if k in want and (k not in newest or (str(r["at"]), r["id"]) > (str(newest[k]["at"]), newest[k]["id"])): + newest[k] = r + return _note_activities(db, list(newest.values())) + + +def _page_activities(db, rows): + """New pages as note.created, by whoever made them (the assistant, for its own).""" + rows = [r for r in rows if r] + who = actors(db, [(_note_label(r, "created_by"), r.get("space")) for r in rows]) + out = [] + for r in rows: + try: + published = events.stamp(r["created_at"]) + except (TypeError, ValueError): + continue + obj = _page_object(r["id"], r) + if r.get("author"): + obj["author"] = r["author"] + actor = who[(_note_label(r, "created_by"), r.get("space"))] + out.append( + events.build( + f"note-created-{r['id']}", + "note.created", + actor=actor, + obj=obj, + summary=f"{actor['name']} wrote the note “{obj['name'][:200]}”", + space=r.get("space"), + source="note_pages", + pos=[r["created_at"], r["id"]], + published=published, + ) + ) + return out + + +class NotePagesSource(events.Source): + """note_page rows as note.created when they're made, read by (created_at, id).""" + + name = "note_pages" + + def read(self, db, cfg, pos, spaces, limit, hint=None): + p: dict = {} + where = f"{events.after_sql('created_at', pos, p, table='note_page')} AND {events.space_sql(spaces, p)}" + rows = db.rows(f"SELECT {PAGE_FIELDS} FROM note_page WHERE {where} ORDER BY created_at, rid LIMIT {int(limit)}", **p) + return _page_activities(db, rows) + + def page(self, db, cfg, before, who, flt, limit): + p: dict = {} + where = _paging("created_at", "note_page", before, who, flt, p) + rows = db.rows( + f"SELECT {PAGE_FIELDS} FROM note_page WHERE {' AND '.join(where)} ORDER BY created_at DESC, rid DESC LIMIT {int(limit)}", **p + ) + return _page_activities(db, rows) + + def get(self, db, cfg, activity_id): + m = re.match(r"^note-created-(\d+)$", activity_id) + if not m: + return None + r = db.one(f"SELECT {PAGE_FIELDS} FROM $r", r=R("note_page", int(m.group(1)))) + got = _page_activities(db, [r]) if r else [] + return got[0] if got else None + + def get_many(self, db, cfg, ids): + ns = sorted({int(m.group(1)) for m in (re.match(r"^note-created-(\d+)$", i) for i in ids) if m}) + if not ns: + return [] + return _page_activities(db, db.rows(f"SELECT {PAGE_FIELDS} FROM note_page WHERE id IN $rs", rs=[R("note_page", n) for n in ns])) + + +events.register_source(NoteVersionsSource()) +events.register_source(NotePagesSource()) + + +# ---------- the entity graph ---------- +GRAPH_FIELDS = ( + "id AS rid, record::id(id) AS id, at, op, actor, via, spaces, entities, origin, " + "ops[WHERE t = 'entity'].{k, b: b.name, a: a.name, ty: (a ?? b).type} AS named" +) +GRAPH_TAGS = 20 # entities an entity.changed activity is tagged with +CREATED_OPS = {"entity.found", "entity.learned", "entity.builtin"} +GRAPH_OPS = { + "entity.rename": "renamed", + "entity.merge": "merged", + "entity.unmerge": "split", + "link.add": "linked", + "link.remove": "unlinked", + "entity.delete": "deleted", + "graph.rollback": "rolled back", +} +OPS = ("created", "renamed", "merged", "split", "linked", "unlinked", "deleted", "rolled back", "changed") + + +def _graph_op(op, named): + """What a graph event did, in a word: an entity made (found by analysis or defined from a phrase), renamed, merged, + split again, linked across namespaces or unlinked, deleted, rolled back, else changed.""" + if op in CREATED_OPS or (op in ("analysis", "entity.define") and any(n.get("b") is None and n.get("a") for n in named.values())): + return "created" + return GRAPH_OPS.get(op, "changed") + + +def _graph_label(r): + """Who changed the graph, as actors() reads it: an automation (its id as actor or `via`), the assistant or an + agent over MCP (through `via`, for the person who asked), else the actor (an email, `routine:`; analysis and + system are Lens).""" + for v in (r.get("actor"), r.get("via")): + if AUTOMATION_RX.match(str(v or "")): + return v + return r.get("via") if r.get("via") in APPS else r.get("actor") + + +def _graph_summary(op, actor, r, named, ents, name, alive): + who = actor["name"] + if op == "renamed" and ents and named.get(ents[0], {}).get("b"): + n = named[ents[0]] + return f"{who} renamed “{str(n['b'])[:200]}” to “{str(n.get('a') or '')[:200]}”" + if op == "merged" and alive: + gone = [name(e) for e in ents if e not in alive] + return f"{who} merged {_and_more(gone)} into “{str(name(alive[0]))[:200]}”" + if op == "rolled back" and _num((r.get("origin") or {}).get("rollback_to")) is not None: + return f"{who} rolled the graph back to version {r['origin']['rollback_to']}" + verb = {"created": "found", "split": "split", "linked": "linked", "unlinked": "unlinked", "deleted": "deleted"}.get(op, op) + return f"{who} {verb} {_and_more([name(e) for e in ents], 2 if op in ('linked', 'unlinked') else 1)}" + + +def _graph_activities(db, rows): + """Graph events as entity.changed, one per event: about its first entity still there, tagged with its entities, + in the first of its namespaces. One that spans namespaces is for admins, since its names come from each.""" + rows = [r for r in rows if r] + if not rows: + return [] + named_of = {r["id"]: {n["k"]: n for n in r.get("named") or [] if isinstance(n, dict) and _num(n.get("k")) is not None} for r in rows} + ents_of = {r["id"]: [e for e in r.get("entities") or [] if _num(e) is not None] for r in rows} + looked = _lookup(db, "entity", {e for r in rows for e in ents_of[r["id"]][:GRAPH_TAGS] if e not in named_of[r["id"]]}, "name, type") + labels = {r["id"]: (_graph_label(r), ([s for s in r.get("spaces") or [] if _num(s) is not None] or [None])[0]) for r in rows} + who = actors(db, labels.values()) + out = [] + for r in rows: + try: + published = events.stamp(r["at"]) + except (TypeError, ValueError): + continue + named, ents = named_of[r["id"]], ents_of[r["id"]] + + def name(e, named=named): + n = named.get(e) + return (n.get("a") or n.get("b")) if n else (looked.get(e) or {}).get("name") + + op = _graph_op(r.get("op"), named) + alive = [e for e in ents if not (e in named and named[e].get("a") is None)] + first = (alive or ents or [None])[0] + n = named.get(first) or looked.get(first) or {} + obj = { + "type": "Object", + "id": f"lens:entity/{first}" if first is not None else "lens:graph", + "name": _text(name(first), 200) or "an entity", + "url": f"/entities/{first}" if first in alive else None, + "entityType": n.get("ty") or n.get("type"), + "op": op, + "change": r.get("op"), + "version": r["id"], + "count": len(ents), + } + if op == "renamed" and (named.get(first) or {}).get("b"): + obj["before"] = _text(named[first]["b"], 200) + spaces = sorted({s for s in r.get("spaces") or [] if _num(s) is not None}) + actor = who[labels[r["id"]]] + out.append( + events.build( + f"graph-{r['id']}", + "entity.changed", + actor=actor, + obj=store.clean(obj), + summary=_graph_summary(op, actor, r, named, ents, name, alive), + space=spaces[0] if spaces else None, + source="graph", + pos=[r["at"], r["id"]], + published=published, + tag=[{"type": "Mention", "id": f"lens:entity/{e}", "name": _text(name(e), 200)} for e in ents[:GRAPH_TAGS] if name(e)], + spaces=spaces if len(spaces) > 1 else None, + audience="admins" if len(spaces) > 1 else None, + ) + ) + return out + + +class GraphSource(events.Source): + """graph_event rows (graph_history.py: one per change to the entity graph, never changed) as entity.changed, read + by (at, event number); the number gives versions out in order, so the two agree.""" + + name = "graph" + + def read(self, db, cfg, pos, spaces, limit, hint=None): + p: dict = {} + where = [events.after_sql("at", pos, p, table="graph_event")] + if spaces is not None: + p["sp"] = sorted(spaces) + where.append("spaces CONTAINSANY $sp") + rows = db.rows(f"SELECT {GRAPH_FIELDS} FROM graph_event WHERE {' AND '.join(where)} ORDER BY at, rid LIMIT {int(limit)}", **p) + return _graph_activities(db, rows) + + def page(self, db, cfg, before, who, flt, limit): + p: dict = {} + where = _paging("at", "graph_event", before, who, flt, p, many=True) + rows = db.rows( + f"SELECT {GRAPH_FIELDS} FROM graph_event WHERE {' AND '.join(where)} ORDER BY at DESC, rid DESC LIMIT {int(limit)}", **p + ) + return _graph_activities(db, rows) + + def get(self, db, cfg, activity_id): + m = re.match(r"^graph-(\d+)$", activity_id) + if not m: + return None + r = db.one(f"SELECT {GRAPH_FIELDS} FROM $r", r=R("graph_event", int(m.group(1)))) + got = _graph_activities(db, [r]) if r else [] + return got[0] if got else None + + def get_many(self, db, cfg, ids): + ns = sorted({int(m.group(1)) for m in (re.match(r"^graph-(\d+)$", i) for i in ids) if m}) + if not ns: + return [] + return _graph_activities( + db, db.rows(f"SELECT {GRAPH_FIELDS} FROM graph_event WHERE id IN $rs", rs=[R("graph_event", n) for n in ns]) + ) + + +events.register_source(GraphSource()) + + +# ---------- their kinds ---------- +SAM = {"type": "Person", "id": "lens:account/1", "name": "Sam"} +_PAGE = { + "type": _s("Object type", enum=["Page"]), + "id": _s("Note"), + "name": _s("Title"), + "url": _s("Link"), + "summary": _s("Summary"), + "about": _s("About"), + "place": _s("Place", enum=["project", "area", "resource", "archive"]), + "author": _s("Written by", enum=["person", "assistant"]), +} +events.kind( + "comment.added", + label="Comment added", + source="comments", + as_type="Create", + object_type="Note", + fields={ + "object": { + "type": "object", + "properties": { + "type": _s("Object type", enum=["Note"]), + "id": _s("Comment"), + "content": _s("Text"), + "url": _s("Link"), + "inReplyTo": _s("Reply to"), + "quote": _s("Quoted passage"), + "t0": _s("From (ms)", "integer"), + "t1": _s("To (ms)", "integer"), + }, + }, + "target": {"type": "object", "properties": _ASSET}, + }, + untrusted=("object.content", "object.quote", "target.name"), + sample=_sample( + "comment-31", + "comment.added", + "Create", + "comments", + ["2026-10-15T16:24:10+00:00", 31], + SAM, + { + "type": "Note", + "id": "lens:comment/31", + "content": "Can we bring this up with the school on Friday?", + "url": "/resources/8", + "t0": 61000, + "t1": 65000, + }, + "Sam commented on “Call with the school”", + ("object.content", "object.quote", "target.name"), + ) + | {"target": {"type": "Audio", "id": "lens:recording/8", "name": "Call with the school", "url": "/resources/8", "media": "audio"}}, +) +events.kind( + "note.saved", + label="Note saved", + source="note_versions", + as_type="Update", + object_type="Page", + fields={ + "object": { + "type": "object", + "properties": {**_PAGE, "version": _s("Version kept from before"), "why": _s("Why")}, + } + }, + untrusted=("object.name", "object.summary"), + sample=_sample( + "note-5-2986802", + "note.saved", + "Update", + "note_versions", + ["2026-10-15T16:24:10.120+00:00", 40], + SAM, + { + "type": "Page", + "id": "lens:note_page/5", + "name": "School trip plans", + "url": "/notes/5", + "summary": "What to pack and who drives", + "place": "project", + "version": "lens:note_version/40", + "author": "person", + }, + "Sam edited the note “School trip plans”", + ("object.name", "object.summary"), + ), +) +events.kind( + "note.created", + label="Note written", + source="note_pages", + as_type="Create", + object_type="Page", + fields={"object": {"type": "object", "properties": _PAGE}}, + untrusted=("object.name", "object.summary"), + sample=_sample( + "note-created-6", + "note.created", + "Create", + "note_pages", + ["2026-10-15T16:24:10+00:00", 6], + APPS["assistant"], + { + "type": "Page", + "id": "lens:note_page/6", + "name": "St Mary's School", + "url": "/notes/6", + "about": "lens:entity/44", + "author": "assistant", + }, + "Assistant wrote the note “St Mary's School”", + ("object.name", "object.summary"), + ), +) +events.kind( + "entity.changed", + label="Entity changed", + source="graph", + as_type="Update", + object_type="Object", + fields={ + "object": { + "type": "object", + "properties": { + "type": _s("Object type", enum=["Object"]), + "id": _s("Entity"), + "name": _s("Name"), + "url": _s("Link"), + "entityType": _s("Entity type"), + "op": _s("What happened", enum=list(OPS)), + "change": _s("Graph change"), + "version": _s("Graph version", "integer"), + "count": _s("Entities changed", "integer"), + "before": _s("Name before"), + }, + } + }, + untrusted=("object.name", "object.before"), + sample=_sample( + "graph-88", + "entity.changed", + "Update", + "graph", + ["2026-10-15T16:24:10+00:00", 88], + SAM, + { + "type": "Object", + "id": "lens:entity/44", + "name": "St Mary's School", + "url": "/entities/44", + "entityType": "organization", + "op": "renamed", + "change": "entity.rename", + "version": 88, + "count": 1, + "before": "St Marys", + }, + "Sam renamed “St Marys” to “St Mary's School”", + ("object.name", "object.before"), + [SCHOOL], + ), +) diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index f954d37e9..c1500e0db 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -1351,6 +1351,7 @@ def close(self): "DEFINE INDEX IF NOT EXISTS graph_change_pair ON graph_change FIELDS pair", # the entity graph's history (graph_history.py): one append-only event per change, and named versions "DEFINE TABLE IF NOT EXISTS graph_event SCHEMALESS", + "DEFINE INDEX IF NOT EXISTS graph_event_at ON graph_event FIELDS at", # read as activities (events_content.py) "DEFINE TABLE IF NOT EXISTS graph_tag SCHEMALESS", "DEFINE TABLE IF NOT EXISTS graph_checkpoint SCHEMALESS", "DEFINE TABLE IF NOT EXISTS graph_checkpoint_part SCHEMALESS", @@ -1435,6 +1436,7 @@ def close(self): "DEFINE INDEX IF NOT EXISTS note_page_parent ON note_page FIELDS parent", "DEFINE INDEX IF NOT EXISTS note_page_about ON note_page FIELDS about_key UNIQUE", "DEFINE INDEX IF NOT EXISTS note_page_refine ON note_page FIELDS refine_pending", + "DEFINE INDEX IF NOT EXISTS note_page_created ON note_page FIELDS created_at", # the links in a page's body (@ and # mentions), for backlinks and the graph "DEFINE TABLE IF NOT EXISTS note_link SCHEMALESS", "DEFINE INDEX IF NOT EXISTS note_link_page ON note_link FIELDS page", @@ -1442,6 +1444,7 @@ def close(self): # what a page was before each change (notebook.snapshot): its history, to look back at and restore "DEFINE TABLE IF NOT EXISTS note_version SCHEMALESS", "DEFINE INDEX IF NOT EXISTS note_version_page ON note_version FIELDS page", + "DEFINE INDEX IF NOT EXISTS note_version_at ON note_version FIELDS at", # deleted pages, under their own id, until they're put back or TRASH_DAYS pass (notebook.delete) "DEFINE TABLE IF NOT EXISTS note_trash SCHEMALESS", "DEFINE INDEX IF NOT EXISTS note_trash_space ON note_trash FIELDS space", @@ -1452,6 +1455,7 @@ def close(self): "DEFINE TABLE IF NOT EXISTS comment SCHEMALESS", "DEFINE INDEX IF NOT EXISTS comment_rec ON comment FIELDS recording", "DEFINE INDEX IF NOT EXISTS comment_parent ON comment FIELDS parent", + "DEFINE INDEX IF NOT EXISTS comment_created ON comment FIELDS created_at", # passages of resources marked in colour by their editors (app/domain/highlights.py) "DEFINE TABLE IF NOT EXISTS highlight SCHEMALESS", "DEFINE INDEX IF NOT EXISTS highlight_rec ON highlight FIELDS recording", diff --git a/fastapi_backend/tests/api/test_activities.py b/fastapi_backend/tests/api/test_activities.py index feb6bf59e..5757d0bad 100644 --- a/fastapi_backend/tests/api/test_activities.py +++ b/fastapi_backend/tests/api/test_activities.py @@ -11,6 +11,7 @@ from tests.helpers import login, make_user, quiet, seed URL = "/api/v1/activities" +ASSETS = {"kinds": "asset.added"} # analysis changes the graph too (entity.changed); these tests read the assets def _env(client, db, cfg, folder): @@ -30,7 +31,7 @@ def test_the_timeline_lists_new_assets_newest_first(client, db, cfg, folder): env = _env(client, db, cfg, folder) pods1, pods2, call = env["ids"] assert client.get(URL).status_code == 401 - page = client.get(URL, headers=env["hr"]).json() + page = client.get(URL, params=ASSETS, headers=env["hr"]).json() assert [a["id"] for a in page["items"]] == [call, pods2, pods1] and page["next"] is None first = page["items"][0] # transcripts imported from the command line are imports (a child of asset.added), by Lens @@ -39,15 +40,15 @@ def test_the_timeline_lists_new_assets_newest_first(client, db, cfg, folder): assert first["object"]["id"] == "lens:recording/" + call.split("-")[-1] and first["lens"]["untrusted"] == ["object.name"] assert first["summary"].startswith("Lens imported “") and sorted(page["namespaces"].values()) == ["calls", "pods"] # each person sees their namespaces' alone - assert [a["id"] for a in client.get(URL, headers=env["hv"]).json()["items"]] == [pods2, pods1] - mine = client.get(URL, headers=env["hc"]).json() + assert [a["id"] for a in client.get(URL, params=ASSETS, headers=env["hv"]).json()["items"]] == [pods2, pods1] + mine = client.get(URL, params=ASSETS, headers=env["hc"]).json() assert [a["id"] for a in mine["items"]] == [call] and list(mine["namespaces"].values()) == ["calls"] # a page at a time - one = client.get(URL, params={"limit": 1}, headers=env["hr"]).json() - two = client.get(URL, params={"limit": 1, "before": one["next"]}, headers=env["hr"]).json() + one = client.get(URL, params={**ASSETS, "limit": 1}, headers=env["hr"]).json() + two = client.get(URL, params={**ASSETS, "limit": 1, "before": one["next"]}, headers=env["hr"]).json() assert [a["id"] for a in one["items"] + two["items"]] == [call, pods2] and two["next"] # filters - assert [a["id"] for a in client.get(URL, params={"namespace": "pods"}, headers=env["hr"]).json()["items"]] == [pods2, pods1] + assert [a["id"] for a in client.get(URL, params={**ASSETS, "namespace": "pods"}, headers=env["hr"]).json()["items"]] == [pods2, pods1] assert client.get(URL, params={"kinds": "asset.added"}, headers=env["hc"]).json()["items"][0]["id"] == call title = first["object"]["name"] where = json.dumps({"all": [{"path": "object.name", "op": "equals", "value": title}]}) @@ -75,8 +76,8 @@ def test_agents_read_the_timeline_too(client, db, cfg, folder): env = _env(client, db, cfg, folder) pods1, pods2, call = env["ids"] - got = tool(client, env["hv"], "timeline") - assert got == client.get(URL, params={"limit": 20}, headers=env["hv"]).json() # the same as the API + got = tool(client, env["hv"], "timeline", kinds=["asset.added"]) + assert got == client.get(URL, params={**ASSETS, "limit": 20}, headers=env["hv"]).json() # the same as the API assert [a["id"] for a in got["items"]] == [pods2, pods1] assert [a["id"] for a in tool(client, env["hr"], "timeline", namespace="calls", kinds=["asset.added"])["items"]] == [call] assert "any needs at least one condition" in tool_error(client, env["hr"], "timeline", where='{"any": []}') @@ -85,7 +86,7 @@ def test_agents_read_the_timeline_too(client, db, cfg, folder): pods = {store.ns_id(db, "pods")} box = ai_tools.Toolbox(db, cfg, {"id": 99, "email": "v"}, pods, set(), {}, None) assert "timeline" in {s["function"]["name"] for s in box.specs()} and box.untrusted("timeline") - out, summary = box.call("timeline", {}) + out, summary = box.call("timeline", {"kinds": ["asset.added"]}) assert [a["id"] for a in json.loads(out)["activities"]] == [pods2, pods1] and summary == "Read 2 activities from the timeline" assert "isn't in this conversation" in json.loads(box.call("timeline", {"namespace": "calls"})[0])["error"] diff --git a/fastapi_backend/tests/api/test_timeline_marks.py b/fastapi_backend/tests/api/test_timeline_marks.py index 29471f0eb..dd46a31b0 100644 --- a/fastapi_backend/tests/api/test_timeline_marks.py +++ b/fastapi_backend/tests/api/test_timeline_marks.py @@ -10,6 +10,7 @@ from tests.helpers import login, make_user, seed URL = "/api/v1/activities" +ASSETS = {"kinds": "asset.added"} # analysis changes the graph too (entity.changed); these tests mark the assets def _env(client, db, cfg, folder): @@ -39,18 +40,18 @@ def test_marks_are_set_cleared_and_filter_the_timeline(client, db, cfg, folder): page = client.get(URL, headers=h).json() assert list(page["marks"]) == [pods2] and page["marks"][pods2]["reviewed"] is True # filters, joined to the page: reviewed, unreviewed, important, hidden - assert _ids(client.get(URL, params={"reviewed": "true"}, headers=h).json()) == [pods2] - assert _ids(client.get(URL, params={"reviewed": "false"}, headers=h).json()) == [call, pods1] + assert _ids(client.get(URL, params={**ASSETS, "reviewed": "true"}, headers=h).json()) == [pods2] + assert _ids(client.get(URL, params={**ASSETS, "reviewed": "false"}, headers=h).json()) == [call, pods1] # a filtered page reads on past what the filter leaves out, so a page of one isn't empty - one = client.get(URL, params={"reviewed": "false", "limit": 1}, headers=h).json() - two = client.get(URL, params={"reviewed": "false", "limit": 1, "before": one["next"]}, headers=h).json() + one = client.get(URL, params={**ASSETS, "reviewed": "false", "limit": 1}, headers=h).json() + two = client.get(URL, params={**ASSETS, "reviewed": "false", "limit": 1, "before": one["next"]}, headers=h).json() assert _ids(one) + _ids(two) == [call, pods1] # marks left out stay as they are assert client.post(f"{URL}/{pods2}/mark", json={"important": True}, headers=h).json()["reviewed"] is True - assert _ids(client.get(URL, params={"important": "true", "reviewed": "true"}, headers=h).json()) == [pods2] + assert _ids(client.get(URL, params={**ASSETS, "important": "true", "reviewed": "true"}, headers=h).json()) == [pods2] assert client.post(f"{URL}/{call}/mark", json={"hidden": True}, headers=h).status_code == 200 - assert _ids(client.get(URL, params={"hidden": "false"}, headers=h).json()) == [pods2, pods1] - assert _ids(client.get(URL, params={"hidden": "true"}, headers=h).json()) == [call] + assert _ids(client.get(URL, params={**ASSETS, "hidden": "false"}, headers=h).json()) == [pods2, pods1] + assert _ids(client.get(URL, params={**ASSETS, "hidden": "true"}, headers=h).json()) == [call] # cleared: a mark with nothing set is gone assert client.post(f"{URL}/{call}/mark", json={"hidden": False}, headers=h).json() == { "reviewed": False, @@ -77,11 +78,11 @@ def test_many_at_once_and_only_what_you_can_see(client, db, cfg, folder): # the calls asset isn't the viewer's to see: it is unknown to them, like one there is none of assert sorted(out["marks"]) == [pods1, pods2] and out["unknown"] == [call, "log-77"] assert all(m["reviewed"] for m in out["marks"].values()) - assert _ids(client.get(URL, params={"reviewed": "false"}, headers=env["hv"]).json()) == [] + assert _ids(client.get(URL, params={**ASSETS, "reviewed": "false"}, headers=env["hv"]).json()) == [] assert client.post(f"{URL}/{call}/mark", json={"reviewed": True}, headers=env["hv"]).status_code == 404 # marks are each person's own: the admin sees none of the viewer's, and their own don't touch the viewer's assert client.get(URL, headers=env["hr"]).json()["marks"] == {} - assert _ids(client.get(URL, params={"reviewed": "false"}, headers=env["hr"]).json()) == [call, pods2, pods1] + assert _ids(client.get(URL, params={**ASSETS, "reviewed": "false"}, headers=env["hr"]).json()) == [call, pods2, pods1] client.post(f"{URL}/mark", json={"ids": [pods2], "reviewed": False, "important": True}, headers=env["hr"]) assert client.get(URL, headers=env["hv"]).json()["marks"][pods2] == {**out["marks"][pods2]} assert client.get(URL, headers=env["hr"]).json()["marks"][pods2]["important"] is True @@ -117,7 +118,7 @@ def test_agents_mark_and_read_what_they_have_not_reviewed(client, db, cfg, folde pods1, pods2, call = env["ids"] got = tool(client, env["hv"], "mark_activities", ids=[pods1, call], reviewed=True) assert list(got["marks"]) == [pods1] and got["unknown"] == [call] - assert [a["id"] for a in tool(client, env["hv"], "timeline", unreviewed=True)["items"]] == [pods2] + assert [a["id"] for a in tool(client, env["hv"], "timeline", unreviewed=True, kinds=["asset.added"])["items"]] == [pods2] assert tool(client, env["hv"], "timeline")["marks"][pods1]["reviewed"] is True assert "say which marks" in tool_error(client, env["hv"], "mark_activities", ids=[pods1]) assert "ids is required" in tool_error(client, env["hv"], "mark_activities", reviewed=True) @@ -129,13 +130,13 @@ def test_agents_mark_and_read_what_they_have_not_reviewed(client, db, cfg, folde assert summary == "Marked 1 activity reviewed" and json.loads(out)["marks"][pods2]["reviewed"] is True assert not box.outside # what it gives back is Lens's own word box.call("mark_activities", {"ids": [pods2], "reviewed": False}) - out, summary = box.call("timeline", {"unreviewed": True}) + out, summary = box.call("timeline", {"unreviewed": True, "kinds": ["asset.added"]}) assert [a["id"] for a in json.loads(out)["activities"]] == [pods2] # the timeline's titles are text from outside Lens: marks that bring things to the person's sight act at once... assert box.wary() out, summary = box.call("mark_activities", {"ids": [pods2], "important": True}) assert summary == "Marked 1 activity important" and json.loads(out)["marks"][pods2]["important"] is True - listed = json.loads(box.call("timeline", {})[0])["activities"] + listed = json.loads(box.call("timeline", {"kinds": ["asset.added"]})[0])["activities"] assert {a["id"]: a.get("marked") for a in listed} == {pods2: ["important"], pods1: ["reviewed"]} # ...and those that take things out of it ask first out, summary = box.call("mark_activities", {"ids": [pods2], "hidden": True}) diff --git a/fastapi_backend/tests/domain/test_events_content.py b/fastapi_backend/tests/domain/test_events_content.py index 12f5820a0..65384dc58 100644 --- a/fastapi_backend/tests/domain/test_events_content.py +++ b/fastapi_backend/tests/domain/test_events_content.py @@ -11,21 +11,25 @@ import pytest from app.domain import ( + comments, convert, deletion, + entities, events, events_content, feeds, gmail, + graph_history, import_hooks, ingest, jobs, + notebook, sources, store, uploads, webcapture, ) -from tests.helpers import PODS1, drain, make_user, write_wav +from tests.helpers import PODS1, drain, make_user, seed, write_wav R = store.R ADMIN = events.viewer(1, None, admin=True) @@ -457,3 +461,179 @@ def test_an_event_that_moves_is_calendar_event_changed(db, cfg, monkeypatch): new_ = db.one("SELECT event FROM $r", r=R("recording", rid))["event"] assert events_content.calendar_moved(db, rid, pods, src, "Parents' evening", old, new_) == a["id"] assert len(db.values("SELECT VALUE id FROM event_log WHERE kind = 'calendar_event.changed'")) == 1 + + +def test_comments_are_comment_added_on_their_asset(db, cfg): + sam = make_user(db, "sam@example.org", "sam password 1") + pods = store.ns_id(db, "pods") + rid = ingest.import_text(db, cfg, "pods", "[00:00] Sam: Remember the forms.", title="Call with the school") + start(db, cfg, "test.comments") + cid = comments.create(db, rid, pods, sam, "Can we bring this up on Friday?", t0=0, t1=1500, quote="Remember the forms.") + reply = comments.create(db, rid, pods, sam, "Yes, I'll ask.", parent=cid) + got = [a for a in new(db, cfg, "test.comments") if a["kind"] == "comment.added"] + assert [a["id"] for a in got] == [f"comment-{cid}", f"comment-{reply}"] + a, b = got + assert a["type"] == "Create" and a["space"] == pods and a["actor"] == {"type": "Person", "id": f"lens:account/{sam}", "name": "sam"} + assert a["object"] == { + "type": "Note", + "id": f"lens:comment/{cid}", + "content": "Can we bring this up on Friday?", + "url": f"/resources/{rid}", + "quote": "Remember the forms.", + "t0": 0, + "t1": 1500, + } + assert a["target"]["id"] == f"lens:recording/{rid}" and a["target"]["name"] == "Call with the school" + assert a["summary"] == "sam commented on “Call with the school”" and b["summary"] == "sam replied on “Call with the school”" + assert b["object"]["inReplyTo"] == f"lens:comment/{cid}" + assert {"object.content", "object.quote", "target.name"} <= set(a["lens"]["untrusted"]) + assert events.get(db, cfg, a["id"], ADMIN) == a and events.get_many(db, cfg, [a["id"], b["id"]], ADMIN) == {a["id"]: a, b["id"]: b} + page = events.page(db, cfg, ADMIN, flt={"kinds": ["comment.added"]})["items"] + assert [x["id"] for x in page] == [b["id"], a["id"]] + assert events.page(db, cfg, events.viewer(9, {store.ns_id(db, "calls")}), flt={"kinds": ["comment.added"]})["items"] == [] + + +def test_notes_written_and_saved_once_a_window(db, cfg): + sam = make_user(db, "sam@example.org", "sam password 1") + ann = make_user(db, "ann@example.org", "ann password 1") + pods = store.ns_id(db, "pods") + start(db, cfg, "test.notes") + pid = notebook.create(db, pods, sam, "School trip plans", "Pack lunches.", summary="What to pack") + mine = notebook.create(db, pods, sam, "St Mary's School", "Notes on the school.", author="assistant") + notebook.update(db, pid, sam, body="Pack lunches and coats.") + notebook.update(db, pid, sam, body="Pack lunches, coats and hats.") # one sitting: one version + notebook.update(db, pid, ann, body="Pack lunches, coats, hats and boots.") + got = new(db, cfg, "test.notes") + made = [a for a in got if a["kind"] == "note.created"] + assert [a["id"] for a in made] == [f"note-created-{pid}", f"note-created-{mine}"] + assert made[0]["actor"]["id"] == f"lens:account/{sam}" and made[0]["summary"] == "sam wrote the note “School trip plans”" + assert made[0]["object"] == { + "type": "Page", + "id": f"lens:note_page/{pid}", + "name": "School trip plans", + "url": f"/notes/{pid}", + "summary": "What to pack", + "author": "person", + } + assert made[1]["actor"] == events_content.APPS["assistant"] and made[1]["object"]["author"] == "assistant" + versions = db.rows("SELECT record::id(id) AS id, at, by FROM note_version WHERE page = $p ORDER BY at", p=pid) + assert [v["by"] for v in versions] == [sam, ann] + slots = sorted({events_content._slot(v["at"]) for v in versions}) + saved = [a for a in got if a["kind"] == "note.saved"] + assert [a["id"] for a in saved] == [f"note-{pid}-{n}" for n in slots] # the first change in each window + assert saved[0]["actor"]["id"] == f"lens:account/{sam}" and saved[0]["summary"] == "sam edited the note “School trip plans”" + assert saved[0]["object"]["version"] == f"lens:note_version/{versions[0]['id']}" and saved[0]["type"] == "Update" + assert "object.name" in saved[0]["lens"]["untrusted"] + # the timeline shows each window once, at its newest change; a viewer of another namespace sees none of it + page = events.page(db, cfg, ADMIN, flt={"kinds": ["note.saved"]})["items"] + assert [x["id"] for x in page] == [f"note-{pid}-{n}" for n in reversed(slots)] + assert page[0]["actor"]["id"] == f"lens:account/{ann}" + assert events.get(db, cfg, page[0]["id"], ADMIN) == page[0] + both = events.get_many(db, cfg, [x["id"] for x in page] + [made[1]["id"]], ADMIN) + assert both == {**{x["id"]: x for x in page}, made[1]["id"]: made[1]} + calls = events.viewer(9, {store.ns_id(db, "calls")}) + assert events.page(db, cfg, calls, flt={"kinds": ["note.saved", "note.created"]})["items"] == [] + + +def test_many_versions_in_a_window_page_as_one(db, cfg): + pods = store.ns_id(db, "pods") + sam = make_user(db, "sam@example.org", "sam password 1") + a = notebook.create(db, pods, sam, "Busy page", "x") + b = notebook.create(db, pods, sam, "Quiet page", "y") + later = events_content._slot(store.now()) + 3 # windows to come, so a follower started now reads them + start(db, cfg, "test.windows") + + def version(pid, slot, sec, by, author="person"): + vid = notebook.snapshot(db, notebook.get(db, pid) | {"body": "z"}, by, author, why="test") + at = dt.datetime.fromtimestamp(slot * events_content.NOTE_WINDOW + sec, dt.UTC).isoformat(timespec="milliseconds") + db.q("UPDATE $r SET at = $t", r=R("note_version", vid), t=at) + + version(b, later, 5, sam) + for i in range(99): # more than a read's batch, all in one window + version(a, later + 1, 1 + i, sam if i < 98 else None, "person" if i < 98 else "assistant") + got = [x for x in new(db, cfg, "test.windows") if x["kind"] == "note.saved"] + assert [x["id"] for x in got] == [f"note-{b}-{later}", f"note-{a}-{later + 1}"] + assert got[1]["actor"]["id"] == f"lens:account/{sam}" # a follower hears of the window's first change + assert [x for x in new(db, cfg, "test.windows") if x["kind"] == "note.saved"] == [] + ids, nxt, seen = [], None, 0 + while True: + pg = events.page(db, cfg, ADMIN, before=nxt, flt={"kinds": ["note.saved"]}, limit=1) + ids += [x["id"] for x in pg["items"]] + seen += 1 + nxt = pg["next"] + if not nxt or seen > 5: + break + assert ids == [f"note-{a}-{later + 1}", f"note-{b}-{later}"] + newest = events.get(db, cfg, f"note-{a}-{later + 1}", ADMIN) + assert newest["actor"] == events_content.APPS["assistant"] and newest["summary"] == "Assistant edited the note “Busy page”" + + +def test_graph_changes_are_entity_changed(db, cfg, folder): + sam = make_user(db, "sam@example.org", "sam password 1") + pods, calls = store.ns_id(db, "pods"), store.ns_id(db, "calls") + start(db, cfg, "test.graph") + seed(db, cfg, folder) + found = [a for a in new(db, cfg, "test.graph") if a["kind"] == "entity.changed"] + assert found and all(a["actor"] == events.LENS and a["object"]["op"] == "created" for a in found) + assert all(a["object"]["change"] == "analysis" and a["tag"] and len(a["tag"]) <= 20 for a in found) + assert found[0]["summary"].startswith("Lens found “") and found[0]["space"] in (pods, calls) + ents = db.rows("SELECT record::id(id) AS id, name, space FROM entity ORDER BY id") + keep = next(e for e in ents if e["space"] == calls and e["name"] == "Dyno Therapeutics") + other = next(e for e in ents if e["space"] == calls and e["name"] == "Alice") + there = next(e for e in ents if e["space"] == pods) + with graph_history.acting(actor="sam@example.org", via="web"): + entities.rename(db, keep["id"], "Dyno") + with graph_history.acting(actor="sam@example.org", via="assistant"): + entities.merge(db, keep["id"], [other["id"]]) + with graph_history.acting(actor="sam@example.org", via="automation:7"): + entities.link(db, keep["id"], there["id"]) + got = [a for a in new(db, cfg, "test.graph") if a["kind"] == "entity.changed"] + assert [a["object"]["op"] for a in got] == ["renamed", "merged", "linked"] + rn, mg, ln = got + assert rn["actor"] == {"type": "Person", "id": f"lens:account/{sam}", "name": "sam"} and rn["type"] == "Update" + assert ( + rn["object"]["id"] == f"lens:entity/{keep['id']}" + and rn["object"]["name"] == "Dyno" + and rn["object"]["before"] == "Dyno Therapeutics" + ) + assert rn["summary"] == "sam renamed “Dyno Therapeutics” to “Dyno”" and rn["space"] == calls and "spaces" not in rn["lens"] + assert rn["id"] == f"graph-{rn['object']['version']}" and rn["tag"][0]["id"] == f"lens:entity/{keep['id']}" + assert mg["actor"] == events_content.APPS["assistant"] and mg["object"]["id"] == f"lens:entity/{keep['id']}" + assert mg["summary"] == "Assistant merged “Alice” into “Dyno”" + # a change across namespaces names entities of each, so only admins see it + assert ln["actor"] == {"type": "Application", "id": "lens:automation/7", "name": "Automation 7"} + assert ln["lens"]["spaces"] == sorted([pods, calls]) and ln["lens"]["audience"] == "admins" and ln["space"] == min(pods, calls) + assert events.get(db, cfg, ln["id"], ADMIN) == ln + assert events.get(db, cfg, ln["id"], events.viewer(sam, {calls})) is None + assert events.get_many(db, cfg, [rn["id"], ln["id"]], events.viewer(sam, {calls})) == {rn["id"]: rn} + mine = [a["id"] for a in events.page(db, cfg, events.viewer(sam, {calls}), flt={"kinds": ["entity.changed"]}, limit=200)["items"]] + assert mg["id"] in mine and rn["id"] in mine and ln["id"] not in mine + assert all(x["space"] == calls for x in events.page(db, cfg, events.viewer(sam, {calls}), flt={"kinds": ["entity.changed"]})["items"]) + everything = [a["id"] for a in events.page(db, cfg, ADMIN, flt={"kinds": ["entity.changed"]}, limit=3)["items"]] + assert everything == [ln["id"], mg["id"], rn["id"]] + + +def test_a_page_between_two_times_shows_the_newest_of_them(db, cfg): + sam = make_user(db, "sam@example.org", "sam password 1") + pods = store.ns_id(db, "pods") + rid = ingest.import_text(db, cfg, "pods", "[00:00] Sam: Remember the forms.", title="Call with the school") + at = [f"2026-10-01T10:00:{i:02d}+00:00" for i in range(10)] + cids = [] + for t in at: + cids.append(comments.create(db, rid, pods, sam, f"Comment at {t}")) + db.q("UPDATE $r SET created_at = $t", r=R("comment", cids[-1]), t=t) + events.emit( + db, + "asset.deleted", + actor=events.LENS, + obj={"type": "Document", "id": "lens:recording/1", "name": t}, + space=pods, + summary=f"Lens deleted “{t}”", + published=t, + ) + flt = {"since": "2026-10-01T10:00:02.000Z", "until": "2026-10-01T10:00:07.000Z"} + for who in (ADMIN, events.viewer(sam, {pods})): + got = events.page(db, cfg, who, flt={**flt, "kinds": ["comment.added"]}, limit=2)["items"] + assert [a["id"] for a in got] == [f"comment-{cids[7]}", f"comment-{cids[6]}"] + log = events.page(db, cfg, who, flt={**flt, "kinds": ["asset.deleted"]}, limit=2)["items"] + assert [a["object"]["name"] for a in log] == [at[7], at[6]]