From 0c2dd7440ad0a5f23644ed1d9531bd494b9886e1 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 01:50:21 +0000 Subject: [PATCH] feat(workflows): graph nodes that publish to feeds and make episodes Select assets and Select activities pick what came in since the routine last ran (or in a period) in its namespaces; Publish to feed adds it to a feed and Podcast episode makes an episode about it, added to a feed once made. A feed must be in a namespace the run covers, items stay bounded by the feed's readers, and an episode drops sources its namespace couldn't read. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01CDEiKGoyQX8Vmnv9bi5JpD --- CHANGELOG.md | 5 + README.md | 3 +- docs/feeds.md | 11 +- docs/podcasts.md | 4 + docs/processing.md | 28 ++ docs/requirements.md | 1 + fastapi_backend/app/domain/feed_nodes.py | 362 +++++++++++++++++++ fastapi_backend/app/domain/organize.py | 3 +- fastapi_backend/app/domain/podcasts.py | 31 +- fastapi_backend/app/domain/workflows.py | 19 +- fastapi_backend/tests/api/test_feed_nodes.py | 302 ++++++++++++++++ threat-model.md | 2 +- threat-model.yaml | 2 +- 13 files changed, 753 insertions(+), 20 deletions(-) create mode 100644 fastapi_backend/app/domain/feed_nodes.py create mode 100644 fastapi_backend/tests/api/test_feed_nodes.py diff --git a/CHANGELOG.md b/CHANGELOG.md index f2a6c97b8..ab81c7cfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version one stops the old address at once, and the feed asks directories and search engines not to list it. Editors make and fill feeds with `POST /api/v1/feeds` and `POST /api/v1/feeds/{id}/items`; agents use the MCP tools `list_feeds`, `create_feed` and `publish_to_feed` ([Published feeds](docs/feeds.md)). + - Routines fill them: the graph workflow nodes Select assets and Select activities pick what came in since the + routine last ran, Publish to feed adds it to a feed, and Podcast episode makes an episode about it (added to a + feed, listed once it's made). Try on the canvas says what they would do + ([Publishing from the graph](docs/processing.md#publishing-from-the-graph)). + ## 0.14.0 October 17, 2026 {id="0.14.0"} - **Dates read as days, for agents.** With Settings → Analysis → Read dates on (off by default), analysis reads the diff --git a/README.md b/README.md index 0237905d6..b6ae45342 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p templates to start from, a search by their words, a download as Markdown, and Undo and Recently deleted instead of a confirm. Export an asset or a collection as an evidence PDF with everything Lens found about it, or the parts you pick. Publish a namespace's resources, episodes, timeline activities or text as a private RSS feed or a podcast - feed that any feed reader or podcast app follows. ([Podcasts](docs/podcasts.md), + feed that any feed reader or podcast app follows, and let a routine fill it on a schedule with what's new, or make + an episode about the week's new resources. ([Podcasts](docs/podcasts.md), [specs from transcripts](docs/notes.md#specs-from-transcripts), [notes](docs/notes.md), [evidence PDFs](docs/api.md#evidence-pdfs), [published feeds](docs/feeds.md); [checks](docs/requirements.md#learn-from-it)) diff --git a/docs/feeds.md b/docs/feeds.md index eff497aac..72aeda12d 100644 --- a/docs/feeds.md +++ b/docs/feeds.md @@ -1,7 +1,7 @@ # Published feeds (RSS and podcasts) -**Status: in progress.** Feeds, their private addresses, the API and the MCP tools are built. Routine workflow nodes -that pick part of the graph or the timeline and publish it, and the Feeds page in the web app, come next. +**Status: in progress.** Feeds, their private addresses, the API, the MCP tools and the workflow nodes that let a +routine fill a feed or make an episode on a schedule are built. The Feeds page in the web app comes next. A namespace can publish what's in it as a feed that any feed reader or podcast app follows: new resources, podcast episodes, things on the timeline, or text a person or an agent wrote. Each feed has a private address; add it to the @@ -48,6 +48,13 @@ again adds only what's new. A feed lists its newest items (50 by default, up to Feed readers poll; the feed answers `304 Not Modified` when nothing changed (its `ETag`), and notes when a reader last read it (`fetched_at`, at most once a minute), so the web app can show that a podcast app is subscribed. +## From a routine + +A routine fills a feed on a schedule with a graph workflow ([Publishing from the graph](processing.md#publishing-from-the-graph)): +**Select assets** or **Select activities** picks what's new since the routine last ran, in the namespaces it covers, +and **Publish to feed** adds it, or **Podcast episode** makes an episode about it and adds that. Each run adds only +what's new, and what the feed's readers couldn't read is refused. + ## API and MCP | | | diff --git a/docs/podcasts.md b/docs/podcasts.md index 05af4fa70..42b9e347f 100644 --- a/docs/podcasts.md +++ b/docs/podcasts.md @@ -61,6 +61,10 @@ The audio is the asset's (`GET /api/v1/recordings/{id}/audio`), and so are shari use the MCP tools `create_podcast` (resources as `recording_ids`, lines as `":"`) and `get_podcast` ([MCP](mcp.md)). +A routine makes episodes on a schedule with the **Podcast episode** workflow node +([Publishing from the graph](processing.md#publishing-from-the-graph)): an episode about the week's new resources, +added to a [feed](feeds.md) your podcast app follows. + ## The prompts are templates The plan, write and fact-check prompts are prompt templates (Templates in the web app) called **Podcast: plan**, diff --git a/docs/processing.md b/docs/processing.md index 5c1cf5e19..d37c035c7 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -756,3 +756,31 @@ Every change is recorded (`GET /api/v1/graph-changes`): editors of the namespace proposals (dismissing says the two are different, so the pair isn't suggested again) and undo applied changes; an admin can undo everything a run applied at once (`POST /api/v1/routine-runs/{id}/undo`). Merges are undone exactly as from an entity's page. + +### Publishing from the graph + +Graph workflows also publish, so a routine fills a [feed](feeds.md) or makes a [podcast episode](podcasts.md) on a +schedule. Their nodes: + +- **Select assets** passes on the resources in the routine's namespaces (or the *namespaces* named), newest first, + *limit* at most (50, up to 500): those that came in *since* the routine's last run (the default; on its first run, + the past week), or in the past *day*, *week* or *month*, or *all* of them; matching *q* (words in the title), + *topics*, *entities* and *status*. Podcast episodes are left out unless *include episodes* is on, so an episode about + the week's resources isn't a source for the next one. +- **Select activities** passes on the [timeline](timeline.md)'s activities in those namespaces, of the *kinds* named + (with their descendants), *since* the same choices, matching *q* and a *where* match tree, newest first, *limit* at + most (50, up to 200). Only what everyone who reads the namespace sees, never someone's own. +- **Publish to feed** adds what comes in to the *feed*: resources and episodes, activities, or text (a template + node's output becomes a text item, with the node's *title*). What's in the feed already isn't added again, so a + routine that runs again adds only what's new; what the feed's readers couldn't read is refused and said in the run's + log. The feed's namespace is one the routine covers. +- **Podcast episode** makes an episode about the resources that come in (*max sources*, 10, of those with text; none + under *min sources*), with the *title*, *prompt*, *length* and *style* of [Podcasts](podcasts.md), placed in the + *namespace* named, else the *feed*'s, else where an admin's episode goes; a source its readers couldn't read is left + out. With a *feed*, the episode is added to it and listed once it's made. + +A run's result counts what was `published` and the `episodes` made, and its log says what each node did. Try on the +canvas runs a graph workflow over the namespaces you pick and says what each node would do, without publishing. So +Select assets → Podcast episode (feed *Weekly*, length 10) on a routine that runs every Monday makes an episode about +the week's new resources and puts it in your podcast app; Select activities (kinds `asset.added`) → Publish to feed +puts the timeline in your feed reader. diff --git a/docs/requirements.md b/docs/requirements.md index f1bb4da07..9b96d0264 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -169,6 +169,7 @@ request, to another check that proves the same thing; loosening a requirement or | A note shows the notes it's inside and the notes inside it, to people and agents | [test_notebook.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_notebook.py) `test_path_and_children`, `test_assistant_keeps_notes`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_note_tools`; [note-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/note-page.test.tsx) | | Undo and Recently deleted instead of a confirm | [test_notebook.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_notebook.py) `test_recently_deleted`; [notes-delete.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/notes-delete.test.tsx) | | A namespace publishes a private RSS feed or a podcast feed (Apple Podcasts' tags, audio with byte ranges) of resources, episodes, timeline activities and text, holding only what its readers can already read; through the API and MCP | [test_feeds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_feeds.py) `test_making_filling_and_reading_a_feed`, `test_a_podcast_feed_lists_what_has_audio`, `test_timeline_activities_in_a_feed`, `test_vaults_and_settings`, `test_agents_use_feeds` | +| A routine fills a feed on a schedule, or makes a podcast episode about what's new: graph workflow nodes pick the resources or timeline activities since the last run, add them to a feed, or make an episode about them; Try says what they would do | [test_feed_nodes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_feed_nodes.py) `test_a_routine_publishes_new_resources_to_a_feed`, `test_a_routine_publishes_the_timeline_and_text`, `test_a_routine_makes_an_episode_and_the_feed_lists_it_once_made`, `test_try_says_what_it_would_do_and_bad_settings_are_refused` | | An asset or a collection exported as an evidence PDF, whole or the parts you pick | [test_evidence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_evidence.py) `test_a_collection_as_one_evidence_pdf`, `test_everything_lens_found_goes_in`, `test_picking_the_parts`; [evidence-parts.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/evidence-parts.test.tsx) | ## Automate it diff --git a/fastapi_backend/app/domain/feed_nodes.py b/fastapi_backend/app/domain/feed_nodes.py new file mode 100644 index 000000000..fd309ea1d --- /dev/null +++ b/fastapi_backend/app/domain/feed_nodes.py @@ -0,0 +1,362 @@ +"""Graph workflow nodes that publish: pick resources or timeline activities from the namespaces a routine runs over, +add them to a published feed (syndication.py), or make a podcast episode about them (podcasts.py). A routine running +such a workflow on a schedule is how a feed fills itself: the new resources every morning, a weekly episode, the +week's timeline. + +- `select_assets`: the resources in the run's namespaces (narrowed to `namespaces`), matching `q`, `topics`, + `entities` and `status`, that came in since `since` (what the routine's last run didn't see, by default), newest + first, `limit` at most; episodes left out unless `include_episodes`. Passes on a list of {id, title, summary, ...}. +- `select_activities`: timeline activities of `kinds` in those namespaces since `since`, matching `q` and `where` + (events.check_filter), newest first; only what every reader of the namespace sees, never someone's own. +- `publish_feed`: adds what comes in (resources, activities, or text a template node wrote) to feed `feed`, whose + namespace the run covers. What's in the feed already isn't added twice; what may not show there is refused and said. +- `podcast_episode`: an episode about the resources that come in (`max_sources` at most, those with text), with + `title`, `prompt`, `length` and `style`, placed in `namespace` (else the feed's, else where podcasts.target puts it) + and added to feed `feed`, where it's listed once it's made. + +A dry run (Try on the canvas) says what each node would do and keeps nothing. +""" + +from __future__ import annotations + +from . import store + +R = store.R +OWN_NODES = ("select_assets", "select_activities", "publish_feed", "podcast_episode") +TERMINAL = {"publish_feed", "podcast_episode"} +CONFIG = { + "select_assets": {"namespaces", "q", "topics", "entities", "status", "since", "limit", "include_episodes"}, + "select_activities": {"namespaces", "kinds", "q", "where", "since", "limit"}, + "publish_feed": {"feed", "title"}, + "podcast_episode": {"feed", "namespace", "title", "prompt", "length", "style", "max_sources", "min_sources"}, +} +SINCE = ("last_run", "day", "week", "month", "all") +DAYS = {"day": 1, "week": 7, "month": 30} +LIMITS = {"select_assets": (50, 500), "select_activities": (50, 200)} # (default, most) +MAX_SOURCES = 20 +ASSET_FIELDS = "record::id(id) AS id, title, summary, recorded_at, created_at, duration_ms, status, media, space" +SUMMARY_MAX = 1000 + + +# ---------- settings ---------- +def _names(db, nid, t, value, what): + """Namespace names (or ids) that exist, as a list; ValueError otherwise.""" + if not isinstance(value, list) or not all(isinstance(x, (str, int)) and not isinstance(x, bool) for x in value): + raise ValueError(f"{t} node {nid}: {what} is a list of namespace names") + if db is not None and value: + known = store.space_names(db) + for x in value: + if (x not in known.values()) if isinstance(x, str) else (x not in known): + raise ValueError(f"{t} node {nid}: no namespace {x}") + return value + + +def _ints(nid, t, value, what): + if not isinstance(value, list) or not all(isinstance(x, int) and not isinstance(x, bool) for x in value): + raise ValueError(f"{t} node {nid}: {what} is a list of ids") + + +def _limit(nid, t, cfg): + _, most = LIMITS[t] + lim = cfg.get("limit") + if lim is not None and (not isinstance(lim, int) or isinstance(lim, bool) or not 1 <= lim <= most): + raise ValueError(f"{t} node {nid}: limit is 1 to {most}") + + +def _text(nid, t, cfg, key, most): + v = cfg.get(key) + if v is not None and (not isinstance(v, str) or len(v) > most): + raise ValueError(f"{t} node {nid}: {key} is text, {most} characters at most") + + +def _feed(db, nid, t, cfg, required): + from . import syndication + + fid = cfg.get("feed") + if fid is None: + if required: + raise ValueError(f"{t} node {nid}: choose a feed") + return + if not isinstance(fid, int) or isinstance(fid, bool): + raise ValueError(f"{t} node {nid}: feed is a feed's id") + if db is not None: + try: + syndication.get(db, fid) + except KeyError: + raise ValueError(f"{t} node {nid}: feed {fid} doesn't exist") from None + + +def check_config(db, nid, t, cfg): + """ValueError naming what is wrong with a publishing node's settings.""" + from . import events, library, podcasts + + if cfg.get("since") is not None and cfg["since"] not in SINCE: + raise ValueError(f"{t} node {nid}: since is {', '.join(SINCE)}") + if t in ("select_assets", "select_activities"): + if cfg.get("namespaces") is not None: + _names(db, nid, t, cfg["namespaces"], "namespaces") + _text(nid, t, cfg, "q", 200) + _limit(nid, t, cfg) + if t == "select_assets": + for k in ("topics", "entities"): + if cfg.get(k) is not None: + _ints(nid, t, cfg[k], k) + st = cfg.get("status") + if st is not None and (not isinstance(st, list) or any(s not in library.STATUSES for s in st)): + raise ValueError(f"{t} node {nid}: status is a list of {', '.join(library.STATUSES)}") + if cfg.get("include_episodes") is not None and not isinstance(cfg["include_episodes"], bool): + raise ValueError(f"{t} node {nid}: include_episodes is true or false") + elif t == "select_activities": + flt = {k: cfg[k] for k in ("kinds", "where") if cfg.get(k) is not None} + if flt.get("kinds") is not None and not isinstance(flt["kinds"], list): + raise ValueError(f"{t} node {nid}: kinds is a list of kinds of activity") + try: + events.check_filter(flt) + except ValueError as e: + raise ValueError(f"{t} node {nid}: {e}") from None + elif t == "publish_feed": + _feed(db, nid, t, cfg, required=True) + _text(nid, t, cfg, "title", 200) + elif t == "podcast_episode": + _feed(db, nid, t, cfg, required=False) + if cfg.get("namespace") is not None: + _names(db, nid, t, [cfg["namespace"]], "namespace") + _text(nid, t, cfg, "title", 200) + _text(nid, t, cfg, "prompt", 2000) + n = cfg.get("length") + if n is not None and (not isinstance(n, int) or isinstance(n, bool) or not 1 <= n <= 180): + raise ValueError(f"{t} node {nid}: length is 1 to 180 minutes") + if cfg.get("style") is not None and cfg["style"] not in podcasts.STYLES: + raise ValueError(f"{t} node {nid}: style is one of {', '.join(podcasts.STYLES)}") + most = cfg.get("max_sources") + if most is not None and (not isinstance(most, int) or isinstance(most, bool) or not 1 <= most <= MAX_SOURCES): + raise ValueError(f"{t} node {nid}: max_sources is 1 to {MAX_SOURCES}") + least = cfg.get("min_sources") + if least is not None and (not isinstance(least, int) or isinstance(least, bool) or not 1 <= least <= (most or MAX_SOURCES)): + raise ValueError(f"{t} node {nid}: min_sources is 1 to max_sources") + + +# ---------- running ---------- +def _by(r): + rid = r.origin.get("routine") + return f"routine:{rid}" if rid else r.user + + +def _spaces(r, c, say, name): + """The run's namespaces, narrowed to the node's `namespaces` (one the run doesn't cover is said and skipped).""" + run = {int(s) for s in r.extra.get("spaces") or ()} + want = c.get("namespaces") + if not want: + return sorted(run) + names = store.space_names(r.db) + by_name = {v: k for k, v in names.items()} + out = set() + for n in want: + sid = by_name.get(n) if isinstance(n, str) else n + if sid in run: + out.add(sid) + else: + say(f"{name}: namespace {n} isn't one this run covers; skipped") + return sorted(out) + + +def _since(r, c): + """The time after which things count, as store.now() writes times; None for everything. `last_run` is when the + routine's last run started (on its first run, or with no routine, the past week).""" + from . import events + + s = c.get("since") or "last_run" + if s == "all": + return None + if s == "last_run": + rid = r.origin.get("routine") + if rid: + prev = r.db.values( + "SELECT VALUE started_at FROM routine_run WHERE routine = $r AND status != 'running' ORDER BY started_at DESC LIMIT 1", + r=int(rid), + ) + if prev and prev[0]: + return prev[0] + s = "week" + return events.ago(DAYS[s] * 86400) + + +def select_assets(r, c, say, name): + from . import library + + spaces = _spaces(r, c, say, name) + since = _since(r, c) + if since: + say(f"{name}: what came in since {since}") + if not spaces: + return [] + only = None + if since: + only = r.db.values("SELECT VALUE record::id(id) FROM recording WHERE space IN $s AND created_at > $t", s=spaces, t=since) + if not only: + return [] + cond, p = library.where( + r.db, spaces, q=c.get("q"), status=c.get("status"), topics=c.get("topics"), entities=c.get("entities"), only=only, cfg=r.cfg + ) + if not c.get("include_episodes"): + cond += " AND engine != 'podcast'" + limit = c.get("limit") or LIMITS["select_assets"][0] + rows = r.db.rows(f"SELECT {ASSET_FIELDS} FROM recording WHERE {cond} ORDER BY created_at DESC, id DESC LIMIT {int(limit)}", **p) + names = store.space_names(r.db) + out = [] + for row in rows: + summary = row.get("summary") + out.append( + store.clean( + { + **row, + "summary": summary[:SUMMARY_MAX] if isinstance(summary, str) else None, + "namespace": names.get(row["space"]), + "path": f"/resources/{row['id']}", + } + ) + ) + return out + + +def select_activities(r, c, say, name): + from . import events + + spaces = _spaces(r, c, say, name) + if not spaces: + return [] + flt = store.clean({"kinds": c.get("kinds") or None, "spaces": spaces, "since": _since(r, c), "q": c.get("q"), "where": c.get("where")}) + who = events.viewer(None, spaces, False) + return events.page(r.db, r.cfg, who, flt=flt, limit=c.get("limit") or LIMITS["select_activities"][0])["items"] + + +def _entries(value, title=None): + """What came in, as things to publish (syndication.entry_from), and how many weren't anything a feed lists.""" + from . import syndication + + items = value if isinstance(value, list) else [value] + out, unusable = [], 0 + for v in items: + if isinstance(v, str) and v.strip(): + v = {"title": title, "summary": v} + e = syndication.entry_from(v) + if e is None: + unusable += 1 + else: + out.append(e) + return out, unusable + + +def _feed_covered(r, fid, name): + from . import syndication + + try: + feed = syndication.get(r.db, int(fid)) + except KeyError: + raise ValueError(f"{name}: feed {fid} is gone; choose another") from None + if feed["space"] not in {int(s) for s in r.extra.get("spaces") or ()}: + raise ValueError(f"{name}: the feed {feed['title']!r} is in a namespace this run doesn't cover") + return feed + + +def _publish(r, feed, entries, node, say, name): + from . import syndication + + got = syndication.publish(r.db, r.cfg, feed["id"], entries, origin={**r.origin, "node": node["id"]}, by=_by(r)) + r.totals["published"] = r.totals.get("published", 0) + len(got["added"]) + say(f"{name}: {len(got['added'])} added to {feed['title']!r}, {got['already']} there already, {len(got['refused'])} refused") + for x in got["refused"][:5]: + say(f"{name}: {x['what']}: {x['why']}") + return got + + +def publish_feed(r, node, value, say, name): + c = node["config"] + feed = _feed_covered(r, c["feed"], name) + entries, unusable = _entries(value, c.get("title")) + if unusable: + say(f"{name}: {unusable} of what came in isn't a resource, an activity or text; skipped") + if r.dry: + say(f"{name}: would add {len(entries)} item(s) to {feed['title']!r}") + return {} + _publish(r, feed, entries, node, say, name) + return {} + + +def podcast_episode(r, node, value, say, name): + from . import podcasts + + c = node["config"] + run_spaces = {int(s) for s in r.extra.get("spaces") or ()} + feed = _feed_covered(r, c["feed"], name) if c.get("feed") is not None else None + space = None + if c.get("namespace") is not None: + names = store.space_names(r.db) + space = c["namespace"] if isinstance(c["namespace"], int) else next((k for k, v in names.items() if v == c["namespace"]), None) + if space is None or space not in run_spaces: + raise ValueError(f"{name}: namespace {c['namespace']} isn't one this run covers") + elif feed is not None: + space = feed["space"] + entries, _ = _entries(value) + wanted = list(dict.fromkeys(int(e["recording"]) for e in entries if e.get("recording") is not None)) + rows = r.db.rows("SELECT record::id(id) AS id, space, status FROM recording WHERE id IN $ids", ids=[R("recording", i) for i in wanted]) + by_id = {x["id"]: x for x in rows} + fits, ids = {}, [] + for i in wanted: + row = by_id.get(i) + if not row or row.get("status") in (None, "new", "error"): + continue # nothing to read yet + if space is not None: + if row["space"] not in fits: + fits[row["space"]] = podcasts.fits(r.db, space, {row["space"]}) + if not fits[row["space"]]: + say(f"{name}: resource {i} is in a namespace the episode's readers couldn't read; left out") + continue + ids.append(i) + ids = ids[: c.get("max_sources") or 10] + if len(ids) < (c.get("min_sources") or 1): + say(f"{name}: {len(ids)} resource(s) with text came in; no episode") + return {} + length = c.get("length") or 10 + where = f" in {c['namespace']}" if c.get("namespace") else " in the feed's namespace" if feed else "" + if r.dry: + say( + f"{name}: would make a {length}-minute episode about {len(ids)} resource(s){where}" + + (f" and add it to {feed['title']!r}" if feed else "") + ) + return {} + made = podcasts.create( + r.db, + r.cfg, + {"recordings": ids}, + by=_by(r), + admin=True, + prompt=c.get("prompt"), + length=length, + style=c.get("style"), + title=c.get("title"), + space=space, + ) + r.totals["episodes"] = r.totals.get("episodes", 0) + 1 + say(f"{name}: episode {made['episode']} about {len(ids)} resource(s) queued (job {made['job']})") + if feed is not None: + _publish(r, feed, [{"recording": made["episode"]}], node, say, name) + return {} + + +def run_node(r, f, node, value, vals): + """What a publishing node does: {output port: value}.""" + t, c, name = node["type"], node["config"], node.get("label") or f"{node['type']} {node['id']}" + if t == "select_assets": + out = select_assets(r, c, r.say, name) + r.say(f"{name}: {len(out)} resource(s)") + return {"out": out} + if t == "select_activities": + out = select_activities(r, c, r.say, name) + r.say(f"{name}: {len(out)} activit{'y' if len(out) == 1 else 'ies'}") + return {"out": out} + if t == "publish_feed": + return publish_feed(r, node, value, r.say, name) + if t == "podcast_episode": + return podcast_episode(r, node, value, r.say, name) + return {} diff --git a/fastapi_backend/app/domain/organize.py b/fastapi_backend/app/domain/organize.py index 4f1acaf06..effe098fb 100644 --- a/fastapi_backend/app/domain/organize.py +++ b/fastapi_backend/app/domain/organize.py @@ -438,7 +438,8 @@ def list_changes(db, spaces, status=None, run=None, limit=200): # ---------- running a graph workflow ---------- def run(db, cfg, wid, spaces, version=None, say=print, origin=None, propose_only=False): """Run one version (default: the current one) of a graph workflow over namespaces `spaces`. Returns - ({node id: done | skipped}, {applied, proposed, skipped}).""" + ({node id: done | skipped}, {applied, proposed, skipped}, with `published` and `episodes` when its nodes publish + (feed_nodes.py)).""" from . import flow, workflows w = workflows.get(db, wid, version) diff --git a/fastapi_backend/app/domain/podcasts.py b/fastapi_backend/app/domain/podcasts.py index 00f00c922..729638f17 100644 --- a/fastapi_backend/app/domain/podcasts.py +++ b/fastapi_backend/app/domain/podcasts.py @@ -119,20 +119,27 @@ def _vault(db, sid): return bool((db.one("SELECT vault FROM $r", r=R("data_key", int(sid))) or {}).get("vault")) +def fits(db, sid, source_spaces): + """Whether an episode about sources in `source_spaces` may go in namespace `sid` (None: one not made yet, read by + admins alone): everyone who can read it can read every source's namespace, and no source is in a vault.""" + if any(_vault(db, s) for s in source_spaces): + return False + readers = _readers(db, sid) if sid is not None else set() + return all(readers <= _readers(db, s, collections=False) for s in source_spaces if s != sid) + + def target(db, cfg, source_spaces, roles, admin): """(namespace id, why) for a new episode. The podcasts namespace (podcasts.namespace) when the person may add to - it and everyone who can read it can read every source's namespace, and no source is in a vault; else the source's - own namespace when all sources share one the person may add to. Raises Problem when neither will do.""" + it and it fits (`fits`); else the source's own namespace when all sources share one the person may add to. Raises + Problem when neither will do.""" from . import auth name = (cfg.get("podcasts") or {}).get("namespace") or "podcasts" row = db.one("SELECT record::id(id) AS id FROM space WHERE name = $n LIMIT 1", n=name) sid = row["id"] if row else None can_add = admin or (sid is not None and auth.allows(roles, sid, "editor")) - if can_add and not any(_vault(db, s) for s in source_spaces): - readers = _readers(db, sid) if sid is not None else set() - if all(readers <= _readers(db, s, collections=False) for s in source_spaces if s != sid): - return (sid if sid is not None else store.ns_id(db, name)), "podcasts" + if can_add and fits(db, sid, source_spaces): + return (sid if sid is not None else store.ns_id(db, name)), "podcasts" if len(source_spaces) == 1: (only,) = source_spaces if admin or auth.allows(roles, only, "editor"): @@ -158,11 +165,14 @@ def create( style=None, title=None, voices=None, + space=None, ): """A new episode about what was picked, queued to be made. Returns {episode, job, namespace, placed}. `roles` ({space: role}) and `admin` decide where it may go; `readable` (None: everything) and `also` (recordings - seen through a role on their collection) bound the sources it may read, and are kept for the job. Callers check the person may read every picked recording first.""" + seen through a role on their collection) bound the sources it may read, and are kept for the job. Callers check the person may read every picked recording first. + `space` is where it goes when the caller has decided (a routine's podcast episode node): it must fit (`fits`), + Problem otherwise.""" sel = clean_selection(selection) opts = options(cfg, length, style, prompt, voices) rids = selected_recordings(sel) @@ -172,7 +182,12 @@ def create( also = sorted({int(x) for x in also or ()}) if readable is not None and any(r["space"] not in readable and r["id"] not in also for r in rows): raise KeyError("recording") - sid, placed = target(db, cfg, {r["space"] for r in rows}, roles or {}, admin) + if space is not None: + if not fits(db, int(space), {r["space"] for r in rows}): + raise Problem("an episode about these sources can't go in that namespace: people there couldn't read them all") + sid, placed = int(space), "chosen" + else: + sid, placed = target(db, cfg, {r["space"] for r in rows}, roles or {}, admin) rid = db.next_id("recording") t = store.now() first = next(r["title"] for r in rows if r["id"] == rids[0]) or "Untitled" diff --git a/fastapi_backend/app/domain/workflows.py b/fastapi_backend/app/domain/workflows.py index b4278bac1..2b4509255 100644 --- a/fastapi_backend/app/domain/workflows.py +++ b/fastapi_backend/app/domain/workflows.py @@ -23,7 +23,8 @@ Positions are only for the canvas. A workflow's scope says what it runs on: `recording` (the nodes above, run by pipelines), `graph` (run over -namespaces, usually by a routine, to organise their entities: see organize.py for its nodes) or `event` (what an +namespaces, usually by a routine, to organise their entities, see organize.py for its nodes, or to publish from them +to a feed or as a podcast episode, see feed_nodes.py) or `event` (what an automation does when it runs, from the run context: see automation_nodes.py for its action and decide nodes). """ @@ -37,6 +38,7 @@ automation_nodes, classify as classifymod, decide, + feed_nodes, fields as fieldmod, flow, llm, @@ -51,7 +53,7 @@ R = store.R OWN_NODES = ("llm", "extract_rules", "extract_llm", "classify", "output", "field", "save_entities") -TERMINAL = {"output", "field", "save_entities", "apply_changes"} +TERMINAL = {"output", "field", "save_entities", "apply_changes", *feed_nodes.TERMINAL} OPS = flow.OPS OWN_CONFIG = { "llm": {"template", "version", "model"}, @@ -101,6 +103,8 @@ def validate_graph(db, graph, scope="recording"): def _check_config(db, nid, t, cfg): if t in organize.CONFIG: organize.check_config(nid, t, cfg) + elif t in feed_nodes.CONFIG: + feed_nodes.check_config(db, nid, t, cfg) elif t in ("pick", "condition"): flow._check_primitive(db, KITS["recording"], nid, t, cfg, None, 0, "") elif t == "llm": @@ -494,13 +498,16 @@ def _fit(f, v): KITS = { "recording": flow.Kit( - "recording", OWN_NODES, OWN_CONFIG, _check_config, _run_node, TERMINAL - {"apply_changes"}, + "recording", OWN_NODES, OWN_CONFIG, _check_config, _run_node, TERMINAL - {"apply_changes", *feed_nodes.TERMINAL}, "a workflow needs an output, field or save entities node (or a classify node that acts), or it keeps nothing", keeps=lambda n: n["type"] == "classify" and bool((n.get("config") or {}).get("apply")), ), + # organising the graph (organize.py) and publishing from it (feed_nodes.py), run over namespaces by a routine "graph": flow.Kit( - "graph", organize.OWN_NODES, organize.CONFIG, _check_config, organize.run_node, {"apply_changes"}, - "a graph workflow needs an apply changes node, or it changes nothing", + "graph", organize.OWN_NODES + feed_nodes.OWN_NODES, {**organize.CONFIG, **feed_nodes.CONFIG}, _check_config, + lambda r, f, n, v, vals: (feed_nodes.run_node if n["type"] in feed_nodes.OWN_NODES else organize.run_node)(r, f, n, v, vals), + {"apply_changes", *feed_nodes.TERMINAL}, + "a graph workflow needs an apply changes, publish to feed or podcast episode node, or it changes nothing", ), # an assistant tool drawn on the canvas (tool_nodes.py): only ever a body, from its parameters to what it gives back "tool": flow.Kit( @@ -517,4 +524,4 @@ def _fit(f, v): SCOPES = {s: k.all_types for s, k in KITS.items()} RECORDING_NODES = SCOPES["recording"] NODE_TYPES = RECORDING_NODES + tuple(dict.fromkeys(t for s in ("graph", "tool", "event") for t in SCOPES[s] if t not in RECORDING_NODES)) -CONFIG = {**flow.CONFIG, **OWN_CONFIG, **organize.CONFIG, **tool_nodes.CONFIG, **automation_nodes.CONFIG} +CONFIG = {**flow.CONFIG, **OWN_CONFIG, **organize.CONFIG, **feed_nodes.CONFIG, **tool_nodes.CONFIG, **automation_nodes.CONFIG} diff --git a/fastapi_backend/tests/api/test_feed_nodes.py b/fastapi_backend/tests/api/test_feed_nodes.py new file mode 100644 index 000000000..560d3c549 --- /dev/null +++ b/fastapi_backend/tests/api/test_feed_nodes.py @@ -0,0 +1,302 @@ +"""Graph workflow nodes that publish: a routine picks new resources or timeline activities and adds them to a feed, or +makes a podcast episode about them; Try on the canvas says what they would do, and bad settings are refused.""" + +from __future__ import annotations + +import xml.etree.ElementTree as ET + +import pytest + +from app.domain import events, routines, store +from tests import fake_llm +from tests.helpers import drain, login, make_user, seed + +R = store.R + + +@pytest.fixture +def env(client, db, cfg, folder): + srv, url = fake_llm.start() + cfg["llm"].update(base_url=url, model="fake") + fake_llm.Handler.podcast_script = None + a, b, c = seed(db, cfg, folder) + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + make_user(db, "ca@x.io", "calls password 1", roles={"calls": "editor"}) # so calls has a reader pods' readers aren't + make_user(db, "root@x.io", "root password 1", admin=True) + yield { + "ids": (a, b, c), + "ed": login(client, "ed@x.io", "editor password 1"), + "vi": login(client, "vi@x.io", "viewer password 1"), + "root": login(client, "root@x.io", "root password 1"), + } + srv.shutdown() + + +def _graph(*nodes): + """A straight line of nodes after the input node.""" + ns = [{"id": "in", "type": "input", "config": {}}] + [{"id": f"n{i}", **n} for i, n in enumerate(nodes)] + return {"nodes": ns, "edges": [{"source": ns[i]["id"], "target": ns[i + 1]["id"]} for i in range(len(ns) - 1)]} + + +def _workflow(client, h, name, graph): + r = client.post("/api/v1/workflows", headers=h, json={"name": name, "scope": "graph", "graph": graph}) + assert r.status_code == 200, r.text + return r.json()["id"] + + +def _routine(client, db, cfg, h, wid, namespaces): + r = client.post( + "/api/v1/routines", + headers=h, + json={ + "name": f"Publish {wid}", + "namespaces": [store.ns_id(db, n, create=False) for n in namespaces], + "actions": [{"type": "workflow", "workflow": wid}], + }, + ) + assert r.status_code == 200, r.text + return r.json()["id"] + + +def _run(client, db, cfg, h, rid): + run_id = routines.run(db, cfg, rid, by="test") + run = routines.get_run(db, run_id) + assert run["status"] == "done", run["results"] + return run["results"][0]["result"], run["log"] + + +def _feed(client, h, kind="rss", title="Pods"): + r = client.post("/api/v1/feeds", headers=h, json={"namespace": "pods", "title": title, "kind": kind}) + assert r.status_code == 201, r.text + return r.json() + + +def _items(client, path): + r = client.get(path) + assert r.status_code == 200, r.text + return ET.fromstring(r.content).find("channel").findall("item") + + +def test_a_routine_publishes_new_resources_to_a_feed(client, db, cfg, env): + a, b, c = env["ids"] + h = env["root"] + feed = _feed(client, env["ed"]) + wid = _workflow( + client, + h, + "New to the feed", + _graph( + {"type": "select_assets", "config": {"since": "all", "limit": 10}}, {"type": "publish_feed", "config": {"feed": feed["id"]}} + ), + ) + types = {n["type"]: n for n in client.get("/api/v1/workflows", headers=h).json()["node_types"]} + assert types["publish_feed"]["scopes"] == ["graph"] and types["publish_feed"]["outputs"] == [] + assert types["publish_feed"]["keeps"] and types["podcast_episode"]["keeps"] and not types["select_assets"]["keeps"] + assert types["select_assets"]["outputs"] == ["out"] and "since" in types["select_assets"]["settings"] + + # over pods: both resources there; a second run adds nothing new + rid = _routine(client, db, cfg, h, wid, ["pods"]) + res, log = _run(client, db, cfg, h, rid) + assert res["published"] == 2 and res["nodes"] == 3, res + assert any("2 resource(s)" in line for line in log) and any("2 added to 'Pods'" in line for line in log) + res, log = _run(client, db, cfg, h, rid) + assert res["published"] == 0 and any("2 there already" in line for line in log) + listed = client.get(f"/api/v1/feeds/{feed['id']}/items", headers=env["vi"]).json() + assert {i["recording"] for i in listed} == {a, b} + assert listed[0]["origin"]["routine"] == rid and listed[0]["origin"]["node"] == "n1" and listed[0]["by"] == f"routine:{rid}" + assert {i.findtext("title") for i in _items(client, feed["path"])} == {"ep1", "ep2"} + + # what the feed's readers couldn't read is refused, said in the log, and nothing else is published + wid2 = _workflow( + client, + h, + "Calls to pods", + _graph( + {"type": "select_assets", "config": {"since": "all", "namespaces": ["calls"]}}, + {"type": "publish_feed", "config": {"feed": feed["id"]}}, + ), + ) + rid2 = _routine(client, db, cfg, h, wid2, ["pods", "calls"]) + res, log = _run(client, db, cfg, h, rid2) + assert res["published"] == 0 and any("not everyone who reads the feed can read" in line for line in log), log + assert len(client.get(f"/api/v1/feeds/{feed['id']}/items", headers=env["vi"]).json()) == 2 + + # a routine that doesn't cover the feed's namespace can't publish into it + rid3 = _routine(client, db, cfg, h, wid, ["calls"]) + run = routines.get_run(db, routines.run(db, cfg, rid3, by="test")) + assert run["status"] == "error" and "doesn't cover" in str(run["results"][0]), run + + # since last_run: what came in since the routine last ran + wid3 = _workflow( + client, h, "Fresh", _graph({"type": "select_assets", "config": {}}, {"type": "publish_feed", "config": {"feed": feed["id"]}}) + ) + rid4 = _routine(client, db, cfg, h, wid3, ["pods"]) + res, log = _run(client, db, cfg, h, rid4) # the first run takes the past week, which holds a and b (there already) + assert res["published"] == 0 and any("2 there already" in line for line in log), log + db.q("UPDATE $r SET created_at = $t", r=R("recording", a), t="2020-01-01T00:00:00+00:00") + db.q("DELETE feed_item") + res, log = _run(client, db, cfg, h, rid4) + assert res["published"] == 0 and any("0 resource(s)" in line for line in log), log # nothing since the run just before + db.q( + "UPDATE $r SET created_at = $t", r=R("recording", b), t=events.ago(-60) + ) # a minute ahead: after the run before, whatever the second + res, _ = _run(client, db, cfg, h, rid4) + assert res["published"] == 1 and [i["recording"] for i in client.get(f"/api/v1/feeds/{feed['id']}/items", headers=h).json()] == [b] + + +def test_a_routine_publishes_the_timeline_and_text(client, db, cfg, env): + a, b, _c = env["ids"] + h = env["root"] + feed = _feed(client, env["ed"], title="Pods timeline") + wid = _workflow( + client, + h, + "Timeline to the feed", + _graph( + {"type": "select_activities", "config": {"kinds": ["asset.added"], "since": "all"}}, + {"type": "publish_feed", "config": {"feed": feed["id"]}}, + ), + ) + rid = _routine(client, db, cfg, h, wid, ["pods"]) + res, log = _run(client, db, cfg, h, rid) + assert res["published"] == 2 and any("2 activities" in line for line in log), (res, log) + listed = client.get(f"/api/v1/feeds/{feed['id']}/items", headers=env["vi"]).json() + assert {i["kind"] for i in listed} == {"activity"} and {i["activity"]["kind"] for i in listed} == {"import.received"} + assert all(i["activity"]["object"] and i["categories"] == ["import.received"] for i in listed) + assert _run(client, db, cfg, h, rid)[0]["published"] == 0 + + # text a template node wrote becomes a text item, titled by the node; the same text isn't added twice + wid2 = _workflow( + client, + h, + "Digest", + _graph( + {"type": "select_assets", "config": {"since": "all"}}, + {"type": "template", "config": {"template": "{{ input | length }} new this week"}}, + {"type": "publish_feed", "config": {"feed": feed["id"], "title": "Weekly digest"}}, + ), + ) + rid2 = _routine(client, db, cfg, h, wid2, ["pods"]) + assert _run(client, db, cfg, h, rid2)[0]["published"] == 1 + assert _run(client, db, cfg, h, rid2)[0]["published"] == 0 + text = [i for i in client.get(f"/api/v1/feeds/{feed['id']}/items", headers=h).json() if i["kind"] == "text"] + assert len(text) == 1 and text[0]["title"] == "Weekly digest" and text[0]["summary"] == "2 new this week" + assert {i.findtext("title") for i in _items(client, feed["path"])} >= {"Weekly digest"} + + +def test_a_routine_makes_an_episode_and_the_feed_lists_it_once_made(client, db, cfg, env): + a, b, c = env["ids"] + h = env["root"] + feed = _feed(client, env["ed"], title="Pods weekly") + wid = _workflow( + client, + h, + "Weekly episode", + _graph( + {"type": "select_assets", "config": {"since": "all"}}, + { + "type": "podcast_episode", + "config": {"feed": feed["id"], "length": 5, "style": "recap", "title": "This week in pods", "max_sources": 2}, + }, + ), + ) + rid = _routine(client, db, cfg, h, wid, ["pods", "calls"]) + res, log = _run(client, db, cfg, h, rid) + assert res["episodes"] == 1 and res["published"] == 1, (res, log) + ep = db.one("SELECT recording, space, status, request FROM podcast LIMIT 1") + assert ep["status"] == "queued" and ep["space"] == feed["space"] + assert ( + sorted(ep["request"]["selection"]["recordings"]) == [a, b] and ep["request"]["length"] == 5 + ) # c is in calls: the feed's readers can't read it + rec = db.one("SELECT title, created_by FROM $r", r=R("recording", ep["recording"])) + assert rec["title"] == "This week in pods" and rec["created_by"] == f"routine:{rid}" + listed = client.get(f"/api/v1/feeds/{feed['id']}/items", headers=env["vi"]).json() + assert [(i["kind"], i["recording"]) for i in listed] == [("episode", ep["recording"])] + assert _items(client, feed["path"]) == [] # not until it's made + drain(db, cfg) + assert client.get(f"/api/v1/podcasts/{ep['recording']}", headers=env["vi"]).json()["status"] == "script_only" + items = _items(client, feed["path"]) + # the outline's title replaces the one asked for, as it does for any episode + assert len(items) == 1 and items[0].findtext("title") == "Capsids and shipments" and "Sources:" in items[0].findtext("description") + + # an episode about the new resources only: nothing new, no episode; and the episode isn't a source for the next one + wid2 = _workflow( + client, + h, + "Fresh episode", + _graph({"type": "select_assets", "config": {}}, {"type": "podcast_episode", "config": {"min_sources": 2}}), + ) + rid2 = _routine(client, db, cfg, h, wid2, ["pods"]) + db.q("UPDATE recording SET created_at = $t WHERE id IN $ids", ids=[R("recording", a), R("recording", b)], t="2020-01-01T00:00:00+00:00") + res, log = _run(client, db, cfg, h, rid2) + assert res.get("episodes", 0) == 0 and any("no episode" in line for line in log), log + assert len(db.rows("SELECT id FROM podcast")) == 1 + # placed where the person asked, with sources its readers couldn't read left out + wid3 = _workflow( + client, + h, + "Into calls", + _graph( + {"type": "select_assets", "config": {"since": "all", "namespaces": ["pods"]}}, + {"type": "podcast_episode", "config": {"namespace": "calls"}}, + ), + ) + rid3 = _routine(client, db, cfg, h, wid3, ["pods", "calls"]) + res, log = _run(client, db, cfg, h, rid3) + assert res.get("episodes", 0) == 0 and sum("couldn't read; left out" in line for line in log) == 2, log + + +def test_try_says_what_it_would_do_and_bad_settings_are_refused(client, db, cfg, env): + h = env["root"] + feed = _feed(client, env["ed"]) + graph = _graph( + {"type": "select_assets", "config": {"since": "all"}}, + {"type": "podcast_episode", "config": {"feed": feed["id"], "length": 7}}, + ) + r = client.post("/api/v1/workflows/test", headers=h, json={"scope": "graph", "graph": graph, "namespaces": ["pods"]}) + assert r.status_code == 200, r.text + out = r.json() + assert out["error"] is None and any( + "would make a 7-minute episode about 2 resource(s) in the feed's namespace and add it to 'Pods'" in line for line in out["log"] + ), out["log"] + assert not db.rows("SELECT id FROM podcast") and not db.rows("SELECT id FROM feed_item") + graph = _graph( + {"type": "select_activities", "config": {"since": "all", "kinds": ["asset.added"]}}, + {"type": "publish_feed", "config": {"feed": feed["id"]}}, + ) + out = client.post("/api/v1/workflows/test", headers=h, json={"scope": "graph", "graph": graph, "namespaces": ["pods"]}).json() + assert out["error"] is None and any("would add 2 item(s) to 'Pods'" in line for line in out["log"]), out["log"] + + def bad(graph, want): + r = client.post("/api/v1/workflows", headers=h, json={"name": "x", "scope": "graph", "graph": graph}) + assert r.status_code == 400 and want in r.json()["detail"], r.text + + bad(_graph({"type": "select_assets", "config": {}}), "needs an apply changes, publish to feed or podcast episode node") + bad( + _graph({"type": "select_assets", "config": {"since": "yesterday"}}, {"type": "publish_feed", "config": {"feed": feed["id"]}}), + "since is", + ) + bad( + _graph({"type": "select_assets", "config": {"namespaces": ["nope"]}}, {"type": "publish_feed", "config": {"feed": feed["id"]}}), + "no namespace nope", + ) + bad( + _graph({"type": "select_assets", "config": {"limit": 0}}, {"type": "publish_feed", "config": {"feed": feed["id"]}}), + "limit is 1 to 500", + ) + bad( + _graph( + {"type": "select_activities", "config": {"kinds": ["email.nope"]}}, {"type": "publish_feed", "config": {"feed": feed["id"]}} + ), + "no kind email.nope", + ) + bad(_graph({"type": "publish_feed", "config": {}}), "choose a feed") + bad(_graph({"type": "publish_feed", "config": {"feed": 999}}), "feed 999 doesn't exist") + bad(_graph({"type": "podcast_episode", "config": {"style": "opera"}}), "style is one of") + bad(_graph({"type": "podcast_episode", "config": {"max_sources": 2, "min_sources": 3}}), "min_sources is 1 to max_sources") + # the nodes are for graph workflows only + r = client.post( + "/api/v1/workflows", headers=h, json={"name": "x", "graph": _graph({"type": "publish_feed", "config": {"feed": feed["id"]}})} + ) + assert r.status_code == 400 and "node types are" in r.json()["detail"] diff --git a/threat-model.md b/threat-model.md index 9cb118871..5506abb8f 100644 --- a/threat-model.md +++ b/threat-model.md @@ -527,7 +527,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-NO-INJECTION | Request text does not inject into database queries, a shell or the processing tools. | integrity-bypass | security-critical | *(documented, SECURITY.md "injection into the database queries, the shell")* | Voided by: nothing. Shell: `grep -rnE "shell=True\|os\.system\|os\.popen" fastapi_backend/app` — one hit, a deny-list string in `domain/code_tools.py`. Database binding is per call site: `grep -rnE 'f"[^"]*(SELECT\|UPDATE\|DELETE\|CREATE\|UPSERT)[^"]*\{' fastapi_backend/app/domain fastapi_backend/app/api` — 151 hits; about 35 read, all code-built fragments (Q8). `grep -rn protocol_whitelist fastapi_backend/app` — 0 hits (Q21). | | P-MEDIA-LINK | Media, frames, documents and reports open without a session only through a link the server signed for that path, or a live share link. Text people or models write is never signed. | info-leak | security-critical | *(documented, docs/architecture.md "is never signed")* | Voided by: Knowing `ACCESS_SECRET_KEY` signs any path — `core/security.py:43`; dev compose key — `docker-compose.yml:24`. Lifetime `MEDIA_URL_EXPIRE_SECONDS` — `config.py:30`. Report pages sign every asset of the namespace — `api/pages.py:140-141`. Signing is limited to allow-listed fields — `api/media.py:25-30`. | | P-SHARE | A share link opens one asset, read-only. It expires and can be revoked. Only hashes are stored. A dead link answers the same 410 whatever went wrong. | integrity-bypass | security-critical | *(documented, docs/authentication.md "give read-only access to one asset's player and embed")* | Voided by: Liveness plus asset match only — `domain/auth.py:441-444`; `?s=` routes — `api/deps.py:252` (Q7). | -| P-FEEDS | A published feed's address opens that feed alone, read-only, with no session: the feed's XML and the audio of items it lists. Only an editor of the namespace makes a feed, and what it lists is bounded by who already reads that namespace: an item's resource or activity is in the namespace, or in one every reader of the feed's namespace reads too, never a vault's, and a namespace made a vault stops its feeds. The token is found by its hash; rotating or deleting the feed stops the old address at once. Text in an item cannot add markup to a reader. | info-leak | security-critical | *(documented, docs/feeds.md "Private by default")* | Voided by: the token is sealed with the server key, not only hashed, so the namespace's readers (and whoever holds `secret.key`) see the address again — `domain/syndication.py` `token_of`; anyone given the address reads the feed until it is rotated; editors choose what goes in, and a resource's title and summary show as they are now, so an edit after publishing shows; the fit is by namespace roles (`domain/syndication.py` `Fit`), not per-asset grants, so an asset later closed by access level still shows while its namespace fits. | +| P-FEEDS | A published feed's address opens that feed alone, read-only, with no session: the feed's XML and the audio of items it lists. Only an editor of the namespace makes a feed, and what it lists is bounded by who already reads that namespace: an item's resource or activity is in the namespace, or in one every reader of the feed's namespace reads too, never a vault's, and a namespace made a vault stops its feeds. The token is found by its hash; rotating or deleting the feed stops the old address at once. Text in an item cannot add markup to a reader. | info-leak | security-critical | *(documented, docs/feeds.md "Private by default")* | Voided by: the token is sealed with the server key, not only hashed, so the namespace's readers (and whoever holds `secret.key`) see the address again — `domain/syndication.py` `token_of`; anyone given the address reads the feed until it is rotated; editors choose what goes in, as does a routine (admins make them) whose workflow nodes publish what they pick, bounded the same way; a resource's title and summary show as they are now, so an edit after publishing shows; the fit is by namespace roles (`domain/syndication.py` `Fit`), not per-asset grants, so an asset later closed by access level still shows while its namespace fits. | | P-AUTH-REQ | A request is authenticated only if its bearer verifies and the account and session are live. Sign-out and disabling take effect at once, for every token kind. Revoking an app grant or API token ends its tokens at once. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Every request checks that the account is still active")* | Voided by: Liveness checked per request — `api/deps.py:92`; JWT algorithm from `ALGORITHM` — `config.py:27`; dev secrets — `docker-compose.yml:24`. No bypass flag: `grep -rniE "disable_auth\|skip_auth\|no_auth\|auth_bypass\|AUTH_DISABLED\|ALLOW_ANON\|DEV_MODE" fastapi_backend/app nextjs-frontend/lib nextjs-frontend/auth.ts nextjs-frontend/proxy.ts docker-compose*.yml install.sh` — 0 hits. | | P-REMOVE | Removing an account ends every way it signs in at once (sessions, passkeys, sign-in links, connected accounts, API tokens, app grants, IIIF tokens) and takes every role, collection role and recording permission. Its email can make a new account, which inherits nothing. Admins only; never yourself, never the Matterbridge account, never the last passkey into a vault unless the admin says its files may be lost. | integrity-bypass | security-critical | *(documented, docs/authentication.md "Removing someone")* | Voided by: the account row stays as a marker with an `.invalid` email — `domain/auth.py` `remove_account`; shared notes, views and comments keep its id. | | P-BEARER-ONLY | `/api/v1` takes bearer tokens only, never cookies. The IIIF authorization flow and the web app's NextAuth handlers are the cookie surfaces. | integrity-bypass | security-critical | *(documented, docs/architecture.md "so there is no CSRF surface")* | Voided by: IIIF auth cookies — `api/iiif.py:232`, `api/iiif.py:333`; external-login state cookie — `api/v1/routes/external.py:34`. | diff --git a/threat-model.yaml b/threat-model.yaml index 061ada6d9..9157300d4 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -1476,7 +1476,7 @@ properties_claimed: tier: security-critical conditions: "a published feed's address opens that feed alone, read-only, with no session; only an editor of the namespace makes one; it lists only items whose resource or activity the readers of the feed's namespace can read, never a vault's, and a namespace made a vault stops its feeds; the token is found by its hash, and rotating or deleting the feed stops the old address at once; text in an item cannot add markup to a reader" violation_symptoms: [info-leak] - x-voided-by: "the token is sealed with the server key, not only hashed, so the namespace's readers and whoever holds secret.key see the address again (domain/syndication.py token_of); anyone given the address reads the feed until it is rotated; editors choose what goes in, and a resource's title and summary show as they are now; the fit is by namespace roles (domain/syndication.py Fit), not per-asset grants" + x-voided-by: "the token is sealed with the server key, not only hashed, so the namespace's readers and whoever holds secret.key see the address again (domain/syndication.py token_of); anyone given the address reads the feed until it is rotated; editors choose what goes in, as does a routine (admins make them) whose workflow nodes publish what they pick, bounded the same way; a resource's title and summary show as they are now; the fit is by namespace roles (domain/syndication.py Fit), not per-asset grants" provenance: {kind: documented, source: 'docs/feeds.md "Private by default"'} - id: P-AUTH-REQ kind: authentication