From c2ed52fdad6c95a4cd101b182c6561ff2d7be461 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Fri, 16 Oct 2026 00:53:41 +0000 Subject: [PATCH] feat(mcp): subscribe to news feeds in one step Agents can now follow a news feed without the two-step source-then-watch dance: subscribe_feed (MCP and the assistant, admins) takes an address or a site's page and a namespace, reads the feed first and keeps it only if it works, names it by the feed's title, and adds the watch; a feed already followed is reused. news_feeds lists what's followed (addresses stay sealed) and unsubscribe_feed stops one, keeping the posts. The assistant's changes ask first outside a setup conversation, and its result never echoes the feed's own words. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01SWH8a26M9wkAPq2eRrfWZK --- CHANGELOG.md | 3 + docs/assistant.md | 5 +- docs/mcp.md | 5 +- docs/processing.md | 10 +- docs/requirements.md | 1 + fastapi_backend/app/api/mcp_tools.py | 64 +++++++++++ fastapi_backend/app/domain/ai_tools.py | 14 ++- fastapi_backend/app/domain/ops_tools.py | 68 ++++++++++- fastapi_backend/app/domain/rss.py | 107 +++++++++++++++++- fastapi_backend/tests/api/test_mcp.py | 2 + .../tests/api/test_news_feed_source.py | 38 +++++++ .../tests/api/test_setup_assistant.py | 36 +++++- nextjs-frontend/components/settings/model.ts | 3 + threat-model.json | 2 +- threat-model.md | 2 +- threat-model.yaml | 2 +- 16 files changed, 347 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ce8efc0d9..712e490ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,9 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version - **Automations on feed posts.** Each new post of a news feed is a `feed_post.received` activity with its address, site, feed, author, date and categories, so an automation can start on posts from one feed, author or category ([Kinds](docs/automations.md#kinds)). +- **Agents follow news feeds.** `subscribe_feed` (MCP and the assistant, admins) follows a feed into a namespace in one + step, checked before it's kept; `news_feeds` lists them and `unsubscribe_feed` stops one + ([News feeds](docs/processing.md#news-feeds-rss-and-atom)). ## 0.14.0 October 17, 2026 {id="0.14.0"} diff --git a/docs/assistant.md b/docs/assistant.md index 4ab1e52b0..2320413cc 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -648,8 +648,9 @@ and stops after 15 idle minutes. Installs that don't run on Docker don't have th ## Setting up and running the server by chat Admins also get the server tools: `server_status` (what's set up and what's missing), `find_model_servers`, -`read_settings` and `change_settings` (the processing and AI sections, not the server's hosts, cookies or tokens) and -`create_namespace`, and the people tools: `list_people`, `invite_person` (an account, optionally with a role in one +`read_settings` and `change_settings` (the processing and AI sections, not the server's hosts, cookies or tokens), +`create_namespace`, the news feed tools: `news_feeds`, `subscribe_feed` (follow a feed into a namespace in one step, +[News feeds](processing.md#news-feeds-rss-and-atom)) and `unsubscribe_feed`, and the people tools: `list_people`, `invite_person` (an account, optionally with a role in one namespace; Lens emails the sign-in link when mail is set up) and `change_person` (a role in a namespace, or disabling and enabling an account). Removing an account for good stays in the web app and the API. In an ordinary conversation their changes are approval cards. In a **setup conversation** diff --git a/docs/mcp.md b/docs/mcp.md index 261fb36c8..0f01c8219 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -128,6 +128,9 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `set_picture_descriptions` | choose what describes pictures (`engine` `llm` or `florence-2`), the provider's `vision_model`, Florence-2's size (`florence_model`) and `describe_max`. Admins only, with a write-scope token | | `install_settings` | whether other devices on the network reach Lens and its edition (full or lean), whether the machine's upgrade agent can change them, and the install line to run when it can't. Admins only | | `set_install_settings` | open Lens to the network or keep it to this machine, or change the edition: the machine sets itself up again and Lens restarts ([Updates](updates.md#setting-the-machine-up-again)). Admins only, with a write-scope token | +| `news_feeds` | the news feeds (RSS or Atom) Lens follows, the namespaces each brings posts into and how the last scan went; addresses stay sealed. Admins only | +| `subscribe_feed` | follow a news feed into a namespace in one step: its address or a page of the site, read first and kept only if it works; `backfill` also brings in the posts it lists now ([News feeds](processing.md#news-feeds-rss-and-atom)). Admins only, with a write-scope token | +| `unsubscribe_feed` | stop following a news feed; the posts it brought in stay. Admins only, with a write-scope token | | `propose_graph_change` | ask for two entities to be merged or linked; it waits in Proposed changes unless `apply` makes it at once (it can be undone). Needs a write-scope token and editor access | | `undo_graph_change` | take back a merge or link `propose_graph_change` made. Needs a write-scope token and editor access | | `edit_entity` | rename an entity, change its type, describe it, set the other ways it's said, or hide or show it. Needs a write-scope token and editor access | @@ -147,7 +150,7 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed `merge_topics`, `delete_topic` and `undo_topic_change`, which curate entities and topics as an editor does in the web app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which queues an episode, `write_spec`, which queues a spec, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and -editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring`, +editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), and `set_monitoring`, `set_sandbox_gateway` (the [code sandbox](assistant.md#code-sandbox)'s gateway) and `set_install_settings`, which have the machine change how Lens runs (admins). Importing stays in the web app and the API. diff --git a/docs/processing.md b/docs/processing.md index 548206785..5c1cf5e19 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -570,9 +570,11 @@ importing chosen ones and watching work as above. [News feeds](#news-feeds-rss-a ### News feeds (RSS and Atom) -A news feed (type `rss`) is a source too: Settings › Sources › Add › News feed, then watch it. Watching a feed -subscribes to it: each new post becomes a text resource of the namespace, titled and dated by the post, so answers cite -it and its link. +A news feed (type `rss`) is a source too: on the Sources page (Sensors › Files, email and calendars), add a News feed, +then watch it. Watching a feed subscribes to it: each new post becomes a text resource of the namespace, titled and +dated by the post, so answers cite it and its link. An agent does both in one step: `subscribe_feed` over +[MCP](mcp.md) or the assistant's tool of that name (admins) takes the address and a namespace, reads the feed first and +keeps it only if it works; `news_feeds` lists what's followed and `unsubscribe_feed` stops one. - **Address**: the feed's own (`https://`, or `feed://`), or a page of the site: when the page names its feed (``, as blogs and news sites do), the first feed it names is read. @@ -601,7 +603,7 @@ it and its link. Refine later: JSON Feed (feedparser reads RSS and Atom only); an option to capture each post's whole page, for feeds that carry only a summary; podcast feeds bringing in each episode's audio to transcribe; OPML import of a feed list; -subscribing in one step (address and namespace) from Settings. +subscribing in one step (address and namespace) on the Sources page, as agents can. ### Gmail and Google Workspace diff --git a/docs/requirements.md b/docs/requirements.md index aed08229d..a614ac9dc 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -110,6 +110,7 @@ request, to another check that proves the same thing; loosening a requirement or | IMAP mail, Gmail and Google Workspace | [test_mail_calendar_sources.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mail_calendar_sources.py) `test_an_imap_account_as_a_source`, `test_an_imap_message_is_an_email_document_with_its_attachments`; [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_a_workspace_mailbox_through_the_gmail_api`, `test_a_connected_google_account` | | iCal calendars and Google Calendar | [test_mail_calendar_sources.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mail_calendar_sources.py) `test_a_calendar_feed_as_a_source`; [test_google_calendar_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_google_calendar_source.py) `test_a_shared_calendar_read_as_a_service_account`, `test_a_connected_google_account` | | RSS and Atom news feeds: a feed or a site's page as a source, each post imported as text with its link, author and date; private networks only when allowed | [test_news_feed_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_news_feed_source.py) `test_a_news_feed_as_a_source`, `test_feed_addresses`, `test_what_a_feed_holds_stays_data`, `test_a_feed_file_imported_as_text` | +| Agents follow news feeds in one step: MCP `subscribe_feed` checks a feed and watches it into a namespace, `news_feeds` lists them with addresses sealed, `unsubscribe_feed` stops one; the assistant's tools ask first | [test_news_feed_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_news_feed_source.py) `test_subscribing_in_one_step`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_follows_news_feeds` | | A built-in MQTT broker, syslog and webhook streams | [test_sensors.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sensors.py) `test_hub_takes_devices_in_and_organises_them`, `test_syslog_senders_become_sensors`, `test_webhooks_and_one_list_of_sensors`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `sensor-webhook`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `sensor-webhook` ≤ 3 clicks | | Import webhooks for scanners and phone shortcuts | [test_import_hooks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_hooks.py) `test_pushing_files_urls_and_text`, `test_owners_manage_hooks`; [import-hooks.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/import-hooks.test.tsx) | | Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` | diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index e6ca14a38..f2c2f6441 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -61,6 +61,7 @@ nsfw, rdf, render, + rss, settings, sources, store, @@ -2534,6 +2535,69 @@ def set_install_settings(ctx: Context, public: bool | None = None, target: str | return updates.install_view(ctx.db, ctx.cfg) | {"changing": asked} +# ---------- news feeds (admins) ---------- +def _feeds_admin(ctx: Context, write: bool = False) -> None: + if not ctx.user.admin: + raise ToolError("only admins see or change the news feeds Lens follows") + if write and not ctx.user.can_write: + raise ToolError("this token is read-only: ask for a token with the write scope to change news feeds") + + +@tool( + "news_feeds", + "News feeds", + "The news feeds (RSS or Atom) Lens follows: each one's id (feed), name, the feed's own title, whether it could be " + "read last time (health), and the namespaces it brings new posts into, with how each one's last scan went. Their " + "addresses stay sealed. Each post comes in as text with its link, author and date; a new one is a " + "feed_post.received activity. Admins only.", + admin=True, +) +def news_feeds(ctx: Context) -> dict[str, Any]: + _feeds_admin(ctx) + return {"feeds": rss.subscriptions(ctx.db), "settings": f"{ctx.web}/sources"} + + +@tool( + "subscribe_feed", + "Subscribe to a news feed", + "Follow a news feed (RSS or Atom) into a namespace: url is the feed's address, or a page of the site that names its " + "feed. It's read first and kept only if it works; it's named by the feed's title unless name says. Each new post " + "then comes in as text, and with backfill also the posts the feed lists now. The namespace is made if it's new. " + "A feed already followed is reused. Feeds that need a password are added in the web app (Sources). Admins only, " + "with a token that has the write scope.", + Arg("url", "string", "the feed's address, or a page of the site", required=True, max_length=2000), + Arg("namespace", "string", "the namespace its posts go in", required=True, max_length=41), + Arg("name", "string", "what to call it (else the feed's title)", max_length=80), + Arg("backfill", "boolean", "also bring in the posts the feed lists now", default=False), + writes=True, + admin=True, +) +def subscribe_feed(ctx: Context, url: str, namespace: str, name: str | None = None, backfill: bool = False) -> dict[str, Any]: + _feeds_admin(ctx, write=True) + out = rss.subscribe(ctx.db, ctx.cfg, url, namespace, name, bool(backfill), by=ctx.user.email) + if out["added"]: + auth.audit(ctx.db, ctx.user.as_audit(), "source.create", f"storage_source:{out['feed']}", ["mcp"]) + if not out["already"]: + auth.audit(ctx.db, ctx.user.as_audit(), "watch.create", f"watch_path:{out['watch']}", ["mcp"]) + return out + + +@tool( + "unsubscribe_feed", + "Stop following a news feed", + "Stop following a news feed (its id from news_feeds): no more posts come in from it. The posts it brought in stay. " + "Admins only, with a token that has the write scope.", + Arg("feed", "integer", "the feed's id, from news_feeds", required=True), + writes=True, + admin=True, +) +def unsubscribe_feed(ctx: Context, feed: int) -> dict[str, Any]: + _feeds_admin(ctx, write=True) + rss.unsubscribe(ctx.db, feed) + auth.audit(ctx.db, ctx.user.as_audit(), "source.delete", f"storage_source:{feed}", ["mcp"]) + return {"unsubscribed": feed} + + # ---------- the stream of activities ---------- @tool( "timeline", diff --git a/fastapi_backend/app/domain/ai_tools.py b/fastapi_backend/app/domain/ai_tools.py index 03f0ef2fc..71b56d481 100644 --- a/fastapi_backend/app/domain/ai_tools.py +++ b/fastapi_backend/app/domain/ai_tools.py @@ -2148,7 +2148,8 @@ def cited(self, text): def approve(db, cfg, aid, user, editable, decision="approve", base=None, admin=False, readable=None, params=None, via="web"): """Carry out an approved action: a batch run (or a sample of it), an entity change, a note an automation's agent - proposed, or a change to the server (ops_tools.py: settings, a namespace, importing attached files or web pages). + proposed, or a change to the server (ops_tools.py: settings, a namespace, importing attached files or web pages, + following a news feed). An automation's action (an approval with an intent) is decided by approvals.decide: `params` are an edit's, `via` where the pick was made (web, chat, token, oauth or agent; only the person, signed in or in their own chat, approves it).""" @@ -2198,7 +2199,16 @@ def _carry_out(db, cfg, a, user, editable, decision, base, admin, readable): result = {"result": sandboxmod.run(db, cfg, a["chat"], a.get("account") or user["id"], kind, source, args.get("files") or [])} except sandboxmod.SandboxError as e: raise ValueError(str(e)) from None - elif a["tool"] in ("change_settings", "create_namespace", "import_files", "import_urls", "invite_person", "change_person"): + elif a["tool"] in ( + "change_settings", + "create_namespace", + "import_files", + "import_urls", + "invite_person", + "change_person", + "subscribe_feed", + "unsubscribe_feed", + ): result = ops_tools.apply(db, cfg, base or cfg, a["tool"], args, user, editable, admin) elif a["tool"] == "decline_approval": # the person's yes to declining another of their approvals from . import approvals diff --git a/fastapi_backend/app/domain/ops_tools.py b/fastapi_backend/app/domain/ops_tools.py index 24691ce56..9b8e2f500 100644 --- a/fastapi_backend/app/domain/ops_tools.py +++ b/fastapi_backend/app/domain/ops_tools.py @@ -11,7 +11,7 @@ from __future__ import annotations -from . import auth, blobs, convert, decide, feeds, jobs, links, llm, metadata, roles, settings, setup, sources, store, uploads +from . import auth, blobs, convert, decide, feeds, jobs, links, llm, metadata, roles, rss, settings, setup, sources, store, uploads R = store.R _S = {"type": "string"} @@ -96,6 +96,31 @@ [], False, ), + ( + "news_feeds", + "The news feeds (RSS or Atom) Lens follows: each one's id, name and title, whether it could be read last time, " + "and the namespaces it brings new posts into, with how the last scan went.", + {}, + [], + False, + ), + ( + "subscribe_feed", + "Follow a news feed (RSS or Atom) into a namespace: url is the feed's address, or a page of the site that names " + "it. It's read first and kept only if it works, named by the feed's title unless name says. Each new post then " + "comes in as text with its link; backfill also brings in the posts the feed lists now. A feed that needs a " + "password is added on the Sources page.", + {"url": _S, "namespace": _S, "name": _S, "backfill": {"type": "boolean"}}, + ["url", "namespace"], + True, + ), + ( + "unsubscribe_feed", + "Stop following a news feed, by its id from news_feeds. The posts it brought in stay.", + {"feed": {"type": "integer"}}, + ["feed"], + True, + ), ( "create_namespace", "Create a namespace: a separate set of assets (recordings, documents and images) with its own people and access. " @@ -238,6 +263,21 @@ def apply(db, cfg, base, tool, args, user, editable, admin): auth.audit(db, user, "upload", f"recording:{up['recording']}", {"file": up["filename"], "namespace": ns, "assistant": True}) placed.append({"file": up["filename"], "recording": up["recording"], "duplicate": bool(up.get("duplicate"))}) return {"namespace": ns, "imported": placed} + if tool == "subscribe_feed": + if not admin: + raise PermissionError("only admins add news feeds") + out = rss.subscribe(db, cfg, args["url"], args["namespace"], args.get("name"), bool(args.get("backfill")), by=user["email"]) + if out["added"]: + auth.audit(db, user, "source.create", f"storage_source:{out['feed']}", ["assistant"]) + if not out["already"]: + auth.audit(db, user, "watch.create", f"watch_path:{out['watch']}", ["assistant"]) + return {k: out[k] for k in ("feed", "watch", "namespace", "posts", "added", "already")} # not the feed's own words + if tool == "unsubscribe_feed": + if not admin: + raise PermissionError("only admins remove news feeds") + rss.unsubscribe(db, args["feed"]) + auth.audit(db, user, "source.delete", f"storage_source:{args['feed']}", ["assistant"]) + return {"unsubscribed": args["feed"]} if tool in ("invite_person", "change_person"): if not admin: raise PermissionError("only admins manage people") @@ -410,6 +450,32 @@ def t_file_storage(self, check=False): out["check"] = blobs.test(self.db, self.cfg) return out, "Checked where files are kept" + (": " + ("it works" if out["check"]["ok"] else "it failed") if check else "") + def t_news_feeds(self): + found = rss.subscriptions(self.db) + return {"feeds": found}, f"Listed {len(found)} news feeds" + + def t_subscribe_feed(self, url, namespace, name=None, backfill=False): + try: # checked here, so a wrong address is said before anything waits for approval + shown = rss.address(url) + except RuntimeError as e: + raise ValueError(str(e)) from None + ns = str(namespace or "").strip().lower() + if not store.NS_RX.match(ns): + raise ValueError("a namespace's name uses lowercase letters, digits, - and _") + args = store.clean({"url": shown, "namespace": ns, "name": name, "backfill": bool(backfill) or None}) + return self._change( + "subscribe_feed", args, f"Follow the news feed {shown} into {ns}" + (" with its current posts" if backfill else "") + ) + + def t_unsubscribe_feed(self, feed): + try: + src = sources.get(self.db, int(feed)) + except (KeyError, TypeError, ValueError): + raise ValueError("there's no news feed with that id: news_feeds lists them") from None + if src["type"] != "rss": + raise ValueError("that source isn't a news feed") + return self._change("unsubscribe_feed", {"feed": src["id"]}, f"Stop following the news feed {src['name']}") + def t_create_namespace(self, name, graph="shared"): name = name.strip().lower() if not store.NS_RX.match(name): diff --git a/fastapi_backend/app/domain/rss.py b/fastapi_backend/app/domain/rss.py index 93768f67e..43137b0e6 100644 --- a/fastapi_backend/app/domain/rss.py +++ b/fastapi_backend/app/domain/rss.py @@ -78,7 +78,12 @@ def _html_text(markup): def _url(cfg, src): from . import feeds # it imports this - url = feeds._secret(cfg, src, "url").strip() + return address(feeds._secret(cfg, src, "url")) + + +def address(url): + """A feed's address as it's fetched: https:// for feed://, and checked to be a web address.""" + url = str(url or "").strip() url = re.sub(r"^feed:(?=https?://)", "", url, flags=re.I) # feed:https://…, as some sites link their feeds url = re.sub(r"^feed://", "https://", url, flags=re.I) if not re.match(r"^https?://[^\s/]+\S*$", url, re.I): @@ -411,3 +416,103 @@ def forget(src): def test(cfg, src): forget(src) read(cfg, src) + + +# ---------- subscribing in one step: a feed as a source and a watch (MCP, the assistant) ---------- +def _watching(w, names): + return { + "watch": w["id"], + "namespace": names.get(w["space"]), + "enabled": w.get("enabled", True), + "backfill": bool(w.get("backfill")), + "last_scan_at": w.get("last_scan_at"), + "next_scan_at": w.get("next_scan_at"), + "last_error": w.get("last_error"), + "new_last_scan": (w.get("last_stats") or {}).get("new"), + } + + +def subscriptions(db): + """Every news feed, the namespaces it's watched into and how its last scan went: [{feed, name, title, health, + watching: [...]}]. Their addresses stay sealed. `title` is the feed's own, once read since Lens started.""" + from . import sources, store + + names, watches = store.space_names(db), sources.list_watches(db) + out = [] + for s in sources.list_sources(db): + if s["type"] != "rss": + continue + hit = _FEEDS.get(s["id"]) + out.append( + { + "feed": s["id"], + "name": s["name"], + "title": hit["title"] if hit else None, + "health": s.get("health"), + "watching": [_watching(w, names) for w in watches if w["source"] == s["id"]], + } + ) + return out + + +def subscribe(db, cfg, url, namespace, name=None, backfill=False, by=None, user=None, password=None): + """Subscribe `namespace` (made if it's new) to the news feed at `url` (or the feed a site's page names): a news feed + source, checked before it's kept, named by the feed's title unless `name` says, and a watch that brings each new + post in as text, and with `backfill` those it lists already. A feed already a source is reused, and a namespace + already watching it is told so: {feed, name, title, watch, namespace, posts, added, already}.""" + from . import sources, store + + try: + url = address(url) + except RuntimeError as e: + raise ValueError(str(e)) from None + ns = str(namespace or "").strip().lower() + if not store.NS_RX.match(ns): + raise ValueError("a namespace's name uses lowercase letters, digits, - and _") + same = None + for s in db.rows("SELECT record::id(id) AS id, type, sealed FROM storage_source WHERE type = 'rss'"): + try: + if _url(cfg, s) == url: + same = s["id"] + break + except (RuntimeError, ValueError): + continue + added = same is None + if added: + secrets = {"url": url, **({"pass": password} if password else {})} + sid = sources.create(db, cfg, (name or "News feed")[:80], "rss", {"user": user} if user else {}, secrets, by) + health = sources.test(db, cfg, sid) + if not health["ok"]: + sources.remove(db, sid) + raise ValueError(f"that feed can't be read, so nothing was added: {health['error']}") + else: + sid = same + try: + feed = read(cfg, sources.get(db, sid)) + except RuntimeError as e: + raise ValueError(f"that feed can't be read right now: {e}") from None + space = store.ns_id(db, ns) + if added and not name and feed["title"]: + db.q("UPDATE $r SET name = $n", r=store.R("storage_source", sid), n=feed["title"][:80]) + have = next((w for w in sources.list_watches(db) if w["source"] == sid and w["space"] == space), None) + wid = have["id"] if have else sources.create_watch(db, cfg, sid, "", space, by, backfill=bool(backfill)) + return { + "feed": sid, + "name": sources.get(db, sid)["name"], + "title": feed["title"], + "watch": wid, + "namespace": ns, + "posts": len(feed["entries"]), + "added": added, + "already": have is not None, + } + + +def unsubscribe(db, sid): + """Stop following a news feed: its source and its watches go; the posts it brought in stay.""" + from . import sources + + if sources.get(db, sid)["type"] != "rss": + raise ValueError("that source isn't a news feed") + sources.remove(db, sid) + _FEEDS.pop(sid, None) diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index 27ec14753..03a52d077 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -92,6 +92,8 @@ def test_clients_are_sent_to_sign_in(client, app, env): "set_monitoring", "set_sandbox_gateway", "set_install_settings", + "subscribe_feed", + "unsubscribe_feed", "mark_activities", "decline_approval", "set_automation_vars", diff --git a/fastapi_backend/tests/api/test_news_feed_source.py b/fastapi_backend/tests/api/test_news_feed_source.py index 5f8f1cf25..9aca4d829 100644 --- a/fastapi_backend/tests/api/test_news_feed_source.py +++ b/fastapi_backend/tests/api/test_news_feed_source.py @@ -5,6 +5,7 @@ import base64 import http.server +import json import socket import threading @@ -296,3 +297,40 @@ def test_a_feed_file_imported_as_text(client, db, admin): assert r.status_code == 200, r.text said = _segments(db, r.json()["id"]) assert "Two days on the water." in said and "https://harbour.example.org/posts/regatta" in said + + +def test_subscribing_in_one_step(site, client, db, cfg, admin): + """An agent follows a feed into a namespace with one call (MCP), sees it listed, and stops following it.""" + from tests.api.test_mcp import tool, tool_error + + viewer = make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + assert viewer and "only admins" in tool_error(client, login(client, "vi@x.io", "viewer password 1"), "news_feeds") + assert "can't be read, so nothing was added" in tool_error(client, admin, "subscribe_feed", url=f"{site}/plain", namespace="news") + assert "news" not in store.space_names(db).values() and tool(client, admin, "news_feeds")["feeds"] == [] + assert "https://" in tool_error(client, admin, "subscribe_feed", url="ftp://x/feed.xml", namespace="news") + + got = tool(client, admin, "subscribe_feed", url=f"{site}/blog", namespace="news", backfill=True) + assert (got["name"], got["title"], got["namespace"], got["posts"], got["added"], got["already"]) == ( + "Harbour news", + "Harbour news", + "news", + 2, + True, + False, + ) + assert sources.poll_watch(db, cfg, got["watch"])["new"] == 2 + # the same feed again: reused, and already watched into that namespace; another namespace gets its own watch + again = tool(client, admin, "subscribe_feed", url=f"{site}/blog", namespace="news") + assert (again["feed"], again["watch"], again["added"], again["already"]) == (got["feed"], got["watch"], False, True) + other = tool(client, admin, "subscribe_feed", url=f"{site}/blog", namespace="harbour") + assert other["feed"] == got["feed"] and other["watch"] != got["watch"] and not other["already"] + + (feed,) = tool(client, admin, "news_feeds")["feeds"] + assert (feed["feed"], feed["name"], feed["title"], feed["health"]["ok"]) == (got["feed"], "Harbour news", "Harbour news", True) + assert [(w["namespace"], w["new_last_scan"]) for w in feed["watching"]] == [("news", 2), ("harbour", None)] + assert site not in json.dumps(feed) # the address stays sealed + + assert tool(client, admin, "unsubscribe_feed", feed=got["feed"]) == {"unsubscribed": got["feed"]} + assert tool(client, admin, "news_feeds")["feeds"] == [] and db.rows("SELECT id FROM watch_path") == [] + assert len(db.rows("SELECT id FROM recording")) == 2 # what it brought in stays + assert tool_error(client, admin, "unsubscribe_feed", feed=got["feed"]) diff --git a/fastapi_backend/tests/api/test_setup_assistant.py b/fastapi_backend/tests/api/test_setup_assistant.py index 056ba6cec..13c561980 100644 --- a/fastapi_backend/tests/api/test_setup_assistant.py +++ b/fastapi_backend/tests/api/test_setup_assistant.py @@ -7,9 +7,10 @@ import pytest -from app.domain import settings, setup, store +from app.domain import rss, settings, setup, store from tests import fake_llm from tests.api._assist import Assist, sse, start_llm +from tests.api.test_news_feed_source import site # noqa: F401 - a news feed served from this process from tests.helpers import write_wav @@ -226,3 +227,36 @@ def test_the_assistant_captures_web_pages(app, db, cfg, folder, new_client, llm, vid = cv.post("/api/v1/chats", headers=hv, json={}).json()["id"] ev = sse(cv.post(f"/api/v1/chats/{vid}/messages", headers=hv, json={"content": "keep https://8.8.8.8/x"}).text) assert "approval" not in ev + + +def test_the_assistant_follows_news_feeds(app, db, cfg, folder, new_client, llm, site): # noqa: F811 - the fixture + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["admin"] + llm.tool_script = [ + {"content": "", "tool_calls": [call(1, "subscribe_feed", {"url": "ftp://x/feed.xml", "namespace": "news"})]}, + {"content": "", "tool_calls": [call(2, "subscribe_feed", {"url": f"{site}/blog", "namespace": "news"})]}, + {"content": "", "tool_calls": [call(3, "news_feeds", {})]}, + {"content": "Following Harbour news into news."}, + ] + cid = c.post("/api/v1/chats", headers=h, json={"kind": "setup"}).json()["id"] + ev = sse(c.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": "follow the harbour blog in news"}).text) + assert "https://" in ev["step"][0]["summary"] and "approval" not in ev + assert ev["step"][1]["summary"] == f"Follow the news feed {site}/blog into news: done" + done = json.loads(llm.seen[-2]["messages"][-1]["content"]) + assert (done["status"], done["posts"], done["added"], "title" in done) == ("done", 2, True, False) # not the feed's words + (feed,) = rss.subscriptions(db) + assert feed["name"] == "Harbour news" and [w["namespace"] for w in feed["watching"]] == ["news"] + audit = db.values("SELECT VALUE action FROM audit_log WHERE detail CONTAINS 'assistant'") + assert {"source.create", "watch.create"} <= set(audit) + + # elsewhere, stopping waits for approval + llm.tool_script = [ + {"content": "", "tool_calls": [call(1, "unsubscribe_feed", {"feed": feed["feed"]})]}, + {"content": "I asked to stop following it."}, + ] + other = c.post("/api/v1/chats", headers=h, json={}).json()["id"] + ev = sse(c.post(f"/api/v1/chats/{other}/messages", headers=h, json={"content": "stop the harbour feed"}).text) + appr = ev["approval"][0] + assert appr["summary"] == "Stop following the news feed Harbour news" and rss.subscriptions(db) + assert c.post(f"/api/v1/approvals/{appr['id']}", headers=h, json={"decision": "approve"}).json()["status"] == "done" + assert rss.subscriptions(db) == [] diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index b9419a77b..a08eef157 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -518,6 +518,9 @@ export const AI_TOOLS: { name: string; label: string; acts: boolean }[] = [ { name: "read_settings", label: "Read settings (admins)", acts: false }, { name: "change_settings", label: "Change settings (admins)", acts: true }, { name: "create_namespace", label: "Create namespaces (admins)", acts: true }, + { name: "news_feeds", label: "List the news feeds Lens follows (admins)", acts: false }, + { name: "subscribe_feed", label: "Follow news feeds (admins)", acts: true }, + { name: "unsubscribe_feed", label: "Stop following news feeds (admins)", acts: true }, ]; /** The speech providers (the backend's app/domain/speech.py), in the order Settings shows them. */ diff --git a/threat-model.json b/threat-model.json index c9e859ae0..6a6b687ab 100644 --- a/threat-model.json +++ b/threat-model.json @@ -436,7 +436,7 @@ "parameter": "the feed's address (or a site's page that names its feed), user and password; the feed's XML and its posts' HTML", "attacker_controllable": "yes", "component": "outbound", - "caller_must_enforce": "admin creates sources; fetched as iCal feeds are; feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post kept as an Atom entry Lens writes, without the feed's address; post text is marked untrusted in prompts", + "caller_must_enforce": "admins add them (web app, API, MCP subscribe_feed with a write-scope token, or the assistant's tool, which asks first outside a setup conversation or once it has read outside text); fetched as iCal feeds are; feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post kept as an Atom entry Lens writes, without the feed's address; post text is marked untrusted in prompts", "provenance": "documented", "source": "docs/processing.md \"News feeds (RSS and Atom)\"" }, diff --git a/threat-model.md b/threat-model.md index dde8c2ba8..cfc7eacff 100644 --- a/threat-model.md +++ b/threat-model.md @@ -288,7 +288,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /hooks/import` | bearer (a hook's `lih_` token, or a person's `la_`, `lo_` or web token with `?namespace=`), file bytes, URL, text | yes: anyone holding a hook's token, for its namespace; a person's token, for namespaces where its person has the import permission and only with write scope | data, resource-name, size | none; the address form `/hooks/import/{token}` takes hook tokens only, so a person's token never sits in a logged path | *(documented, docs/api.md "the address form takes hook tokens only")* | | watched folders, storage sources, IMAP, Gmail, iCal, news feeds | remote file names, bytes, mail, events, feed posts | yes, whoever writes to the remote, mails the box or serves the feed | data, resource-name, size | admin creates sources | *(inferred, Q17)* | | Google Calendar sources (`gcal.py`, `google_auth.py`) | a service account key file or a connected account's refresh token, calendar ids, `impersonate`; the events Google returns | key and token: admins only, sealed and write-only (P-SECRETS); events: yes, anyone who can invite the account to an event or edit a calendar it reads | data, x-credential, x-prompt-injection, size | admin creates sources; read-only scope (`calendar.readonly`); the token endpoint and API address are Google's own (a key file's `token_uri` is ignored) and calendar ids are quoted into the path; event text is written into its .ics escaped and without control characters, so an event can't add properties; events are data, as iCal feeds' are | *(documented, docs/processing.md "The token endpoint is always Google's own")* | -| News feed sources (`rss.py`) | the feed's address (or a site's page that names its feed), a user and password; the feed's XML and its posts' HTML | address and password: admins only, sealed and write-only (P-SECRETS); XML and HTML: yes, whoever serves the feed or writes its posts | data, x-prompt-injection, size | admin creates sources; fetched as iCal feeds are (P-ADMIN-GUARD); feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post is kept as an Atom entry Lens writes (escaped), without the feed's address; post text reaches prompts as marked, untrusted data | *(documented, docs/processing.md "News feeds (RSS and Atom)")* | +| News feed sources (`rss.py`) | the feed's address (or a site's page that names its feed), a user and password; the feed's XML and its posts' HTML | address and password: admins only, sealed and write-only (P-SECRETS); XML and HTML: yes, whoever serves the feed or writes its posts | data, x-prompt-injection, size | admins add them (web app, API, MCP `subscribe_feed` with a write-scope token, or the assistant's tool, which asks first outside a setup conversation or once it has read outside text); fetched as iCal feeds are (P-ADMIN-GUARD); feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post is kept as an Atom entry Lens writes (escaped), without the feed's address; post text reaches prompts as marked, untrusted data | *(documented, docs/processing.md "News feeds (RSS and Atom)")* | | Connect with Google (`POST /sources/{id}/google`, then `/auth/external/google/callback`) | an OAuth client id and secret; on the way back `state`, `code` and the email in Google's id token | client: admins only; `state` and `code`: yes, anyone who can send the admin a link | data, x-credential, x-collaborator-data | admin only, from the web app (API tokens refused), and still an admin when Google sends them back; the flow reuses external sign-in's: `state` is single-use, kept only hashed, expires in 10 minutes and must come back to the browser that started it (cookie), with PKCE (S256); the client secret is sealed on the flow and the refresh token on the source; the token is kept only when Google granted the source's scope (read only, but all of Gmail for a Gmail source read over IMAP, which Google offers no read-only scope for; Lens still only reads; all of Drive for a Drive connection set to full access), and a Drive connection keeps its client's secret sealed with it, for rclone to renew the token; the email is only shown, and comes straight from Google's token endpoint; the browser returns only to the web origin that started it | *(documented, docs/processing.md "the consent itself is a person's")* | | the assistant's mailbox (`mailbox.py`) | mail headers, text, attachments, forwarded mail, `Authentication-Results` | yes, anyone who knows the address; From is forgeable | data, x-prompt-injection, size | answered only for `mailbox.users` whose provider vouched for From; forwarded text marked untrusted, and changes after it ask first | *(documented, docs/assistant-email.md "it answers only mail that is both")* | | `POST /imports/web`, extension `http` tools, notification targets | URL | yes: editors (web, tools), owners (targets) | resource-name, x-network-destination | none | *(maintainer, 2026-10)* | diff --git a/threat-model.yaml b/threat-model.yaml index 567079594..941dc347e 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -361,7 +361,7 @@ entry_points: x-condition: "address and password: admins only, sealed and write-only; XML and HTML: whoever serves the feed or writes its posts" control_kinds: [data, x-prompt-injection, size] obligation_id: admin-creates-sources - caller_must_enforce: "admin creates sources; fetched as iCal feeds are; feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post kept as an Atom entry Lens writes, without the feed's address; post text is marked untrusted in prompts" + caller_must_enforce: "admins add them (web app, API, MCP subscribe_feed with a write-scope token, or the assistant's tool, which asks first outside a setup conversation or once it has read outside text); fetched as iCal feeds are; feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post kept as an Atom entry Lens writes, without the feed's address; post text is marked untrusted in prompts" provenance: {kind: documented, source: 'docs/processing.md "News feeds (RSS and Atom)"'} - id: "Connect with Google" component: outbound