From b88178555c42f67b7bb87f032b486d35e7cad00c Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Fri, 16 Oct 2026 00:42:12 +0000
Subject: [PATCH 1/2] feat(sources): RSS and Atom news feeds as a source
A news feed is a new non-storage source type, "rss". Its address (a feed,
or a site's page that names one) is sealed like an iCal link; each post is
listed as its own Atom entry file (.atom), watched like any source and
imported as a text resource with its link, author and dates, so answers
cite the post. Feeds are fetched through the SSRF guard with conditional
GET, parsed with feedparser (BSD) with external entities off, and post HTML
is cleaned before it is kept. Adds the feed_post content type, docs, the
threat model entry point and the web app's source form.
Co-Authored-By: Claude Opus 5.5 (1M context)
Claude-Session: https://claude.ai/code/session_01SWH8a26M9wkAPq2eRrfWZK
---
CHANGELOG.md | 5 +
README.md | 15 +-
docs/configuration.md | 2 +-
docs/processing.md | 39 +-
docs/sensors.md | 2 +-
fastapi_backend/app/api/v1/routes/imports.py | 2 +-
fastapi_backend/app/domain/content_types.py | 1 +
fastapi_backend/app/domain/feeds.py | 54 ++-
fastapi_backend/app/domain/ingest.py | 12 +-
fastapi_backend/app/domain/rss.py | 397 ++++++++++++++++++
fastapi_backend/app/domain/sources.py | 14 +-
fastapi_backend/app/schemas/sources.py | 4 +-
fastapi_backend/pyproject.toml | 1 +
.../tests/api/test_content_types.py | 1 +
.../tests/api/test_news_feed_source.py | 283 +++++++++++++
fastapi_backend/uv.lock | 23 +
.../__tests__/sources-model.test.ts | 21 +
.../app/openapi-client/types.gen.ts | 29 +-
.../components/library/source-labels.ts | 2 +
.../components/library/sources-strip.tsx | 3 +
.../components/sources/connection-dialog.tsx | 4 +-
.../components/sources/connection-fields.tsx | 17 +
.../components/sources/source-icons.tsx | 2 +
.../components/sources/source-model.ts | 31 +-
.../components/storage/file-store.ts | 2 +-
nextjs-frontend/openapi.json | 9 +-
threat-model.json | 30 +-
threat-model.md | 21 +-
threat-model.yaml | 34 +-
29 files changed, 973 insertions(+), 87 deletions(-)
create mode 100644 fastapi_backend/app/domain/rss.py
create mode 100644 fastapi_backend/tests/api/test_news_feed_source.py
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0f9be8450..6fea61cf9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
## Unreleased
+- **News feeds.** Subscribe to a blog's or a news site's RSS or Atom feed: add it in Settings › Sources (its address,
+ or just the site's) and watch it, and each new post comes into a namespace as text, with its link, author and date,
+ so answers cite it. Feeds are asked for again only when they've changed, and a private feed's address is kept like a
+ password ([News feeds](docs/processing.md#news-feeds-rss-and-atom)).
+
## 0.12.0 October 15, 2026 {id="0.12.0"}
- **microsandbox runs on Debian 13 too.** Its sandbox now starts from `python:3.12-slim-trixie`, like the backend
diff --git a/README.md b/README.md
index 4b2f686bd..4bfe141ec 100644
--- a/README.md
+++ b/README.md
@@ -84,13 +84,14 @@ Everything below is built and merged. Work that is still in a pull request is li
([Extending the assistant](docs/assistant.md#extending-the-assistant-tools-skills-hooks-and-plugins),
[the MCP runner](docs/assistant.md#the-mcp-runner))
- **Bring in everything.** Watched folders on S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV or local disks (a
- local folder in four clicks); IMAP mail, Gmail and Google Workspace, iCal calendars and Google Calendar; a built-in
- MQTT broker, syslog and webhook streams; import webhooks for scanners and phone shortcuts. Google accounts connect
- with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google Drive can sign in through
- rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs, Office files, photos, JSON
- and zip or tar archives, and, with an Apache Tika server (an opt-in extra), documents nothing else here can read;
- capture web pages one at a time, from a link list or a bookmarks file, by API or by asking the assistant, from private
- networks you allow in Settings too; and import videos and podcasts by their address (off by default).
+ local folder in four clicks); IMAP mail, Gmail and Google Workspace, iCal calendars, Google Calendar and RSS and Atom
+ news feeds; a built-in MQTT broker, syslog and webhook streams; import webhooks for scanners and phone shortcuts.
+ Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google
+ Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs,
+ Office files, photos, JSON and zip or tar archives, and, with an Apache Tika server (an opt-in extra), documents
+ nothing else here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking
+ the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off
+ by default).
([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika),
[import webhooks](docs/api.md#import-webhooks))
- **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a
diff --git a/docs/configuration.md b/docs/configuration.md
index 3297c8eb4..a8c979373 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -259,7 +259,7 @@ text (Word, text, Markdown) are imported as text. Settings → Documents:
| `documents.convert_seconds` | 300 | how long making one PDF may take before its job fails, 10–3600 seconds |
| `documents.attachment_resources` | true | whether an email's attachments that Lens can read (documents, images, audio, video, emails) also become resources of their own; they're kept as its files either way |
| `documents.video_urls` | false | whether videos and podcasts can be imported by their address ([API](api.md#videos-and-podcasts)): their media is downloaded with yt-dlp, which Lens fetches on first use |
-| `documents.web_networks` | `[]` | private networks (CIDR, such as `192.168.1.0/24`) that web pages, calendar feeds and videos may be fetched from besides the public internet: for an intranet, or a calendar server at home. The web app can't list this machine (loopback) or link-local addresses, where clouds answer with the server's own credentials; archive.yaml or `LENS_WEB_NETWORKS` in `.env` (comma-separated, added to archive.yaml's) can. Those stay whatever Settings saves: the networks listed in Settings are added to them |
+| `documents.web_networks` | `[]` | private networks (CIDR, such as `192.168.1.0/24`) that web pages, calendar and news feeds and videos may be fetched from besides the public internet: for an intranet, or a calendar server at home. The web app can't list this machine (loopback) or link-local addresses, where clouds answer with the server's own credentials; archive.yaml or `LENS_WEB_NETWORKS` in `.env` (comma-separated, added to archive.yaml's) can. Those stay whatever Settings saves: the networks listed in Settings are added to them |
| `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it |
| `documents.anytopdf_url` | none | a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work`, such as `https://convert.home:8640`. Set, it converts instead of the program here |
| `documents.anytopdf_token` | none | the node's bearer token (its `ANYTOPDF_QUEUE_TOKEN`), kept encrypted |
diff --git a/docs/processing.md b/docs/processing.md
index cd0cddf95..3335dde12 100644
--- a/docs/processing.md
+++ b/docs/processing.md
@@ -293,7 +293,8 @@ A folder scan and a routine run are each claimed first, so several processes doi
## Storage sources
Admins add sources in the app (`/api/v1/sources`): S3 or S3-compatible, Dropbox, Google Drive, OneDrive, SFTP, SMB,
-WebDAV, or a folder on this machine; or an email account (IMAP, or Gmail and Google Workspace), a calendar feed (iCal) or Google Calendar, below. A watched folder maps a path on a source to a namespace, with include/exclude
+WebDAV, or a folder on this machine; or an email account (IMAP, or Gmail and Google Workspace), a calendar feed (iCal),
+Google Calendar or a [news feed](#news-feeds-rss-and-atom) (RSS or Atom), below. A watched folder maps a path on a source to a namespace, with include/exclude
patterns, audio and/or transcripts, a polling interval, how long a file must be unchanged before it is picked up, and
whether files already there are imported (backfill). New audio is queued for the full pipeline; new transcripts are
imported and analysed. Audio stays where it is: it is copied to a cache for processing and streamed from the source for
@@ -345,8 +346,8 @@ files, then Import (`POST /api/v1/import/source`). The listing marks files that
### Email (IMAP), calendar feeds (iCal) and Google Calendar
-Three kinds of source aren't storage and don't use rclone; their messages and events are shown as files, so browsing,
-importing chosen ones and watching work as above.
+These kinds of source aren't storage and don't use rclone; their messages and events are shown as files, so browsing,
+importing chosen ones and watching work as above. [News feeds](#news-feeds-rss-and-atom) work the same way.
- **Email (IMAP)**: host, port, security (SSL/TLS, STARTTLS or none), user and password (an app password where the
provider has them). Mailboxes are the folders, and each message is a file `/-.eml`,
@@ -414,6 +415,38 @@ importing chosen ones and watching work as above.
a Google calendar's secret address in iCal format (Settings › Integrate calendar) works as a calendar feed.
- `.eml` and `.ics` files in any source, and uploaded through Import, can be read as text this way too.
+### News feeds (RSS and Atom)
+
+A news feed (type `rss`) is a source too: Settings › Sources › Add › News feed, then watch it. Watching a feed
+subscribes to it: each new post becomes a text resource of the namespace, titled and dated by the post, so answers cite
+it and its link.
+
+- **Address**: the feed's own (`https://`, or `feed://`), or a page of the site: when the page names its feed
+ (``, as blogs and news sites do), the first feed it names is read.
+ A user and password if the feed asks for one. The address is kept encrypted like a password and never shown again,
+ since a private feed's address often carries its key. It is fetched as calendar feeds are: public addresses only
+ (and the networks in `documents.web_networks`), on ports 80 and 443, and the password is never sent to another
+ server, including one that a page names as its feed's.
+- **Posts**: the feed is one folder of posts, each a file `.atom` named by its date and title. RSS (0.9x, 1.0,
+ 2.0) and Atom (0.3, 1.0) are read by [feedparser](https://github.com/kurtmckee/feedparser) (BSD), which drops
+ scripts, styles and event handlers from the posts' HTML and makes their links absolute. Whatever the feed, each post
+ is kept as an Atom entry document of its own (RFC 4287): its title, the post's page, enclosures (a podcast's audio),
+ when it was published and updated, its authors, categories, summary and content, and the feed's title and site
+ (never its address). It comes in as text: the title, its link, who wrote it, when, the feed, its categories and
+ enclosures, then its text. Its content type is Feed post, so a namespace can give feed posts a pipeline of their own
+ (Settings › Content types).
+- **New posts only, unless you backfill**: as for any watch, a new watch on a feed takes the posts published after it
+ starts; turn on backfill to bring in those the feed lists already. The newest 500 posts of a feed are listed.
+- **Light on the network**: a feed read is reused for a minute, then asked for again with its `ETag` and
+ `Last-Modified`, so a feed that hasn't changed costs one small request. A post is known by its id (an RSS item's
+ `guid`), so a feed that reorders or retitles its posts doesn't bring them in twice; a post whose text changes is read
+ again into the resource it already is. A post the feed drops later keeps its resource and its file.
+- `.atom` files uploaded through Import (a single post, or a whole Atom feed) are read as text the same way.
+
+Refine later: JSON Feed (feedparser reads RSS and Atom only); an option to capture each post's whole page, for feeds
+that carry only a summary; podcast feeds bringing in each episode's audio to transcribe; OPML import of a feed list;
+subscribing in one step (address and namespace) from Settings.
+
### Gmail and Google Workspace
A Gmail or Google Workspace mailbox (type `gmail`) is read through Google's Gmail API, read only (`gmail.readonly`),
diff --git a/docs/sensors.md b/docs/sensors.md
index 809c4a41c..7d9eabcca 100644
--- a/docs/sensors.md
+++ b/docs/sensors.md
@@ -4,7 +4,7 @@ A **sensor** is anything that feeds Lens. There are two families, in one list (`
| Family | Kinds | Channels | What it reports becomes |
| --- | --- | --- | --- |
-| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP, Gmail and Google Workspace); calendar feeds (iCal); Google Calendar | the folders it watches | resources, run through their pipelines (as sources always have) |
+| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP, Gmail and Google Workspace); calendar feeds (iCal); Google Calendar; news feeds (RSS and Atom) | the folders it watches | resources, run through their pipelines (as sources always have) |
| Streams | MQTT devices, syslog senders, webhooks | streams: an MQTT topic, a syslog program, a webhook's stream name | readings, kept with retention, and hourly summaries of anything numeric |
File sensors are the storage sources: they are added and changed as sources (`/api/v1/sources`), and nothing about
diff --git a/fastapi_backend/app/api/v1/routes/imports.py b/fastapi_backend/app/api/v1/routes/imports.py
index 494e5525b..691ed9f94 100644
--- a/fastapi_backend/app/api/v1/routes/imports.py
+++ b/fastapi_backend/app/api/v1/routes/imports.py
@@ -34,7 +34,7 @@
router = APIRouter(prefix="/import", tags=["imports"])
-IMPORT_EXT = {".txt", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".json", ".jsonl", ".srt", ".vtt", ".eml", ".ics"}
+IMPORT_EXT = {".txt", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".json", ".jsonl", ".srt", ".vtt", ".eml", ".ics", ".atom"}
def _upload_name(filename: str | None) -> str:
diff --git a/fastapi_backend/app/domain/content_types.py b/fastapi_backend/app/domain/content_types.py
index 43850ac13..91d3c685e 100644
--- a/fastapi_backend/app/domain/content_types.py
+++ b/fastapi_backend/app/domain/content_types.py
@@ -55,6 +55,7 @@
("archive", "text", "Archive", "A zip or tar file, and the files in it", {"extensions": [".tar", ".tgz", ".zip"]}),
("email", "text", "Email", "A message, with its attachments", {"extensions": [".eml", ".msg"]}),
("calendar_event", "text", "Calendar event", "An invitation or a calendar entry", {"extensions": [".ics"]}),
+ ("feed_post", "text", "Feed post", "A post from a news feed (RSS or Atom)", {"extensions": [".atom"]}),
] # fmt: skip
diff --git a/fastapi_backend/app/domain/feeds.py b/fastapi_backend/app/domain/feeds.py
index bcbef94f9..1a2aff36a 100644
--- a/fastapi_backend/app/domain/feeds.py
+++ b/fastapi_backend/app/domain/feeds.py
@@ -1,6 +1,6 @@
-"""Sources that aren't file storage: IMAP mailboxes, Gmail and Google Workspace mailboxes, iCalendar feeds and Google
-Calendar (gcal.py), shown as folders of files so browsing, choosing files, watching and importing work for them as they
-do for rclone's storage (sources.py).
+"""Sources that aren't file storage: IMAP mailboxes, Gmail and Google Workspace mailboxes, iCalendar feeds, Google
+Calendar (gcal.py) and news feeds (RSS and Atom: rss.py), shown as folders of files so browsing, choosing files,
+watching and importing work for them as they do for rclone's storage (sources.py).
- An IMAP account's folders are its mailboxes, and each message is a file, `/-.eml`: a
UID stands for one message only while the mailbox's UIDVALIDITY stays the same, so a mailbox the server rebuilt
@@ -16,7 +16,7 @@
read as text: its title, when and where, who, and its description. The address is kept sealed, like a password
(a private calendar's address is all it takes to read it), and fetched as web pages are captured (netguard.py):
public addresses only, and the networks in documents.web_networks, on ports 80 and 443. Its password is never sent
- on to another server it redirects to.
+ on to another server it redirects to (`get`, which news feeds are fetched with too).
Both keep the names of their files (`name`), a `title` and `when` for the resource made from one, beside what rclone's
listings have (path, rel, size, modified).
@@ -38,10 +38,10 @@
import urllib.request
from email import policy
-from . import calendars, gcal, netguard
+from . import calendars, gcal, netguard, rss
# sources a module of their own handles: browse, list_files, fetch, immutable and test, as below
-OWN = {"gcal": gcal}
+OWN = {"gcal": gcal, "rss": rss}
TYPES = {
"imap": {
"label": "Email (IMAP)",
@@ -455,38 +455,50 @@ def _opener(guard):
return o
-def _download(cfg, src):
+def get(cfg, src, url, what, accept, limit, agent, headers=None, auth=True):
+ """GET a source's address (a calendar's, a news feed's) as web pages are captured: through netguard, to public
+ addresses and the networks in documents.web_networks only, on ports 80 and 443. The source's user and password go
+ with it (unless `auth` is False), never on to another server it redirects to; `headers` too (a conditional GET's
+ If-None-Match, say). Returns (status, the response's headers, its bytes, the address it ended at): a 304 answers
+ no bytes. RuntimeError, in words naming `what` ("calendar"), for what went wrong."""
from . import webcapture # the networks an admin allowed besides public ones; it imports what imports this
- url = _url(cfg, src)
- hit = _FEEDS.get(src["id"])
- if hit and hit[0] == url and time.monotonic() - hit[1] < FEED_SECONDS:
- return hit[2]
try:
webcapture.check_url(cfg, url)
except ValueError as e:
if "public address" in str(e):
host = urllib.parse.urlsplit(url).hostname
raise RuntimeError(
- f"that calendar can't be fetched: {host} is on a private network. An admin can allow that network in "
+ f"that {what} can't be fetched: {host} is on a private network. An admin can allow that network in "
"Settings → Documents (Private networks Lens may fetch from), e.g. 192.168.1.0/24"
) from None
- raise RuntimeError(f"that calendar can't be fetched: {e}") from None
- req = urllib.request.Request(url, headers={"User-Agent": "Lens calendar source", "Accept": "text/calendar, */*;q=0.5"})
- user, pw = _params(src).get("user", ""), _secret(cfg, src, "pass")
+ raise RuntimeError(f"that {what} can't be fetched: {e}") from None
+ req = urllib.request.Request(url, headers={"User-Agent": agent, "Accept": accept, **(headers or {})})
+ user, pw = (_params(src).get("user", ""), _secret(cfg, src, "pass")) if auth else ("", "")
if user or pw:
req.add_header("Authorization", "Basic " + base64.b64encode(f"{user}:{pw}".encode()).decode("ascii"))
- with netguard.Guard(forward=True, networks=webcapture.networks(cfg), max_bytes=MAX_CALENDAR + 65536) as guard:
+ with netguard.Guard(forward=True, networks=webcapture.networks(cfg), max_bytes=limit + 65536) as guard:
try:
with _opener(guard).open(req, timeout=60) as r:
- raw = r.read(MAX_CALENDAR + 1)
+ status, got, raw, final = r.status, r.headers, r.read(limit + 1), r.geturl()
except urllib.error.HTTPError as e:
- raise RuntimeError(f"the calendar's server answered {e.code} {e.reason}") from None
+ if e.code == 304:
+ return 304, e.headers, b"", url
+ raise RuntimeError(f"the {what}'s server answered {e.code} {e.reason}") from None
except (urllib.error.URLError, OSError) as e:
refused = f"; refused: {guard.refused[0]}" if guard.refused else ""
- raise RuntimeError(f"couldn't fetch the calendar: {getattr(e, 'reason', e)}{refused}") from None
- if len(raw) > MAX_CALENDAR:
- raise RuntimeError("the calendar is larger than 50 MB")
+ raise RuntimeError(f"couldn't fetch the {what}: {getattr(e, 'reason', e)}{refused}") from None
+ if len(raw) > limit:
+ raise RuntimeError(f"the {what} is larger than {limit // 2**20} MB")
+ return status, got, raw, final
+
+
+def _download(cfg, src):
+ url = _url(cfg, src)
+ hit = _FEEDS.get(src["id"])
+ if hit and hit[0] == url and time.monotonic() - hit[1] < FEED_SECONDS:
+ return hit[2]
+ _, _, raw, _ = get(cfg, src, url, "calendar", "text/calendar, */*;q=0.5", MAX_CALENDAR, "Lens calendar source")
text = raw.decode("utf-8", "replace")
if "BEGIN:VCALENDAR" not in text[:4096].upper():
raise RuntimeError("that address doesn't give a calendar (no BEGIN:VCALENDAR)")
diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py
index 7a333e6bd..2eb1bf7c5 100644
--- a/fastapi_backend/app/domain/ingest.py
+++ b/fastapi_backend/app/domain/ingest.py
@@ -859,8 +859,16 @@ def read_calendar(path):
return calendars.text(pathlib.Path(path).read_text(encoding="utf-8-sig", errors="replace"))
+def read_feed(path):
+ """A news feed's post (.atom, as a feed source keeps each), or a whole feed, as Markdown, a section per post (rss.py)."""
+ from . import rss
+
+ return rss.text(pathlib.Path(path).read_bytes())
+
+
DOC_READERS = {".docx": read_docx, ".doc": read_doc, ".pdf": read_pdf}
-MARKDOWN_READERS = {".eml": read_email, ".ics": read_calendar} # their subject or event title is the title
+# their subject, event title or post title is the title
+MARKDOWN_READERS = {".eml": read_email, ".ics": read_calendar, ".atom": read_feed}
FORMATS = ("auto", "text", "markdown", "mdx", "json", "jsonl", "srt", "vtt", "chat")
@@ -1037,7 +1045,7 @@ def _store_import(db, cfg, ns, t, title, fp, src, st, audio, speaker_names, engi
def import_transcript(db, cfg, ns, tpath, audio=None, title=None, speaker_names=None, fmt="auto", log=print, collection=None):
- """A transcript file (txt, md, mdx, docx, doc, pdf, json, jsonl, srt, vtt, eml, ics), optionally with its audio, into a
+ """A transcript file (txt, md, mdx, docx, doc, pdf, json, jsonl, srt, vtt, eml, ics, atom), optionally with its audio, into a
collection of the namespace (default: its default collection)."""
src = pathlib.Path(audio or tpath)
t = read_transcript(tpath, fmt)
diff --git a/fastapi_backend/app/domain/rss.py b/fastapi_backend/app/domain/rss.py
new file mode 100644
index 000000000..a8c0ded65
--- /dev/null
+++ b/fastapi_backend/app/domain/rss.py
@@ -0,0 +1,397 @@
+"""News feeds (RSS and Atom) as a source, a file sensor as feeds.py's calendar feeds are: a feed is one folder of
+posts, each a file `.atom` of its own, so browsing, choosing posts, watching and importing work as they do for any
+source. Watching a feed subscribes to it: each new post becomes a text resource of the namespace, titled and dated by
+the post, with its address, author and feed written in, so answers cite it.
+
+The address can be the feed's own or a page of the site that names its feed (``, as blogs and news sites do): the first feed the page names is read. feedparser (BSD)
+reads RSS 0.9x, 1.0 and 2.0 and Atom 0.3 and 1.0, drops scripts, styles and event handlers from the posts' HTML and
+makes their links absolute. Whatever the feed was, each post is written as an Atom entry document (RFC 4287
+§4.1.2): its id, title, the post's page and its enclosures (a podcast's audio), when it was published and updated, its
+authors, categories, summary and content, and the feed it came from (its title and site, never its address). `text`
+reads one as Markdown, as ingest.py reads emails and calendar events.
+
+The address is kept sealed, like a password (a private feed's address often carries its key), and fetched as calendar
+feeds are (feeds.get): public addresses only, and the networks in documents.web_networks, on ports 80 and 443; the
+password, for a feed that asks for one, is never sent to another server, a feed that a page names elsewhere included.
+A feed read is reused for FEED_SECONDS, then asked for again with its ETag and Last-Modified, so one that hasn't
+changed costs a small request and no parsing. Its posts stay in memory for KEEP_SECONDS (a scan reads the new ones
+right after listing them), its listing for as long as Lens runs: a post wanted later is read again, so many feeds
+don't fill a small machine's memory.
+
+A post's file is named by its id (an RSS item's guid), so it stays the same file however the feed reorders or retitles
+it. Its time is when it was published; a post whose text changes (its size) is read again into the resource it
+already is.
+"""
+
+from __future__ import annotations
+
+import calendar
+import datetime as dt
+import hashlib
+import io
+import re
+import time
+import urllib.parse
+import xml.etree.ElementTree as ET
+from html.parser import HTMLParser
+
+TYPE = {
+ "label": "News feed (RSS or Atom)",
+ "fields": {"user": ""},
+ "secrets": ["url", "pass"], # the address is kept like a password: a private feed's carries its key
+}
+MAX_FEED = 20 * 2**20
+MAX_POSTS = 500 # a feed's first posts (its newest, as feeds list them) that are listed
+FEED_SECONDS = 60 # a feed read for a listing is reused this long (a scan then reads each of its posts)
+KEEP_SECONDS = 300 # how long a feed's posts stay in memory after it was read (its listing stays)
+AGENT = "Lens feed reader"
+ACCEPT = "application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.9, */*;q=0.5"
+FEED_TYPES = ("application/rss+xml", "application/atom+xml", "application/rdf+xml")
+ATOM = "http://www.w3.org/2005/Atom"
+EPOCH = "1970-01-01T00:00:00+00:00" # for posts that don't say when: their size tells
+_XML_BAD = re.compile("[\x00-\x08\x0b\x0c\x0e-\x1f]")
+# source id -> the feed as last read: {url, found, at, read_at, etag, last_modified, title, site, entries, bodies}
+_FEEDS: dict = {}
+
+
+def _plain(v):
+ """Text on one line, without control characters."""
+ return " ".join(re.sub(r"[\x00-\x1f\x7f]", " ", str(v or "")).split())
+
+
+def _href(v):
+ v = _plain(v)
+ return v if urllib.parse.urlsplit(v).scheme in ("http", "https") else None
+
+
+def _xml(v):
+ return _XML_BAD.sub("", str(v))
+
+
+def _html_text(markup):
+ from . import mailbox # its reader of HTML mail bodies, a line per block
+
+ return mailbox.html_text(markup)
+
+
+def _url(cfg, src):
+ from . import feeds # it imports this
+
+ url = feeds._secret(cfg, src, "url").strip()
+ url = re.sub(r"^feed:(?=https?://)", "", url, flags=re.I) # feed:https://…, as some sites link their feeds
+ url = re.sub(r"^feed://", "https://", url, flags=re.I)
+ if not re.match(r"^https?://[^\s/]+\S*$", url, re.I):
+ raise RuntimeError("the feed's address starts with https:// (or feed://)")
+ return url
+
+
+def _server(url):
+ u = urllib.parse.urlsplit(url)
+ return u.hostname, u.port or (443 if u.scheme == "https" else 80)
+
+
+# ---------- reading a feed ----------
+class _Alternates(HTMLParser):
+ """The feeds a web page names: its (or Atom's)."""
+
+ def __init__(self):
+ super().__init__(convert_charrefs=True)
+ self.found = []
+
+ def handle_starttag(self, tag, attrs):
+ if tag != "link":
+ return
+ a = {k.lower(): (v or "").strip() for k, v in attrs}
+ kind = a.get("type", "").lower().split(";")[0].strip()
+ if "alternate" in a.get("rel", "").lower().split() and kind in FEED_TYPES and a.get("href"):
+ self.found.append(a["href"])
+
+
+def _named_feed(raw, base):
+ """The address of the first feed a page names, or None."""
+ p = _Alternates()
+ p.feed(raw[: 2 * 2**20].decode("utf-8", "replace"))
+ p.close()
+ for href in p.found:
+ url = urllib.parse.urljoin(base, href)
+ if urllib.parse.urlsplit(url).scheme in ("http", "https"):
+ return url
+ return None
+
+
+def _page(ctype, raw):
+ """Whether what came back is a web page rather than a feed (some servers call their feeds text/html)."""
+ head = raw[:1024].lstrip(b"\xef\xbb\xbf \t\r\n").lower()
+ if head.startswith((b" KEEP_SECONDS:
+ h["bodies"] = None
+ hit = _FEEDS.get(src["id"])
+ if hit and hit["url"] != url:
+ hit = None
+ whole = hit is not None and (hit["bodies"] is not None or not bodies) # what's kept is all that's wanted
+ if whole and now - hit["at"] < FEED_SECONDS:
+ return hit
+
+ def get(address, headers=None):
+ # the source's password goes only to the server its address is on
+ auth = _server(address) == _server(url)
+ return feeds.get(cfg, src, address, "feed", ACCEPT, MAX_FEED, AGENT, headers, auth=auth)
+
+ target = hit["found"] if hit else url
+ cond = {k: v for k, v in (("If-None-Match", hit["etag"]), ("If-Modified-Since", hit["last_modified"])) if v} if whole else {}
+ status, headers, raw, final = get(target, cond)
+ if status == 304 and whole:
+ hit["at"] = now
+ return hit
+ if _page(headers.get("Content-Type"), raw):
+ named = _named_feed(raw, final) if target == url else None
+ if not named:
+ raise RuntimeError(
+ "that address is a web page that names no news feed: give the feed's own address (it often ends in /feed, /rss or .xml)"
+ )
+ target = named
+ status, headers, raw, final = get(target)
+ if _page(headers.get("Content-Type"), raw):
+ raise RuntimeError("the feed that page names is a web page too, not a news feed")
+ feed = parse(raw, final, headers.get("Content-Type"))
+ out = {
+ "url": url,
+ "found": target,
+ "at": now,
+ "read_at": now,
+ "etag": headers.get("ETag"),
+ "last_modified": headers.get("Last-Modified"),
+ "title": feed["title"],
+ "site": feed["site"],
+ "entries": {k: e for k, (e, _) in feed["posts"].items()},
+ "bodies": {k: b for k, (_, b) in feed["posts"].items()},
+ }
+ _FEEDS[src["id"]] = out
+ return out
+
+
+def _feedparser(raw, headers=None):
+ import feedparser # loaded when a feed is read: a server without feeds never needs it
+
+ # from a stream: given a string, feedparser would fetch an address, or open a file of that name
+ return feedparser.parse(io.BytesIO(raw), response_headers=headers or {})
+
+
+def parse(raw, base, ctype=None):
+ """A feed's title, site and posts, {title, site, posts: {file: (entry, Atom entry document)}}, from its bytes;
+ `base` is the address it came from, which relative links are read against (without its query, so a private feed's
+ key stays out of what is kept)."""
+ u = urllib.parse.urlsplit(base)
+ base = urllib.parse.urlunsplit((u.scheme, u.netloc, u.path, "", ""))
+ d = _feedparser(raw, {"content-location": base, **({"content-type": ctype} if ctype else {})})
+ if not d.get("version") and not d.get("entries"):
+ raise RuntimeError("that address doesn't give a news feed (RSS or Atom)")
+ feed = d.get("feed") or {}
+ title = _plain(feed.get("title")) or u.hostname or "News feed"
+ site = _href(feed.get("link"))
+ feed_id = _plain(feed.get("id")) or site
+ posts = {}
+ for e in d.get("entries") or []:
+ if len(posts) >= MAX_POSTS:
+ break
+ p = _post(e)
+ if p is None:
+ continue
+ rel = f"{hashlib.sha1(p['id'].encode()).hexdigest()[:20]}.atom"
+ if rel in posts: # the same post twice: the first (newest) one
+ continue
+ body = _atom(p, title, site, feed_id)
+ day = (p["published"] or p["updated"] or "")[:10]
+ name = re.sub(r"[\x00-\x1f\x7f/\\]", "", f"{day} {p['title']}".strip())[:150].strip()
+ entry = {
+ "path": rel,
+ "rel": rel,
+ "name": f"{name or 'Post'}.atom",
+ "dir": False,
+ "size": len(body),
+ "modified": p["published"] or p["updated"] or EPOCH,
+ "title": p["title"] or None,
+ "when": p["published"] or p["updated"],
+ "settled": True, # a post is whole once its feed lists it: no waiting for it to settle
+ }
+ posts[rel] = (entry, body)
+ return {"title": title, "site": site, "posts": posts}
+
+
+def _iso(t):
+ try:
+ return dt.datetime.fromtimestamp(calendar.timegm(t), dt.UTC).isoformat(timespec="seconds") if t else None
+ except (OverflowError, ValueError, TypeError):
+ return None
+
+
+def _post(e):
+ """What a feed's entry says of a post, tidied: None when it has nothing to tell it by."""
+ links = [l for l in e.get("links") or [] if isinstance(l, dict)]
+ page = _href(e.get("link")) or next((_href(l.get("href")) for l in links if l.get("rel") == "alternate" and _href(l.get("href"))), None)
+ enclosures = [
+ {"href": _href(l.get("href")), "type": _plain(l.get("type"))[:100], "length": _plain(l.get("length"))[:20]}
+ for l in links
+ if l.get("rel") == "enclosure" and _href(l.get("href"))
+ ][:20]
+ title = e.get("title") or ""
+ if "html" in str((e.get("title_detail") or {}).get("type") or ""):
+ title = _html_text(title)
+ title = _plain(title)[:300]
+ content = next((c for c in e.get("content") or [] if isinstance(c, dict) and c.get("value")), {})
+ summary = e.get("summary") or ""
+ summary_type = str((e.get("summary_detail") or {}).get("type") or "text/html")
+ if summary == content.get("value"): # feedparser repeats an Atom entry's content as its summary
+ summary = ""
+ authors = []
+ for a in e.get("authors") or ([{"name": e.get("author")}] if e.get("author") else []):
+ if isinstance(a, dict) and (a.get("name") or a.get("email")):
+ authors.append({"name": _plain(a.get("name"))[:200], "email": _plain(a.get("email"))[:200], "uri": _href(a.get("href"))})
+ published, updated = _iso(e.get("published_parsed")), _iso(e.get("updated_parsed"))
+ text_of = content.get("value") or summary
+ pid = _plain(e.get("id")) or page
+ if not pid and (title or text_of):
+ pid = "sha1:" + hashlib.sha1(f"{title}\n{published}\n{text_of}".encode()).hexdigest()
+ if not pid:
+ return None
+ return {
+ "id": pid[:2000],
+ "title": title,
+ "page": page,
+ "enclosures": enclosures,
+ "published": published,
+ "updated": updated,
+ "authors": authors[:20],
+ "categories": [
+ c for c in dict.fromkeys(_plain(t.get("label") or t.get("term"))[:100] for t in e.get("tags") or [] if isinstance(t, dict)) if c
+ ][:30],
+ "summary": summary,
+ "summary_type": "html" if "html" in summary_type else "text",
+ "content": content.get("value") or "",
+ "content_type": "html" if "html" in str(content.get("type") or "text/html") else "text",
+ }
+
+
+def _atom(p, feed_title, site, feed_id):
+ """A post as an Atom entry document of its own (RFC 4287 §4.1.2), the feed it came from as its ."""
+
+ def add(parent, tag, text=None, **attrs):
+ el = ET.SubElement(parent, tag, {k: _xml(v) for k, v in attrs.items() if v})
+ if text:
+ el.text = _xml(text)
+ return el
+
+ root = ET.Element("entry", {"xmlns": ATOM})
+ add(root, "id", p["id"])
+ add(root, "title", p["title"] or "(no title)", type="text")
+ if p["page"]:
+ add(root, "link", rel="alternate", href=p["page"])
+ for x in p["enclosures"]:
+ add(root, "link", rel="enclosure", href=x["href"], type=x["type"], length=x["length"])
+ if p["published"]:
+ add(root, "published", p["published"])
+ add(root, "updated", p["updated"] or p["published"] or EPOCH)
+ for a in p["authors"]:
+ el = add(root, "author")
+ add(el, "name", a["name"] or a["email"])
+ if a["email"]:
+ add(el, "email", a["email"])
+ if a["uri"]:
+ add(el, "uri", a["uri"])
+ for c in p["categories"]:
+ add(root, "category", term=c)
+ if p["summary"]:
+ add(root, "summary", p["summary"], type=p["summary_type"])
+ if p["content"]:
+ add(root, "content", p["content"], type=p["content_type"])
+ source = add(root, "source")
+ if feed_id:
+ add(source, "id", feed_id)
+ add(source, "title", feed_title, type="text")
+ if site:
+ add(source, "link", rel="alternate", href=site)
+ ET.indent(root)
+ return b'\n' + ET.tostring(root, encoding="utf-8") + b"\n"
+
+
+# ---------- a post as text ----------
+def _who(a):
+ return f"{a['name']} ({a['email']})" if a["name"] and a["email"] and a["name"] != a["email"] else a["name"] or a["email"]
+
+
+def text(raw):
+ """A post's file, or any RSS or Atom feed (one section per post), as Markdown: the post's title as its heading,
+ 'Label — value' lines (its address, who wrote it, when, the feed, its categories and enclosures), then its text."""
+ d = _feedparser(raw)
+ feed = d.get("feed") or {}
+ out = []
+ for e in d.get("entries") or []:
+ p = _post(e)
+ if p is None:
+ continue
+ source = e.get("source") or {}
+ name, site = _plain(source.get("title") or feed.get("title")), _href(source.get("link") or feed.get("link"))
+ rows = [
+ ("Link", p["page"]),
+ ("By", ", ".join(_who(a) for a in p["authors"])),
+ ("Published", p["published"]),
+ ("Updated", p["updated"] if p["updated"] not in (p["published"], EPOCH) else None),
+ ("Feed", f"{name} ({site})" if name and site else name or site),
+ ("Categories", ", ".join(p["categories"])),
+ ("Enclosures", ", ".join(x["href"] + (f" ({x['type']})" if x["type"] else "") for x in p["enclosures"])),
+ ]
+ body = p["content"] or p["summary"]
+ kind = p["content_type"] if p["content"] else p["summary_type"]
+ body = _html_text(body) if kind == "html" else body.strip()
+ head = [f"{k} — {v}" for k, v in rows if v]
+ out.append("\n".join([f"# {p['title'] or '(no title)'}", "", *head, "", body]).strip())
+ if not out:
+ raise SystemExit("no posts found: that isn't a news feed or a post of one (RSS or Atom)")
+ return "\n\n".join(out) + "\n"
+
+
+# ---------- what feeds.py calls ----------
+def browse(cfg, src, path):
+ if path:
+ raise ValueError("a news feed has no folders")
+ return sorted(read(cfg, src)["entries"].values(), key=lambda e: e["when"] or "", reverse=True)
+
+
+def list_files(cfg, src, path, cursor=None):
+ return browse(cfg, src, path)
+
+
+def fetch(cfg, src, path):
+ body = read(cfg, src, bodies=True)["bodies"].get(path)
+ if body is None:
+ raise FileNotFoundError("that post isn't in the feed any more")
+ return body
+
+
+def immutable(src):
+ return False # a post can be edited
+
+
+def forget(src):
+ """Drop what was read of this source's feed, so the next call fetches it whole."""
+ _FEEDS.pop(src["id"], None)
+
+
+def test(cfg, src):
+ forget(src)
+ read(cfg, src)
diff --git a/fastapi_backend/app/domain/sources.py b/fastapi_backend/app/domain/sources.py
index 1b433fd15..8060f96d5 100644
--- a/fastapi_backend/app/domain/sources.py
+++ b/fastapi_backend/app/domain/sources.py
@@ -2,8 +2,8 @@
A source is one connection: S3 or S3-compatible, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, or a folder on
this machine (only inside sources.local_roots); or, without rclone, an email account (IMAP, or Gmail and Google
-Workspace), a calendar feed (iCal) or Google Calendar, whose messages and events are shown as files (feeds.py). A watch
-maps a folder on a source to a namespace: new audio, video, documents (PDFs) and images become resources queued for
+Workspace), a calendar feed (iCal), Google Calendar or a news feed (RSS or Atom), whose messages, events and posts are
+shown as files (feeds.py). A watch maps a folder on a source to a namespace: new audio, video, documents (PDFs) and images become resources queued for
the whole pipeline, their files staying on the source; new transcripts are imported and queued for analysis. A watch on an email account can carry routing rules that send some
messages to other namespaces, or skip them (mail_routes.py). Credentials are stored encrypted and handed to rclone in a private
temporary config file per call; OAuth tokens rclone refreshes are saved.
@@ -55,7 +55,8 @@
OBSCURED = {"pass"} # rclone wants these obscured in its config file
NOT_RCLONE = {"account"} # parameters Lens keeps for itself, not rclone options
SKIPPED = "not audio, video, a document, an image or text"
-TRANSCRIPT_EXT = {".txt", ".text", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".srt", ".vtt", ".json", ".jsonl", ".eml", ".ics"}
+TRANSCRIPT_EXT = {".txt", ".text", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".srt", ".vtt", ".json", ".jsonl"}
+TRANSCRIPT_EXT |= {".eml", ".ics", ".atom"} # emails, calendar events and news feeds' posts, read as text
# what a watched folder picks up (its `kinds`): the first three are from before documents, and read PDFs as transcripts
TAKES = {
"audio": {"audio"},
@@ -316,7 +317,12 @@ def cached_copy(db, cfg, sid, path, space=None):
tmp = dest.with_name(dest.name + ".part")
try:
if feeds.handles(src):
- tmp.write_bytes(feeds.fetch(cfg, src, p))
+ try:
+ tmp.write_bytes(feeds.fetch(cfg, src, p))
+ except FileNotFoundError:
+ if dest.exists(): # an event or post gone from its feed since: the copy fetched last
+ return dest
+ raise
else:
run(db, cfg, src, lambda n: ["copyto", f"{n}:{p}", str(tmp)], timeout=6 * 3600)
if encrypt:
diff --git a/fastapi_backend/app/schemas/sources.py b/fastapi_backend/app/schemas/sources.py
index 5e2aefa56..82b98d5c1 100644
--- a/fastapi_backend/app/schemas/sources.py
+++ b/fastapi_backend/app/schemas/sources.py
@@ -1,4 +1,4 @@
-"""Storage sources (where files live, through rclone; email accounts, Gmail and calendar feeds) and watched folders (admins)."""
+"""Storage sources (where files live, through rclone; email accounts, Gmail, calendar feeds and news feeds) and watched folders (admins)."""
from __future__ import annotations
@@ -8,7 +8,7 @@
from app.schemas.common import Created, Ok, RequestModel, ResponseModel
-SourceType = Literal["s3", "dropbox", "drive", "onedrive", "sftp", "smb", "webdav", "local", "imap", "gmail", "ical", "gcal"]
+SourceType = Literal["s3", "dropbox", "drive", "onedrive", "sftp", "smb", "webdav", "local", "imap", "gmail", "ical", "gcal", "rss"]
WatchKinds = Literal["audio", "transcripts", "both", "documents", "all"]
KINDS_HELP = (
"what to pick up: audio (and video), transcripts, both of those (PDFs read as text), documents (PDFs and "
diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml
index 3ed5dbad1..83f1d5705 100644
--- a/fastapi_backend/pyproject.toml
+++ b/fastapi_backend/pyproject.toml
@@ -25,6 +25,7 @@ dependencies = [
"opentelemetry-sdk>=1.27,<2",
"opentelemetry-exporter-otlp-proto-http>=1.45.1,<2",
"webauthn>=3.0.1,<4",
+ "feedparser>=6.0.11,<7",
]
[project.optional-dependencies]
diff --git a/fastapi_backend/tests/api/test_content_types.py b/fastapi_backend/tests/api/test_content_types.py
index 6588358af..9cf854e02 100644
--- a/fastapi_backend/tests/api/test_content_types.py
+++ b/fastapi_backend/tests/api/test_content_types.py
@@ -105,6 +105,7 @@ def sub(path, media=None, source="audio"):
assert sub("/in/meetup 2024.mp4")[1] == "video"
assert sub("/in/My Podcast Ep 12.mp3")[1] == "podcast"
assert sub("/in/invite.ics", source="document")[1] == "calendar_event"
+ assert sub("harbour news:9f2c.atom", source="transcript")[1] == "feed_post"
assert sub("/in/re: contract.eml", source="document")[1] == "email"
diff --git a/fastapi_backend/tests/api/test_news_feed_source.py b/fastapi_backend/tests/api/test_news_feed_source.py
new file mode 100644
index 000000000..ead2c63f0
--- /dev/null
+++ b/fastapi_backend/tests/api/test_news_feed_source.py
@@ -0,0 +1,283 @@
+"""News feeds (RSS and Atom) as a source: a feed's posts shown as files, browsed, watched and imported like any
+source's. Everything runs against a feed served over HTTP from this process."""
+
+from __future__ import annotations
+
+import base64
+import http.server
+import socket
+import threading
+
+import pytest
+
+from app.domain import content_types, feeds, netguard, rss, sources, store
+from tests.helpers import drain, login, make_user
+
+R = store.R
+
+POSTS = {
+ "night": (
+ "Night crossing",
+ "Tue, 29 Sep 2026 09:30:00 +0100",
+ "<p>The ships <script>steal()</script>arrived <a href='/pier' onclick='x()'>at dawn</a>."
+ "</p><p>The lighthouse keeper wrote twice about the fog.</p>",
+ ),
+ "buoys": ("New buoys", "Wed, 30 Sep 2026 14:00:00 +0000", "The new buoys are in, painted red."),
+ "regatta": ("Autumn regatta", "Thu, 01 Oct 2026 08:00:00 +0000", "Two days on the water. Bring the logbook."),
+}
+
+
+def _rss(keys, edits=None):
+ items = []
+ for k in keys:
+ title, date, text = POSTS[k]
+ text = (edits or {}).get(k, text)
+ items.append(
+ f"{title}https://harbour.example.org/posts/{k}harbour-{k}"
+ f"{date}Mara KeaneHarbour"
+ f"{text}"
+ )
+ return (
+ ""
+ "Harbour newshttps://harbour.example.org/From the pier"
+ + "".join(reversed(items))
+ + ""
+ )
+
+
+PAGE = """Harbour
+
+
Welcome to the harbour.
"""
+
+
+class _Site(http.server.BaseHTTPRequestHandler):
+ feed = ""
+ etag = '"v1"'
+ auth: str | None = None
+ feed_path = "/feed.xml?key=s3cret" # where the page says the feed is: a private feed's address carries its key
+ hits: list = []
+
+ def do_GET(self):
+ _Site.hits.append((self.path, self.headers.get("Host"), self.headers.get("Authorization"), self.headers.get("If-None-Match")))
+ if self.path == "/blog":
+ self._send(PAGE.format(feed=_Site.feed_path).encode(), "text/html; charset=utf-8")
+ elif self.path == "/plain":
+ self._send(b"No feed here", "text/html")
+ elif self.path.split("?")[0] == "/feed.xml":
+ if _Site.auth and self.headers.get("Authorization") != _Site.auth:
+ self.send_response(401)
+ self.end_headers()
+ elif self.headers.get("If-None-Match") == _Site.etag:
+ self.send_response(304)
+ self.end_headers()
+ else:
+ self._send(_Site.feed.encode(), "application/rss+xml", _Site.etag)
+ else:
+ self.send_response(404)
+ self.end_headers()
+
+ def _send(self, body, ctype, etag=None):
+ self.send_response(200)
+ self.send_header("Content-Type", ctype)
+ self.send_header("Content-Length", str(len(body)))
+ if etag:
+ self.send_header("ETag", etag)
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *a):
+ pass
+
+
+@pytest.fixture
+def site(cfg, monkeypatch):
+ srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Site)
+ threading.Thread(target=srv.serve_forever, daemon=True).start()
+ _Site.feed, _Site.etag, _Site.auth, _Site.feed_path, _Site.hits = _rss(["night", "buoys"]), '"v1"', None, "/feed.xml?key=s3cret", []
+ rss._FEEDS.clear()
+ monkeypatch.setattr(netguard, "PORTS", netguard.PORTS | {srv.server_address[1]}) # the test server's port, as if 443
+ cfg["documents"]["web_networks"] = ["127.0.0.0/8"] # a network an admin allowed: the test server's
+ yield f"http://127.0.0.1:{srv.server_address[1]}"
+ srv.shutdown()
+
+
+@pytest.fixture
+def admin(client, db):
+ make_user(db, "root@x.io", "root password 1", admin=True)
+ return login(client, "root@x.io", "root password 1")
+
+
+def _segments(db, rid):
+ return " ".join(s["text"] for s in db.rows("SELECT idx, text FROM segment WHERE recording = $r ORDER BY idx", r=rid))
+
+
+def _stale():
+ for hit in rss._FEEDS.values(): # past FEED_SECONDS: the next listing asks the server again
+ hit["at"] = 0
+
+
+def test_a_news_feed_as_a_source(site, client, db, cfg, admin): # site first: it sets what the app reads
+ h = admin
+ assert client.get("/api/v1/sources/backends", headers=h).json()["rss"]["label"] == "News feed (RSS or Atom)"
+ # the blog's page, not its feed: the feed it names is read
+ r = client.post("/api/v1/sources", headers=h, json={"name": "harbour", "type": "rss", "secrets": {"url": f"{site}/blog"}})
+ assert r.status_code == 200 and r.json()["health"]["ok"], r.text
+ sid = r.json()["id"]
+ src = client.get("/api/v1/sources", headers=h).json()[0]
+ assert (src["type"], src["label"], src["secrets"]["url"]) == ("rss", "News feed (RSS or Atom)", {"secret": True, "set": True})
+
+ entries = client.get(f"/api/v1/sources/{sid}/browse", headers=h).json()
+ assert [e["name"] for e in entries] == ["2026-09-30 New buoys.atom", "2026-09-29 Night crossing.atom"]
+ night = entries[1]
+ assert (night["title"], night["when"], night["modified"], night["dir"]) == (
+ "Night crossing",
+ "2026-09-29T08:30:00+00:00",
+ "2026-09-29T08:30:00+00:00",
+ False,
+ )
+ assert site not in client.get(f"/api/v1/sources/{sid}/browse", headers=h).text # the address stays sealed
+ assert client.get(f"/api/v1/sources/{sid}/browse", params={"path": "x"}, headers=h).status_code == 400
+
+ # a post's file is an Atom entry of its own: the feed's HTML cleaned, its links made absolute
+ atom = b"".join(sources.stream(db, cfg, sid, night["path"]))
+ assert atom.startswith(b'\n')
+ assert b"harbour-night" in atom and b'href="https://harbour.example.org/posts/night"' in atom
+ assert b"steal()" not in atom and b"onclick" not in atom and f"{site}/pier".encode() in atom # read against the feed's address
+ assert b'Harbour news' in atom and b"s3cret" not in atom # ... which isn't kept
+
+ # watched with backfill: every post becomes a text resource, titled and dated by the post
+ r = client.post("/api/v1/watches", headers=h, json={"source": sid, "namespace": "pods", "stable_seconds": 600, "backfill": True})
+ assert r.status_code == 200, r.text
+ wid = r.json()["id"]
+ assert client.post("/api/v1/watches/preview", headers=h, json={"source": sid}).json()["transcripts"] == 2
+ assert sources.poll_watch(db, cfg, wid)["new"] == 2 # a post is whole once listed: no waiting for it to settle
+ drain(db, cfg)
+ recs = {r["title"]: r for r in db.rows("SELECT record::id(id) AS id, title, source, recorded_at, status, remote, path FROM recording")}
+ assert set(recs) == {"Night crossing", "New buoys"}
+ one = recs["Night crossing"]
+ assert (one["source"], one["recorded_at"], one["status"]) == ("transcript", "2026-09-29T08:30:00+00:00", "analyzed")
+ assert one["remote"] == {"source": sid, "path": night["path"]} and one["path"] == f"harbour:{night['path']}"
+ said = _segments(db, one["id"])
+ assert "https://harbour.example.org/posts/night" in said and "Mara Keane" in said and "Harbour news" in said
+ assert "keeper wrote twice about the fog" in said and "steal" not in said
+ types = content_types.all_types(db)
+ rec = db.one("SELECT source, path, title, media FROM $r", r=R("recording", one["id"]))
+ assert content_types.recognise(db, rec, types)["key"] == "feed_post"
+
+ # asked again once the copy read is old: it hasn't changed (304), so nothing is new and nothing is parsed
+ _stale()
+ _Site.hits = []
+ assert sources.poll_watch(db, cfg, wid)["new"] == 0
+ assert [(p, etag) for p, _, _, etag in _Site.hits] == [("/feed.xml?key=s3cret", '"v1"')]
+ # a new post, and a post edited since
+ _Site.feed, _Site.etag = _rss(["night", "buoys", "regatta"], {"buoys": "The new buoys are in, painted red and white."}), '"v2"'
+ _stale()
+ assert sources.poll_watch(db, cfg, wid)["new"] == 2
+ drain(db, cfg)
+ titles = sorted(r["title"] for r in db.rows("SELECT title FROM recording"))
+ assert titles == ["Autumn regatta", "New buoys", "Night crossing"] # the edited post read into the one it was
+ assert "red and white" in _segments(db, recs["New buoys"]["id"])
+
+ # a post the feed has dropped since: its file is still the copy fetched last
+ _Site.feed, _Site.etag = _rss(["regatta"]), '"v3"'
+ _stale()
+ assert sources.poll_watch(db, cfg, wid)["new"] == 0
+ with pytest.raises(FileNotFoundError):
+ rss.fetch(cfg, sources.get(db, sid), night["path"])
+ assert b"Night crossing" in b"".join(sources.stream(db, cfg, sid, night["path"], space=store.ns_id(db, "pods")))
+
+ # chosen on purpose, from the listing: already there
+ regatta = client.get(f"/api/v1/sources/{sid}/browse", headers=h).json()[0]
+ r = client.post("/api/v1/import/source", headers=h, json={"source": sid, "paths": [regatta["path"]], "namespace": "pods"})
+ assert r.json()["results"][0]["status"] == "already"
+
+
+def test_feed_addresses(site, client, db, cfg, admin, monkeypatch):
+ with monkeypatch.context() as m:
+ m.setattr(feeds, "_secret", lambda cfg, src, k: src[k])
+ assert rss._url(cfg, {"url": "feed://harbour.example.org/rss"}) == "https://harbour.example.org/rss"
+ assert rss._url(cfg, {"url": "feed:https://harbour.example.org/rss"}) == "https://harbour.example.org/rss"
+ for bad in ("", "ftp://x/feed.xml", "/etc/passwd", "file:///etc/passwd", "https:///feed.xml"):
+ with pytest.raises(RuntimeError):
+ rss._url(cfg, {"url": bad})
+
+ def health(url, password=None, **params):
+ body = {"type": "rss", "params": params, "secrets": {"url": url, **({"pass": password} if password else {})}}
+ sid = client.post("/api/v1/sources", headers=admin, json=body).json()["id"]
+ rss._FEEDS.clear()
+ _Site.hits = []
+ return sources.test(db, cfg, sid) # with this cfg, as the test changes it
+
+ # what isn't public is refused, unless an admin allowed its network
+ cfg["documents"]["web_networks"] = []
+ refused = health(f"{site}/feed.xml")
+ assert not refused["ok"] and "127.0.0.1 is on a private network" in refused["error"] and "Settings → Documents" in refused["error"]
+ cfg["documents"]["web_networks"] = ["127.0.0.0/8"]
+ assert health(f"{site}/feed.xml")["ok"]
+ # a page that names no feed, and an address that is no feed at all
+ plain = health(f"{site}/plain")
+ assert not plain["ok"] and "names no news feed" in plain["error"]
+ assert "404" in health(f"{site}/missing.xml")["error"]
+
+ # a feed that asks for a password gets it; a feed the page names on another server never does
+ _Site.auth = "Basic " + base64.b64encode(b"lens:feed password 1").decode()
+ assert health(f"{site}/blog", "feed password 1", user="lens")["ok"]
+ assert [a for p, _, a, _ in _Site.hits if p.startswith("/feed.xml")][-1] == _Site.auth
+ real = socket.getaddrinfo # where localhost is ::1 too (Ubuntu), only its IPv4 address: the test server's
+ monkeypatch.setattr(
+ netguard.socket,
+ "getaddrinfo",
+ lambda host, *a, **k: real(host, *a, **({**k, "family": socket.AF_INET} if host == "localhost" else k)),
+ )
+ _Site.feed_path = site.replace("127.0.0.1", "localhost") + "/feed.xml"
+ elsewhere = health(f"{site}/blog", "feed password 1", user="lens")
+ assert not elsewhere["ok"] and "401" in elsewhere["error"]
+ assert [(h.split(":")[0], a) for p, h, a, _ in _Site.hits] == [("127.0.0.1", _Site.auth), ("localhost", None)]
+
+
+def test_posts_leave_memory(site, client, db, cfg, admin):
+ """A feed's posts don't stay in memory: its listing does, and a post wanted after that reads the feed in full."""
+ r = client.post("/api/v1/sources", headers=admin, json={"name": "harbour", "type": "rss", "secrets": {"url": f"{site}/feed.xml"}})
+ sid = r.json()["id"]
+ first = client.get(f"/api/v1/sources/{sid}/browse", headers=admin).json()[0]
+ (hit,) = rss._FEEDS.values()
+ assert set(hit["bodies"]) == set(hit["entries"])
+ hit["read_at"] -= rss.KEEP_SECONDS + 1
+ _Site.hits = []
+ assert client.get(f"/api/v1/sources/{sid}/browse", headers=admin).json()[0] == first # the listing, from memory
+ assert hit["bodies"] is None and _Site.hits == []
+ # a post wanted now: read in full, without asking whether it changed (a 304 would bring no posts)
+ assert b'New buoys' in rss.fetch(cfg, sources.get(db, sid), first["path"])
+ assert [(p, etag) for p, _, _, etag in _Site.hits] == [("/feed.xml", None)]
+ (hit,) = rss._FEEDS.values()
+ assert set(hit["bodies"]) == set(hit["entries"])
+
+
+def test_what_a_feed_holds_stays_data(tmp_path):
+ """An entity naming a file of this machine is never read in, and a post's HTML keeps no scripts."""
+ secret = tmp_path / "secret.txt"
+ secret.write_text("TOP SECRET")
+ feed = (
+ f"]>"
+ "Harbour newsLeak &leak;x"
+ "a <img src=x onerror=steal()> &leak; b"
+ ).encode()
+ posts = rss.parse(feed, "https://harbour.example.org/feed.xml")["posts"]
+ kept = b"".join(body for _, body in posts.values()) + "".join(e["name"] for e, _ in posts.values()).encode()
+ assert b"TOP SECRET" not in kept and b"onerror" not in kept
+ assert b"TOP SECRET" not in rss.text(feed).encode()
+
+
+def test_a_feed_file_imported_as_text(client, db, admin):
+ atom = b"""
+Harbour newsurn:harbour2026-10-01T08:00:00Z
+
+Autumn regattaurn:harbour:regatta2026-10-01T08:00:00Z
+<p>Two days on the water.</p>
+"""
+ r = client.post(
+ "/api/v1/import", headers=admin, json={"namespace": "pods", "filename": "harbour.atom", "data": base64.b64encode(atom).decode()}
+ )
+ assert r.status_code == 200, r.text
+ said = _segments(db, r.json()["id"])
+ assert "Two days on the water." in said and "https://harbour.example.org/posts/regatta" in said
diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock
index 085a4bfec..94f1d063f 100644
--- a/fastapi_backend/uv.lock
+++ b/fastapi_backend/uv.lock
@@ -1080,6 +1080,27 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/05/99/49ee85903dee060d9f08297b4a342e5e0bcfca2f027a07b4ee0a38ab13f9/faster_whisper-1.2.1-py3-none-any.whl", hash = "sha256:79a66ad50688c0b794dd501dc340a736992a6342f7f95e5811be60b5224a26a7", size = 1118909, upload-time = "2025-10-31T11:35:47.794Z" },
]
+[[package]]
+name = "feedparser"
+version = "6.0.14"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "feedparser-sgmllib" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/37/8a/a53da4a77352045d277978a2df322d5379369f9deb1707178899ff7e1121/feedparser-6.0.14.tar.gz", hash = "sha256:088679b0c4b543ee211a820dd544698c76a402122eae7473c04a43425f283d06", size = 286108, upload-time = "2026-07-30T14:07:40.491Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7f/61/f04912e63702e73fb2a378f9c0a1ad9eb17a334a11a6b3fe1daa593903c2/feedparser-6.0.14-py3-none-any.whl", hash = "sha256:e35e3f760151b0c3b22cac9684155cae186a233e16c49bcbc6c49e91e3131137", size = 80668, upload-time = "2026-07-30T14:07:39.175Z" },
+]
+
+[[package]]
+name = "feedparser-sgmllib"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/da/df/38596299216e5c22d60ed7f97902bb2bc72cfb95f732400f4fa976fd2e62/feedparser_sgmllib-2.1.0.tar.gz", hash = "sha256:61facf2918c4389b5b00714f76c5e03431ffcd94cd1f51d657edd6cd7c396579", size = 26845, upload-time = "2026-08-02T21:27:53.891Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/85/a0/79a31f898092e145bd66e2b338fb0656979acb2bbbcae8220940fbfcd820/feedparser_sgmllib-2.1.0-py3-none-any.whl", hash = "sha256:2cab2d43b95a954f920f18aebce7a4dbbb3f539780b127e2aa114f579821e01d", size = 11652, upload-time = "2026-08-02T21:27:52.894Z" },
+]
+
[[package]]
name = "filelock"
version = "4.0.7"
@@ -1809,6 +1830,7 @@ dependencies = [
{ name = "cryptography" },
{ name = "fastapi", extra = ["standard"] },
{ name = "fastapi-mail" },
+ { name = "feedparser" },
{ name = "jinja2" },
{ name = "markdown" },
{ name = "numpy" },
@@ -1916,6 +1938,7 @@ requires-dist = [
{ name = "fastapi", extras = ["standard"], specifier = ">=0.142.2,<0.143" },
{ name = "fastapi-mail", specifier = ">=1.4.1,<2" },
{ name = "faster-whisper", marker = "extra == 'whisper'" },
+ { name = "feedparser", specifier = ">=6.0.11,<7" },
{ name = "funasr", marker = "extra == 'sensevoice'" },
{ name = "impacket", marker = "extra == 'smb'", specifier = ">=0.13" },
{ name = "insightface", marker = "extra == 'insightface'" },
diff --git a/nextjs-frontend/__tests__/sources-model.test.ts b/nextjs-frontend/__tests__/sources-model.test.ts
index 8aba1334a..8502d41fd 100644
--- a/nextjs-frontend/__tests__/sources-model.test.ts
+++ b/nextjs-frontend/__tests__/sources-model.test.ts
@@ -219,6 +219,27 @@ describe("connections", () => {
expect(suggestName("ical", cal.params)).toBe("Calendar feed");
expect([fileKind("Harbour report.eml"), fileKind("2026-10-02 Stand-up.ics")]).toEqual(["document", "transcript"]);
});
+ it("takes news feeds by their address, or their site's", () => {
+ const RSS: BackendSpec = { label: "News feed (RSS or Atom)", fields: { user: "" }, secrets: ["url", "pass"] };
+ expect(sourceSubtitle({ type: "rss", params: { user: "" } })).toBe("News feed · address kept secret");
+ expect(validateForm("rss", RSS, emptyForm(RSS), false)).toHaveProperty("url");
+ const feed = { ...emptyForm(RSS), secrets: { url: "https://harbour.example.org/feed.xml?key=s3cret" } };
+ expect(validateForm("rss", RSS, feed, false)).toEqual({});
+ for (const ok of [
+ "feed://harbour.example.org/rss",
+ "feed:https://harbour.example.org/rss",
+ "http://harbour.example.org",
+ ])
+ expect(validateForm("rss", RSS, { ...feed, secrets: { url: ok } }, false)).toEqual({});
+ expect(validateForm("rss", RSS, { ...feed, secrets: { url: "harbour.example.org" } }, false)).toHaveProperty("url");
+ expect(validateForm("rss", RSS, emptyForm(RSS), true, (k) => k === "url")).toEqual({});
+ expect(buildPayload(RSS, feed, "rss").secrets).toEqual({ url: "https://harbour.example.org/feed.xml?key=s3cret" });
+ // named by its server, never by the rest of its address
+ expect(suggestName("rss", feed.params, feed.secrets)).toBe("News feed · harbour.example.org");
+ expect(suggestName("rss", {}, { url: "feed:https://news.example.org/atom" })).toBe("News feed · news.example.org");
+ expect(suggestName("rss", {})).toBe("News feed");
+ expect(fileKind("2026-09-29 Night crossing.atom")).toBe("transcript");
+ });
it("signs Google Calendar in as a Google account or a service account", () => {
const GCAL: BackendSpec = {
label: "Google Calendar",
diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts
index 0166ad4ce..fe3136ce9 100644
--- a/nextjs-frontend/app/openapi-client/types.gen.ts
+++ b/nextjs-frontend/app/openapi-client/types.gen.ts
@@ -14006,6 +14006,7 @@ export type RcloneSignInStart = {
| "gmail"
| "ical"
| "gcal"
+ | "rss"
| null;
/**
* Source
@@ -17875,7 +17876,19 @@ export type Source = {
* Type
*/
type:
- "s3" | "dropbox" | "drive" | "onedrive" | "sftp" | "smb" | "webdav" | "local" | "imap" | "gmail" | "ical" | "gcal";
+ | "s3"
+ | "dropbox"
+ | "drive"
+ | "onedrive"
+ | "sftp"
+ | "smb"
+ | "webdav"
+ | "local"
+ | "imap"
+ | "gmail"
+ | "ical"
+ | "gcal"
+ | "rss";
/**
* Label
*/
@@ -17920,7 +17933,19 @@ export type SourceCreate = {
* Type
*/
type:
- "s3" | "dropbox" | "drive" | "onedrive" | "sftp" | "smb" | "webdav" | "local" | "imap" | "gmail" | "ical" | "gcal";
+ | "s3"
+ | "dropbox"
+ | "drive"
+ | "onedrive"
+ | "sftp"
+ | "smb"
+ | "webdav"
+ | "local"
+ | "imap"
+ | "gmail"
+ | "ical"
+ | "gcal"
+ | "rss";
/**
* Params
*/
diff --git a/nextjs-frontend/components/library/source-labels.ts b/nextjs-frontend/components/library/source-labels.ts
index a0fc17ea5..304aeb7d2 100644
--- a/nextjs-frontend/components/library/source-labels.ts
+++ b/nextjs-frontend/components/library/source-labels.ts
@@ -11,6 +11,8 @@ export const SOURCE_TYPE_LABEL: Record = {
imap: "Email",
gmail: "Gmail",
ical: "Calendar",
+ gcal: "Google Calendar",
+ rss: "News feed",
};
export function sourceTypeLabel(type: string | null | undefined, fallback = "Source"): string {
diff --git a/nextjs-frontend/components/library/sources-strip.tsx b/nextjs-frontend/components/library/sources-strip.tsx
index daadf2e49..e314de38e 100644
--- a/nextjs-frontend/components/library/sources-strip.tsx
+++ b/nextjs-frontend/components/library/sources-strip.tsx
@@ -10,6 +10,7 @@ import {
ListOrdered,
Mail,
Network,
+ Rss,
Server,
type LucideIcon,
} from "lucide-react";
@@ -33,6 +34,8 @@ const ICON: Record = {
imap: Mail,
gmail: Mail,
ical: CalendarDays,
+ gcal: CalendarDays,
+ rss: Rss,
};
type ChipState = { meta: string; tone: "muted" | "red"; tip: string };
diff --git a/nextjs-frontend/components/sources/connection-dialog.tsx b/nextjs-frontend/components/sources/connection-dialog.tsx
index f0ffaa0f6..6965ccc12 100644
--- a/nextjs-frontend/components/sources/connection-dialog.tsx
+++ b/nextjs-frontend/components/sources/connection-dialog.tsx
@@ -161,7 +161,7 @@ export function ConnectionDialog({
const body = buildPayload(backends[t], f, t);
sent = body.secrets;
if (id == null) {
- name = f.name.trim() || suggestName(t, f.params);
+ name = f.name.trim() || suggestName(t, f.params, f.secrets);
const r = await data(
Sources.createSource({
client,
@@ -241,7 +241,7 @@ export function ConnectionDialog({
let id = createdId;
const body = buildPayload(backends[type], form, type);
if (id == null) {
- const name = form.name.trim() || suggestName(type, form.params);
+ const name = form.name.trim() || suggestName(type, form.params, form.secrets);
id = (await data(Sources.createSource({ client, body: { type, name, ...body } }))).id;
setCreatedId(id);
setCreatedName(name);
diff --git a/nextjs-frontend/components/sources/connection-fields.tsx b/nextjs-frontend/components/sources/connection-fields.tsx
index 128c2152a..6e7db5189 100644
--- a/nextjs-frontend/components/sources/connection-fields.tsx
+++ b/nextjs-frontend/components/sources/connection-fields.tsx
@@ -566,6 +566,23 @@ export function ConnectionFields({
);
+ case "rss":
+ return (
+
+ {secret("url", {
+ label: "Feed address",
+ hint: "The RSS or Atom feed’s address, or the site’s: Lens finds the feed a page names. Kept like a password, since a private feed’s address carries its key",
+ })}
+
+ {text("user", { optional: true })}
+ {secret("pass", { label: "Password", hint: "Only if the feed asks for one" })}
+
+
+ Each post becomes text: its title, link, author, date and what the feed says of it. Watch the feed to bring
+ in new posts as they’re published.
+
+
+ );
case "gmail": {
const keyText = form.secrets.service_account_credentials;
const key = keyText ? parseServiceAccountKey(keyText) : null;
diff --git a/nextjs-frontend/components/sources/source-icons.tsx b/nextjs-frontend/components/sources/source-icons.tsx
index aeb15f226..0f02ac99a 100644
--- a/nextjs-frontend/components/sources/source-icons.tsx
+++ b/nextjs-frontend/components/sources/source-icons.tsx
@@ -10,6 +10,7 @@ import {
type LucideIcon,
Mail,
Network,
+ Rss,
Server,
} from "lucide-react";
@@ -28,6 +29,7 @@ export const TYPE_ICON: Record = {
gmail: Mail,
ical: CalendarDays,
gcal: CalendarCheck,
+ rss: Rss,
};
/** The connection's type in a 32px rounded tile. */
diff --git a/nextjs-frontend/components/sources/source-model.ts b/nextjs-frontend/components/sources/source-model.ts
index 27e109d1d..3fd4f5a26 100644
--- a/nextjs-frontend/components/sources/source-model.ts
+++ b/nextjs-frontend/components/sources/source-model.ts
@@ -20,6 +20,7 @@ export const TYPE_ORDER: SourceType[] = [
"gmail",
"ical",
"gcal",
+ "rss",
];
/** Short names for the type picker and rows (the backend's labels are longer). */
@@ -36,6 +37,7 @@ export const TYPE_NAME: Record = {
gmail: "Gmail",
ical: "Calendar feed",
gcal: "Google Calendar",
+ rss: "News feed",
};
/** How to name the storage itself in sentences ("Files in the bucket aren’t touched"). */
@@ -52,6 +54,7 @@ export const STORAGE_NOUN: Record = {
gmail: "the mailbox",
ical: "the calendar",
gcal: "Google Calendar",
+ rss: "the feed",
};
/** Field labels for the backend's parameter and secret names. */
@@ -170,6 +173,8 @@ export function sourceSubtitle(s: Pick): string {
return `Gmail · ${str(p.user) || "no address"} · ${str(p.via) === "imap" ? "IMAP" : "API"}`;
case "ical":
return "Calendar feed · address kept secret";
+ case "rss":
+ return "News feed · address kept secret";
case "gcal": {
const chosen = str(p.calendars)
.split(/[,\s]+/)
@@ -271,7 +276,7 @@ const AUDIO = [
".m4v",
".avi",
];
-const TRANSCRIPT = [".srt", ".vtt", ".json", ".jsonl", ".ics"];
+const TRANSCRIPT = [".srt", ".vtt", ".json", ".jsonl", ".ics", ".atom"];
const DOCUMENT = [
".pdf",
".doc",
@@ -323,7 +328,8 @@ export type SourceFileKind = "audio" | "transcript" | "document" | "image" | "ot
/**
* What a watched folder would make of a file, by extension (the backend's default lists): PDFs, Office files, text,
* Markdown, web pages and emails are documents, as they are for new watched folders (those set to audio and
- * transcripts read PDFs, Word and text files as transcripts); subtitles, JSON and calendar events are transcripts.
+ * transcripts read PDFs, Word and text files as transcripts); subtitles, JSON, calendar events and feed posts are
+ * transcripts.
*/
export function fileKind(name: string): SourceFileKind {
const n = name.toLowerCase();
@@ -526,10 +532,20 @@ export function emptyForm(spec: BackendSpec, source?: Pick): string {
+/** A name for a new connection, from what was typed ("SFTP · calls-gw", "S3 · eu-central-1", "Dropbox", "News feed ·
+ * harbour.example.org"). A feed's or calendar's address is a secret, but its server names it well. */
+export function suggestName(
+ type: SourceType,
+ params: Record,
+ secrets: Record = {},
+): string {
if (type === "gmail") return params.user?.trim() ? `Gmail · ${params.user.trim()}` : "Gmail";
- const host = (params.host || params.url || "").replace(/^(https?|webcals?):\/\//i, "").split(/[/:]/)[0];
+ const address = params.host || params.url || (type === "rss" || type === "ical" ? (secrets.url ?? "") : "");
+ const host = address
+ .trim()
+ .replace(/^feed:(?=https?:)/i, "")
+ .replace(/^(https?|webcals?|feed):\/\//i, "")
+ .split(/[/:?#]/)[0];
const extra =
type === "s3" ? [params.provider !== "AWS" ? params.provider : "", params.region].filter(Boolean).join(" ") : host;
return extra
@@ -605,6 +621,11 @@ export function validateForm(
if (url === undefined ? !(editing && isSet("url")) : !/^(https?|webcals?):\/\/\S+$/i.test(url.trim()))
e.url = "Enter the calendar’s address, starting with https:// or webcal://";
}
+ if (type === "rss") {
+ const url = form.secrets.url;
+ if (url === undefined ? !(editing && isSet("url")) : !/^(feed:)?(https?|feed):\/\/\S+$/i.test(url.trim()))
+ e.url = "Enter the feed’s address, or the site’s, starting with https://";
+ }
return e;
}
diff --git a/nextjs-frontend/components/storage/file-store.ts b/nextjs-frontend/components/storage/file-store.ts
index deb765794..fe906eae6 100644
--- a/nextjs-frontend/components/storage/file-store.ts
+++ b/nextjs-frontend/components/storage/file-store.ts
@@ -1,7 +1,7 @@
/** Where Lens keeps its own files (Settings → Storage, the backend's `files` section: app/domain/blobs.py). */
/** Connections files can't be kept on: email accounts and calendars are read, not written (feeds.py). */
-export const NOT_STORAGE = new Set(["imap", "gmail", "ical"]);
+export const NOT_STORAGE = new Set(["imap", "gmail", "ical", "gcal", "rss"]);
export type FileStoreTry =
{ store: "local"; crypt: false } | { store: "connection"; connection: number | null; folder: string; crypt: boolean };
diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json
index 0a6d6d491..65dd920ca 100644
--- a/nextjs-frontend/openapi.json
+++ b/nextjs-frontend/openapi.json
@@ -52791,7 +52791,8 @@
"imap",
"gmail",
"ical",
- "gcal"
+ "gcal",
+ "rss"
]
},
{
@@ -59482,7 +59483,8 @@
"imap",
"gmail",
"ical",
- "gcal"
+ "gcal",
+ "rss"
],
"title": "Type"
},
@@ -59579,7 +59581,8 @@
"imap",
"gmail",
"ical",
- "gcal"
+ "gcal",
+ "rss"
],
"title": "Type"
},
diff --git a/threat-model.json b/threat-model.json
index 8e9bd0477..4c518a719 100644
--- a/threat-model.json
+++ b/threat-model.json
@@ -66,11 +66,12 @@
{
"name": "outbound",
"entry_points": [
- "watched folders, storage sources, IMAP, Gmail, iCal",
+ "watched folders, storage sources, IMAP, Gmail, iCal, news feeds",
"Google Calendar sources",
+ "News feed sources",
"Connect with Google",
"POST /imports/web, extension http tools, notification targets",
- "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, MQTT bridge, OIDC issuer)",
+ "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer)",
"replies from remote servers",
".env, archive.yaml, LENS_*, ARCHIVE_SECRET_KEY, RCLONE_BINARY"
],
@@ -183,7 +184,7 @@
},
{
"component": "outbound",
- "boundary": "B4, Lens to the outside: fetches whose URL a non-admin sets, and iCal feeds, go through netguard; other admin-set URLs do not; the telemetry endpoint is only shape-checked (P-ADMIN-GUARD).",
+ "boundary": "B4, Lens to the outside: fetches whose URL a non-admin sets, and iCal and news feeds, go through netguard; other admin-set URLs do not; the telemetry endpoint is only shape-checked (P-ADMIN-GUARD).",
"provenance": "inferred",
"reachability_precondition": "a URL a non-admin sets, or a guarded admin operand, reaches a non-public address; or a stored secret reaches anyone but the host the admin set",
"source": "threat-model.md §1.4 (untagged boundary list)"
@@ -399,8 +400,8 @@
"source": "docs/configuration.md \"Uploads accept only the types\""
},
{
- "entry_point": "watched folders, storage sources, IMAP, Gmail, iCal",
- "parameter": "remote file names, bytes, mail, events",
+ "entry_point": "watched folders, storage sources, IMAP, Gmail, iCal, news feeds",
+ "parameter": "remote file names, bytes, mail, events, feed posts",
"attacker_controllable": "yes",
"component": "outbound",
"caller_must_enforce": "admin creates sources",
@@ -416,6 +417,15 @@
"provenance": "documented",
"source": "docs/processing.md \"The token endpoint is always Google's own\""
},
+ {
+ "entry_point": "News feed sources",
+ "parameter": "the feed's address (or a site's page that names its feed), user and password; the feed's XML and its posts' HTML",
+ "attacker_controllable": "yes",
+ "component": "outbound",
+ "caller_must_enforce": "admin creates sources; fetched as iCal feeds are; feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post kept as an Atom entry Lens writes, without the feed's address; post text is marked untrusted in prompts",
+ "provenance": "documented",
+ "source": "docs/processing.md \"News feeds (RSS and Atom)\""
+ },
{
"entry_point": "Connect with Google",
"parameter": "OAuth client id and secret; on the way back state, code and the email in Google's id token",
@@ -435,7 +445,7 @@
"source": "maintainer ruling, 2026-10"
},
{
- "entry_point": "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, MQTT bridge, OIDC issuer)",
+ "entry_point": "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer)",
"parameter": "URL, credentials, api_key_env",
"attacker_controllable": "no",
"component": "outbound",
@@ -983,7 +993,7 @@
"property": "P-SECRETS",
"violation_symptom": "info-leak",
"severity_tier": "security",
- "conditions": "stored secrets (source credentials, notification secrets, model API keys, iCal addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server",
+ "conditions": "stored secrets (source credentials, notification secrets, model API keys, iCal and news feed addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server",
"provenance": "documented",
"source": "SECURITY.md \"stored credentials for sources and model servers leaking\""
},
@@ -999,7 +1009,7 @@
"property": "P-ADMIN-GUARD",
"violation_symptom": "info-leak",
"severity_tier": "security",
- "conditions": "guards Lens documents on admin-set operands hold: iCal feeds are fetched like web captures (public addresses only and documents.web_networks, ports 80 and 443, the password never goes on to a server the calendar redirects to); the telemetry endpoint cannot carry a user name or password or be a link-local address",
+ "conditions": "guards Lens documents on admin-set operands hold: iCal and news feeds are fetched like web captures (public addresses only and documents.web_networks, ports 80 and 443, the password never goes on to a server the calendar or feed redirects to, or to a feed a site's page names on another server); the telemetry endpoint cannot carry a user name or password or be a link-local address",
"provenance": "documented",
"source": "docs/processing.md \"fetched the way web pages are captured\""
},
@@ -1246,7 +1256,7 @@
},
{
"property": "N-EGRESS",
- "reason": "an egress firewall; netguard guards web import, iCal feeds, extension http tools, notifications and Chromium, nothing else; admin-set URLs other than iCal feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel) are unguarded",
+ "reason": "an egress firewall; netguard guards web import, iCal and news feeds, extension http tools, notifications and Chromium, nothing else; admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel) are unguarded",
"false_friend": true,
"provenance": "documented",
"source": "maintainer ruling, 2026-10"
@@ -1616,7 +1626,7 @@
},
{
"pattern": "Wide LENS_WEB_NETWORKS",
- "why_unsafe": "Editors, and iCal feed redirects, can then reach those networks",
+ "why_unsafe": "Editors, and iCal and news feed redirects, can then reach those networks",
"instead": "List single hosts"
},
{
diff --git a/threat-model.md b/threat-model.md
index 5a6edd772..011750fdc 100644
--- a/threat-model.md
+++ b/threat-model.md
@@ -171,7 +171,7 @@ flowchart LR
- **B3, content to parser.** Uploaded and fetched bytes cross into ffmpeg, LibreOffice, Chromium, poppler, Tesseract,
anytopdf (here, or on an admin-set conversion node, P-CONVERSION-NODE), an admin-set Apache Tika server
(P-TIKA-SERVER) and the LLM prompt.
-- **B4, Lens to the outside.** Fetches whose URL a non-admin sets, and iCal feeds, go through `netguard`. Other
+- **B4, Lens to the outside.** Fetches whose URL a non-admin sets, and iCal and news feeds, go through `netguard`. Other
admin-set URLs do not; the telemetry endpoint is only shape-checked (P-ADMIN-GUARD).
- **B5, peers.** Workers, the API and the web app trust the database and each other. Workers hold the DB credential.
- **B6, devices.** Sensor listeners take traffic from the LAN (or wider, per `sensors.bind`) once an admin enables them.
@@ -283,12 +283,13 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34.
| `/namespaces/{name}/sparql` | SPARQL text | yes, namespace readers | callback/code | none | *(documented, routes/rdf.py sparql docstring "SERVICE and FROM aren't allowed")* |
| `/namespaces/{name}/rdf/import` | Turtle, N-Triples, JSON-LD | yes, editors | serialized state, size | none | *(documented, rdf_import.py docstring "RDF/XML isn't read")* |
| `/uploads`, `/import`, `/recordings/{rid}/files` | file bytes, name, extension, size | yes, editors | data, type/class, size | none | *(documented, docs/configuration.md "Uploads accept only the types")* |
-| watched folders, storage sources, IMAP, Gmail, iCal | remote file names, bytes, mail, events | yes, whoever writes to the remote, mails the box or serves the feed | data, resource-name, size | admin creates sources | *(inferred, Q17)* |
+| watched folders, storage sources, IMAP, Gmail, iCal, news feeds | remote file names, bytes, mail, events, feed posts | yes, whoever writes to the remote, mails the box or serves the feed | data, resource-name, size | admin creates sources | *(inferred, Q17)* |
| Google Calendar sources (`gcal.py`, `google_auth.py`) | a service account key file or a connected account's refresh token, calendar ids, `impersonate`; the events Google returns | key and token: admins only, sealed and write-only (P-SECRETS); events: yes, anyone who can invite the account to an event or edit a calendar it reads | data, x-credential, x-prompt-injection, size | admin creates sources; read-only scope (`calendar.readonly`); the token endpoint and API address are Google's own (a key file's `token_uri` is ignored) and calendar ids are quoted into the path; event text is written into its .ics escaped and without control characters, so an event can't add properties; events are data, as iCal feeds' are | *(documented, docs/processing.md "The token endpoint is always Google's own")* |
+| News feed sources (`rss.py`) | the feed's address (or a site's page that names its feed), a user and password; the feed's XML and its posts' HTML | address and password: admins only, sealed and write-only (P-SECRETS); XML and HTML: yes, whoever serves the feed or writes its posts | data, x-prompt-injection, size | admin creates sources; fetched as iCal feeds are (P-ADMIN-GUARD); feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post is kept as an Atom entry Lens writes (escaped), without the feed's address; post text reaches prompts as marked, untrusted data | *(documented, docs/processing.md "News feeds (RSS and Atom)")* |
| Connect with Google (`POST /sources/{id}/google`, then `/auth/external/google/callback`) | an OAuth client id and secret; on the way back `state`, `code` and the email in Google's id token | client: admins only; `state` and `code`: yes, anyone who can send the admin a link | data, x-credential, x-collaborator-data | admin only, from the web app (API tokens refused), and still an admin when Google sends them back; the flow reuses external sign-in's: `state` is single-use, kept only hashed, expires in 10 minutes and must come back to the browser that started it (cookie), with PKCE (S256); the client secret is sealed on the flow and the refresh token on the source; the token is kept only when Google granted the source's scope (read only, but all of Gmail for a Gmail source read over IMAP, which Google offers no read-only scope for; Lens still only reads; all of Drive for a Drive connection set to full access), and a Drive connection keeps its client's secret sealed with it, for rclone to renew the token; the email is only shown, and comes straight from Google's token endpoint; the browser returns only to the web origin that started it | *(documented, docs/processing.md "the consent itself is a person's")* |
| the assistant's mailbox (`mailbox.py`) | mail headers, text, attachments, forwarded mail, `Authentication-Results` | yes, anyone who knows the address; From is forgeable | data, x-prompt-injection, size | answered only for `mailbox.users` whose provider vouched for From; forwarded text marked untrusted, and changes after it ask first | *(documented, docs/assistant-email.md "it answers only mail that is both")* |
| `POST /imports/web`, extension `http` tools, notification targets | URL | yes: editors (web, tools), owners (targets) | resource-name, x-network-destination | none | *(maintainer, 2026-10)* |
-| admin settings: model, embeddings, speech, decisions, Fedora, bridge, Telegram bot, Slack app tokens, assistant mailbox, telemetry, IIIF import, rclone remotes, IMAP, Gmail (a Google refresh token or a Workspace service account key), iCal, MQTT bridge, OIDC issuer | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* |
+| admin settings: model, embeddings, speech, decisions, Fedora, bridge, Telegram bot, Slack app tokens, assistant mailbox, telemetry, IIIF import, rclone remotes, IMAP, Gmail (a Google refresh token or a Workspace service account key), iCal, news feeds, MQTT bridge, OIDC issuer | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* |
| replies from remote servers | web page HTML and scripts, IIIF manifests, model replies, OIDC discovery, calendars and their redirects | yes, the remote | data, topology | treat as untrusted | *(inferred, Q17)* |
| chat, `/graph/ask`, MCP tools | prompt text; model-chosen tool calls | yes; also indirectly through content | data, x-prompt-injection | none; what people say to Lens, in the web app or a chat app, reaches automations as outside text (`chat.message`, `lens.untrusted`) | *(documented, docs/assistant.md "limited to what the person can read")* |
| Lens extension side panel | answers shown there (model text that can repeat third-party text), the sign-in coming back | yes, indirectly through content | data, x-prompt-injection | none; answers render as Markdown cleaned by DOMPurify (no scripts, images or forms; only http and https links), and the sign-in checks its `state` and PKCE | *(documented, extensions/browser/src/core/assistant.js answerHtml docstring "cleaned of anything that could run or fetch on its own")* |
@@ -359,16 +360,16 @@ Every row is claimed, disclaimed, N/A with a reason, or unresolved. Family short
| AuthZ | x-client-address: compose | claimed | both compose files trust the web app container, so each visitor gets their own address | P-IPGROUP, P-THROTTLE | *(documented, docs/deployment.md "so each visitor gets their own")* |
| AuthZ | x-client-address: other variants | unresolved | web app alone, installers, NAS, Proxmox without a reverse proxy setting `X-Forwarded-For` | Q2 | *(inferred, Q2)* |
| AuthZ | x-share-scope | unresolved | `?s=` share tokens open the audio, video, frames and page API routes | Q7 | *(inferred, Q7)* |
-| Outbound | Numeric domain | claimed | web paths and iCal: ports 80 and 443 | P-SSRF-GUARD, P-ADMIN-GUARD | *(documented, docs/api.md "Only public addresses are reached, on ports 80 and 443")* |
+| Outbound | Numeric domain | claimed | web paths, iCal and news feeds: ports 80 and 443 | P-SSRF-GUARD, P-ADMIN-GUARD | *(documented, docs/api.md "Only public addresses are reached, on ports 80 and 443")* |
| Outbound | Numeric domain: notification ports | unresolved | notification targets may use any port; the guard checks addresses only | Q15 | *(inferred, Q15)* |
| Outbound | Failure atomicity | claimed | each notification event claimed once across processes, then retried | P-NOTIFY | *(documented, docs/notifications.md "claimed once")* |
| Outbound | Recursive / cyclic topology (redirects) | claimed | http and https only, each hop re-checked; credentials never follow a redirect to another server; notifications and an automation's `http_request` follow none | P-SSRF-GUARD, P-SECRETS | *(documented, docs/processing.md "never sent on to another server")* |
| Outbound | Callback / collaborator execution | disclaimed | Chromium may lack its sandbox; the proxy is the barrier | N-CHROMIUM | *(documented, docs/deployment.md "what stops a page is Lens's proxy")* |
| Outbound | Serialization / reconstruction | claimed | sealed secrets are write-only through the API | P-SECRETS | *(documented, docs/configuration.md "the API reports whether one is set, never its value")* |
-| Outbound | x-admin-operand-guards | claimed | iCal public-address guard; telemetry endpoint without user:password or link-local address | P-ADMIN-GUARD | *(documented, docs/processing.md "fetched the way web pages are captured")* |
+| Outbound | x-admin-operand-guards | claimed | iCal and news feed public-address guard; telemetry endpoint without user:password or link-local address | P-ADMIN-GUARD | *(documented, docs/processing.md "fetched the way web pages are captured")* |
| Outbound | x-admin-operand-guards: telemetry headers | unresolved | saved headers dropped when the endpoint moves to another host | Q35 | *(inferred, Q35)* |
| Outbound | x-remote-identity: IMAP | claimed | a rebuilt mailbox gets new names, so an old name never reads another message | P-REMOTE-NAMES | *(documented, docs/processing.md "an old one never reads another message")* |
-| Outbound | x-remote-identity: other sources | unresolved | rclone, iCal and IIIF cache keys and names | Q43 | *(inferred, Q43)* |
+| Outbound | x-remote-identity: other sources | unresolved | rclone, iCal, news feed and IIIF cache keys and names | Q43 | *(inferred, Q43)* |
| Outbound | Reference lifecycle: key placement | unresolved | `secret.key` in `data_dir` with data and backups | Q14 | *(inferred, Q14)* |
| Outbound | Reference lifecycle: key rotation | claimed | a new key makes stored credentials and encrypted files unreadable | §1.13 | *(documented, docs/deployment.md "Stable secrets")* |
| Outbound | Concurrency / reentrancy | unresolved | loopback guard proxy open to local processes during a capture | Q19 | *(inferred, Q19)* |
@@ -509,9 +510,9 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c
| P-HOST | The `Host` header must be a served host. This stops DNS rebinding. | bad-data-accepted | correctness-only | *(documented, docs/configuration.md "stops DNS rebinding")* | Voided by: `*` in `server.allowed_hosts` — `core/middleware.py:70`; `ARCHIVE_ALLOWED_HOSTS` — `domain/settings.py:129`. |
| P-FRAME | The API sends a strict CSP. Only the player (`/embed/`, `/s/`) can be framed, and only by `server.embed_frame_ancestors`. | integrity-bypass | security-critical | *(documented, docs/configuration.md "can be framed")* | Voided by: `server.embed_frame_ancestors` — `core/middleware.py:72` (default `'self'`, `domain/store.py:185`). |
| P-HARDENED | In the compose files the database, API and Fedora ports bind to 127.0.0.1, containers run non-root with `no-new-privileges`, and the web app sends `nosniff`, a referrer policy, a permissions policy and a frame policy (HSTS through Cloudflare). In `docker-compose.prod.yml` the DB password is random and the web app listens on this machine unless `LENS_BIND` is set. | x-missing-hardening | correctness-only | *(documented, docs/deployment.md "What's hardened already")* | Voided by: `LENS_BIND=0.0.0.0` — `docker-compose.prod.yml:153`, which an admin can also ask for from Settings (`PUT /api/v1/admin/install`); dev web port — `docker-compose.yml:95` (§1.3). |
-| P-SECRETS | Stored secrets (source credentials, notification secrets, model API keys, the Hugging Face token, iCal addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only. The API reports whether one is set, never its value. A key from the environment is never shown. They do not leak through the API, logs, exports, or a redirect to another server. | info-leak | security-critical | *(documented, SECURITY.md "stored credentials for sources and model servers leaking")* | Voided by: `secret.key` or `ARCHIVE_SECRET_KEY` holders — `domain/settings.py:196-199` (Q14). An admin moving a model URL sends the key there — `domain/llm.py:30-32` (trusted). LLM errors name the URL as typed — `domain/llm.py:41` (Q18). |
+| P-SECRETS | Stored secrets (source credentials, notification secrets, model API keys, the Hugging Face token, iCal and news feed addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only. The API reports whether one is set, never its value. A key from the environment is never shown. They do not leak through the API, logs, exports, or a redirect to another server. | info-leak | security-critical | *(documented, SECURITY.md "stored credentials for sources and model servers leaking")* | Voided by: `secret.key` or `ARCHIVE_SECRET_KEY` holders — `domain/settings.py:196-199` (Q14). An admin moving a model URL sends the key there — `domain/llm.py:30-32` (trusted). LLM errors name the URL as typed — `domain/llm.py:41` (Q18). |
| P-SSRF-GUARD | URLs a non-admin sets (web import and extension `http` tools by editors; notification targets by owners; an automation's `http_request` by its person, within the notification networks) reach public addresses only, or networks the operator or an admin listed (an admin, in Settings, never loopback or link-local ones). Web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address. An automation's `http_request` follows no redirect, takes 10 seconds and reads 64 KB at most, asks first unless the person set its node to run on its own, and passes the answer on as outside text. | info-leak | security-critical | *(maintainer, 2026-10)* | Voided by: Operator networks — `domain/store.py:475-476`, `domain/netguard.py:40-41`; admin web networks — `domain/settings.py` `_web_networks`; admin notification networks — `domain/notify.py:286-288`; Chromium without sandbox — `domain/convert.py:273`. Search: `grep -rlE 'public_ip\|netguard\.(resolve\|Guard)\|web_networks' fastapi_backend/app` — `netguard.py`, `webcapture.py`, `feeds.py`, `extensions.py`, `notify.py`, `convert.py`, `store.py`, `routes/imports.py`; none disables the check. |
-| P-ADMIN-GUARD | Guards Lens documents on admin-set operands hold. iCal feeds are fetched like web captures: public addresses only (and `documents.web_networks`), ports 80 and 443, and the password never goes on to a server the calendar redirects to *(documented, docs/processing.md "fetched the way web pages are captured")*. The telemetry endpoint cannot carry a user name or password or be a link-local address *(documented, domain/settings.py validation message "can't be a link-local address")*. | info-leak | security-critical | *(documented, docs/processing.md "fetched the way web pages are captured")* | Voided by: `documents.web_networks` widens the iCal guard — `domain/feeds.py:377`; telemetry checks — `domain/settings.py:751`, `domain/settings.py:757`. |
+| P-ADMIN-GUARD | Guards Lens documents on admin-set operands hold. iCal and news feeds are fetched like web captures: public addresses only (and `documents.web_networks`), ports 80 and 443, and the password never goes on to a server the calendar or feed redirects to, or to a feed a site's page names on another server *(documented, docs/processing.md "fetched the way web pages are captured")*. The telemetry endpoint cannot carry a user name or password or be a link-local address *(documented, domain/settings.py validation message "can't be a link-local address")*. | info-leak | security-critical | *(documented, docs/processing.md "fetched the way web pages are captured")* | Voided by: `documents.web_networks` widens the iCal and news feed guard — `domain/feeds.py:377`; telemetry checks — `domain/settings.py:751`, `domain/settings.py:757`. |
| P-NO-FETCH | SPARQL refuses SERVICE and FROM and never loads a named graph. RDF import fetches nothing, refuses remote JSON-LD contexts and does not read RDF/XML. | info-leak | security-critical | *(documented, rdf_import.py docstring "Nothing is fetched while reading")* | Voided by: Process-global rdflib flag, set per call — `domain/rdf.py:576`; SERVICE refused — `domain/rdf.py:551`; remote contexts — `domain/rdf_import.py:54-56`. |
| P-QUERY | Cypher is read-only and runs on the caller's readable namespaces. A query stops within 3 million steps or 8 seconds; paths run up to 8 hops. Threshold: running past the budget is a bug; slow within it is not. | integrity-bypass (writes, scope); hang (budget) | security-critical (scope, writes); correctness-only (budget) | *(documented, docs/graph.md "3 million steps")* | Voided by: Admins see every namespace; write refusals — `domain/cypher.py:39`; budgets — `domain/cypher.py:35-38`. |
| P-RETRIEVAL | Chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at `ai.max_steps` and `ai.max_transcript_reads`. A namespace's chat completions answer from that namespace alone, with only the read tools its owners offered. | info-leak (scope); hang (caps) | security-critical (scope); correctness-only (caps) | *(documented, docs/assistant.md "limited to what the person can read")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. |
@@ -557,7 +558,7 @@ Exported by phase 3.6 (backtest) and de-identified. Closing rows carry their sta
| ID | The project does not provide | Conditions / boundary | Tier | False friend? | Provenance |
| --- | --- | --- | --- | --- | --- |
| N-ADMIN | Protection from the instance admin or host operator, including admin-set URLs reaching internal hosts, admin code, templates, regexes, component installs and `api_key_env` | Covers only operands an admin or operator alone can set. Excludes the guards Lens documents on them (P-ADMIN-GUARD): an escape of one is `VALID`. Also excludes what happens to saved telemetry headers when the endpoint moves to another host; that row is unresolved (Q35), so such a report routes `MODEL-GAP`, not a close. A non-admin reaching any admin operand is in model. | security-critical | no | *(maintainer, 2026-10)* |
-| N-EGRESS | An egress firewall. `netguard` guards web import, iCal feeds, extension `http` tools, notifications and Chromium; nothing else | Admin-set URLs other than iCal feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel) are unguarded | security-critical | yes: "public addresses only" reads as server-wide | *(maintainer, 2026-10)* |
+| N-EGRESS | An egress firewall. `netguard` guards web import, iCal and news feeds, extension `http` tools, notifications and Chromium; nothing else | Admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel) are unguarded | security-critical | yes: "public addresses only" reads as server-wide | *(maintainer, 2026-10)* |
| N-CODE-GUARD | A sandbox for Python code tools. Audit hooks, rlimits and an optional network namespace "raise the bar" only | Admin-written code; runs as the Lens user | security-critical | yes: guard is not isolation | *(documented, domain/code_tools.py docstring "they aren't a wall")* |
| N-DOWNLOADS | Integrity beyond HTTPS and the publisher for llama.cpp, cloudflared, pip installs and `hf:` models | Catalog GGUF files are still checked (P-MODEL-CHECKSUM) | security-critical | no | *(maintainer, 2026-10)* |
| N-DB-PLAIN | Encryption of the SurrealDB content (assets' text, indexes), derived files, renditions, reports, exports and the storage-source cache | Stated limit | security-critical | yes: vaults cover files only | *(documented, docs/encryption.md "the SurrealDB database itself")* |
@@ -659,7 +660,7 @@ The operator (and, for the last rows, integrators that consume Lens output) must
| Enabling a teammate's shared extension without reading it | Its `http` tool can send your readable data to its author's host | Enable only extensions you trust (Q27) |
| Turning on the chat-room bridge without its own account | The room reads as the admin | Give the bridge a low-role account |
| Making someone admin so they can write a Python tool | Admin equals host code execution | Use prompt, graph or `http` tools |
-| Wide `LENS_WEB_NETWORKS` | Editors, and iCal feed redirects, can then reach those networks | List single hosts |
+| Wide `LENS_WEB_NETWORKS` | Editors, and iCal and news feed redirects, can then reach those networks | List single hosts |
| Webhook push token in a URL that proxies log | The token is a credential in the path | Use the header form |
| Pointing the assistant at a hosted model for sensitive archives | Assets' text leaves the machine | Use a local model server |
diff --git a/threat-model.yaml b/threat-model.yaml
index 237d01d42..81d3e5fb9 100644
--- a/threat-model.yaml
+++ b/threat-model.yaml
@@ -2,7 +2,7 @@
# Authority order: prose > yaml > json. Regenerate whenever the prose changes.
schema: threat-model-sidecar/v2
project: lens
-prose_version: "threat-model.md@sha256:a396f25c854ef342e1d269daf00bed3ac90f3346087c8a02c56deb8875878e00"
+prose_version: "threat-model.md@sha256:5f1925aee911e138d2d84f329939a63a0474306cd5844b6798746964b60e7610"
model_status: under-review
triage_policy: strict
confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8}
@@ -321,10 +321,10 @@ entry_points:
control_kinds: [data, type-class, size]
caller_must_enforce: "none — P-UPLOAD"
provenance: {kind: documented, source: 'docs/configuration.md "Uploads accept only the types"'}
- - id: "watched folders, storage sources, IMAP, Gmail, iCal"
+ - id: "watched folders, storage sources, IMAP, Gmail, iCal, news feeds"
component: outbound
parameters:
- - name: "remote file names, bytes, mail, events"
+ - name: "remote file names, bytes, mail, events, feed posts"
attacker_controllable: true
x-condition: "whoever writes to the remote, mails the box or serves the feed"
control_kinds: [data, resource-name, size]
@@ -341,6 +341,16 @@ entry_points:
obligation_id: admin-creates-sources
caller_must_enforce: "admin creates sources; read-only calendar.readonly scope; Google's own token endpoint and API address (a key file's token_uri is ignored); event text escaped into its .ics without control characters"
provenance: {kind: documented, source: 'docs/processing.md "The token endpoint is always Google''s own"'}
+ - id: "News feed sources"
+ component: outbound
+ parameters:
+ - name: "the feed's address (or a site's page that names its feed), user and password; the feed's XML and its posts' HTML"
+ attacker_controllable: true
+ x-condition: "address and password: admins only, sealed and write-only; XML and HTML: whoever serves the feed or writes its posts"
+ control_kinds: [data, x-prompt-injection, size]
+ obligation_id: admin-creates-sources
+ caller_must_enforce: "admin creates sources; fetched as iCal feeds are; feedparser reads the XML with external entities off and drops scripts, styles and event handlers from a post's HTML; each post kept as an Atom entry Lens writes, without the feed's address; post text is marked untrusted in prompts"
+ provenance: {kind: documented, source: 'docs/processing.md "News feeds (RSS and Atom)"'}
- id: "Connect with Google"
component: outbound
parameters:
@@ -360,7 +370,7 @@ entry_points:
control_kinds: [resource-name, x-network-destination]
caller_must_enforce: "none — P-SSRF-GUARD"
provenance: {kind: maintainer, date: "2026-10"}
- - id: "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, MQTT bridge, OIDC issuer)"
+ - id: "admin settings (model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, Gmail sign-in, iCal, news feeds, MQTT bridge, OIDC issuer)"
component: outbound
parameters:
- name: "URL, credentials, api_key_env"
@@ -579,16 +589,16 @@ contract_dimensions:
- {component: authz, dimension: x-client-address-other-variants, status: unresolved, conditions: "web app alone, installers, NAS, Proxmox without a reverse proxy setting X-Forwarded-For", question_id: Q2, provenance: {kind: inferred, question_id: Q2}}
- {component: authz, dimension: x-share-scope, status: unresolved, conditions: "?s= share tokens open the audio, video, frames and page API routes", question_id: Q7, provenance: {kind: inferred, question_id: Q7}}
# --- Outbound
- - {component: outbound, dimension: numeric-domain, status: claimed, property_id: P-SSRF-GUARD, conditions: "web paths and iCal: ports 80 and 443", x-routes-to: "P-SSRF-GUARD, P-ADMIN-GUARD", provenance: {kind: documented, source: 'docs/api.md "Only public addresses are reached, on ports 80 and 443"'}}
+ - {component: outbound, dimension: numeric-domain, status: claimed, property_id: P-SSRF-GUARD, conditions: "web paths, iCal and news feeds: ports 80 and 443", x-routes-to: "P-SSRF-GUARD, P-ADMIN-GUARD", provenance: {kind: documented, source: 'docs/api.md "Only public addresses are reached, on ports 80 and 443"'}}
- {component: outbound, dimension: x-numeric-domain-notification-ports, status: unresolved, conditions: "notification targets may use any port; the guard checks addresses only", question_id: Q15, provenance: {kind: inferred, question_id: Q15}}
- {component: outbound, dimension: failure-atomicity, status: claimed, property_id: P-NOTIFY, conditions: "each notification event claimed once across processes, then retried", provenance: {kind: documented, source: 'docs/notifications.md "claimed once"'}}
- {component: outbound, dimension: recursive-cyclic-topology, status: claimed, property_id: P-SSRF-GUARD, conditions: "redirects: http and https only, each hop re-checked; credentials never follow a redirect to another server; notifications and an automation's http_request follow none", x-routes-to: "P-SSRF-GUARD, P-SECRETS", provenance: {kind: documented, source: 'docs/processing.md "never sent on to another server"'}}
- {component: outbound, dimension: callback-execution, status: disclaimed, property_id: N-CHROMIUM, conditions: "Chromium may lack its sandbox; the proxy is the barrier", provenance: {kind: documented, source: 'docs/deployment.md "what stops a page is Lens''s proxy"'}}
- {component: outbound, dimension: serialization-reconstruction, status: claimed, property_id: P-SECRETS, conditions: "sealed secrets are write-only through the API", provenance: {kind: documented, source: 'docs/configuration.md "the API reports whether one is set, never its value"'}}
- - {component: outbound, dimension: x-admin-operand-guards, status: claimed, property_id: P-ADMIN-GUARD, conditions: "iCal public-address guard; telemetry endpoint without user:password or link-local address", provenance: {kind: documented, source: 'docs/processing.md "fetched the way web pages are captured"'}}
+ - {component: outbound, dimension: x-admin-operand-guards, status: claimed, property_id: P-ADMIN-GUARD, conditions: "iCal and news feed public-address guard; telemetry endpoint without user:password or link-local address", provenance: {kind: documented, source: 'docs/processing.md "fetched the way web pages are captured"'}}
- {component: outbound, dimension: x-admin-operand-guards-telemetry-headers, status: unresolved, conditions: "saved headers dropped when the endpoint moves to another host", question_id: Q35, provenance: {kind: inferred, question_id: Q35}}
- {component: outbound, dimension: x-remote-identity-imap, status: claimed, property_id: P-REMOTE-NAMES, conditions: "a rebuilt mailbox gets new names, so an old name never reads another message", provenance: {kind: documented, source: 'docs/processing.md "an old one never reads another message"'}}
- - {component: outbound, dimension: x-remote-identity-other-sources, status: unresolved, conditions: "rclone, iCal and IIIF cache keys and names", question_id: Q43, provenance: {kind: inferred, question_id: Q43}}
+ - {component: outbound, dimension: x-remote-identity-other-sources, status: unresolved, conditions: "rclone, iCal, news feed and IIIF cache keys and names", question_id: Q43, provenance: {kind: inferred, question_id: Q43}}
- {component: outbound, dimension: reference-lifecycle, status: unresolved, conditions: "key placement: secret.key in data_dir with data and backups", question_id: Q14, provenance: {kind: inferred, question_id: Q14}}
- {component: outbound, dimension: x-reference-lifecycle-key-rotation, status: claimed, property_id: MX-OUTBOUND-KEY-ROTATION, conditions: "a new key makes stored credentials and encrypted files unreadable", x-routes-to: "§1.13", provenance: {kind: documented, source: 'docs/deployment.md "Stable secrets"'}}
- {component: outbound, dimension: concurrency-reentrancy, status: unresolved, conditions: "loopback guard proxy open to local processes during a capture", question_id: Q19, provenance: {kind: inferred, question_id: Q19}}
@@ -1411,7 +1421,7 @@ properties_claimed:
kind: confidentiality
components: [outbound]
tier: security-critical
- conditions: "stored secrets (source credentials, notification secrets, model API keys, iCal addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server"
+ conditions: "stored secrets (source credentials, notification secrets, model API keys, iCal and news feed addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server"
violation_symptoms: [info-leak]
x-voided-by: "secret.key or ARCHIVE_SECRET_KEY holders (domain/settings.py:196-199, Q14); an admin moving a model URL sends the key there (domain/llm.py:30-32, trusted); LLM errors name the URL as typed (domain/llm.py:41, Q18)"
provenance: {kind: documented, source: 'SECURITY.md "stored credentials for sources and model servers leaking"'}
@@ -1428,9 +1438,9 @@ properties_claimed:
components: [outbound]
tier: security-critical
x-tier-question: Q37
- conditions: "guards Lens documents on admin-set operands hold: iCal feeds are fetched like web captures (public addresses only and documents.web_networks, ports 80 and 443, the password never goes on to a server the calendar redirects to); the telemetry endpoint cannot carry a user name or password or be a link-local address"
+ conditions: "guards Lens documents on admin-set operands hold: iCal and news feeds are fetched like web captures (public addresses only and documents.web_networks, ports 80 and 443, the password never goes on to a server the calendar or feed redirects to, or to a feed a site's page names on another server); the telemetry endpoint cannot carry a user name or password or be a link-local address"
violation_symptoms: [info-leak]
- x-voided-by: "documents.web_networks widens the iCal guard (domain/feeds.py:377); telemetry checks (domain/settings.py:751, :757)"
+ x-voided-by: "documents.web_networks widens the iCal and news feed guard (domain/feeds.py:377); telemetry checks (domain/settings.py:751, :757)"
provenance: {kind: documented, source: 'docs/processing.md "fetched the way web pages are captured"'}
- id: P-NO-FETCH
kind: confidentiality
@@ -1673,7 +1683,7 @@ properties_claimed:
# ---------------------------------------------------------------- §1.12
properties_disclaimed:
- {id: N-ADMIN, components: [outbound, ingest, assistant, optional], tier: security-critical, false_friend: false, conditions: "protection from the instance admin or host operator (admin-set URLs reaching internal hosts, admin code, templates, regexes, component installs, api_key_env). Covers only operands an admin or operator alone can set. Excludes the guards Lens documents on them (P-ADMIN-GUARD): an escape of one is VALID. Excludes saved telemetry headers when the endpoint moves host (unresolved, Q35 -> MODEL-GAP). A non-admin reaching any admin operand is in model.", provenance: {kind: maintainer, date: "2026-10"}}
- - {id: N-EGRESS, components: [outbound], tier: security-critical, false_friend: true, conditions: "an egress firewall; netguard guards web import, iCal feeds, extension http tools, notifications and Chromium, nothing else; admin-set URLs other than iCal feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel) are unguarded", x-false-friend-note: "'public addresses only' reads as server-wide", provenance: {kind: maintainer, date: "2026-10"}}
+ - {id: N-EGRESS, components: [outbound], tier: security-critical, false_friend: true, conditions: "an egress firewall; netguard guards web import, iCal and news feeds, extension http tools, notifications and Chromium, nothing else; admin-set URLs other than iCal and news feeds (model, speech, Fedora, IIIF import, rclone sources, IMAP, MQTT bridge, OIDC, telemetry, tunnel) are unguarded", x-false-friend-note: "'public addresses only' reads as server-wide", provenance: {kind: maintainer, date: "2026-10"}}
- {id: N-CODE-GUARD, components: [assistant], tier: security-critical, false_friend: true, conditions: "a sandbox for Python code tools; audit hooks, rlimits and an optional network namespace only raise the bar; admin-written code; runs as the Lens user", x-false-friend-note: "guard is not isolation", provenance: {kind: documented, source: 'domain/code_tools.py docstring "they aren''t a wall"'}}
- {id: N-DOWNLOADS, components: [optional], tier: security-critical, false_friend: false, conditions: "integrity beyond HTTPS and the publisher for llama.cpp, cloudflared, pip installs and hf: models; catalog GGUF files are still checked (P-MODEL-CHECKSUM)", provenance: {kind: maintainer, date: "2026-10"}}
- {id: N-DB-PLAIN, components: [outbound], tier: security-critical, false_friend: true, conditions: "encryption of the SurrealDB content (assets' text, indexes), derived files, renditions, reports, exports and the storage-source cache; stated limit", x-false-friend-note: "vaults cover files only", provenance: {kind: documented, source: 'docs/encryption.md "the SurrealDB database itself"'}}
@@ -1762,7 +1772,7 @@ known_misuses:
- {id: km-enable-unread-extension, components: [assistant], pattern: "Enabling a teammate's shared extension without reading it", x-why-unsafe: "Its http tool can send your readable data to its author's host", safer_alternative: "Enable only extensions you trust (Q27)", x-provenance-basis: "§1.13 row 'Enable another person's extension only when you trust its author' (Q27)", provenance: {kind: inferred, question_id: Q27}}
- {id: km-bridge-without-account, components: [outbound], pattern: "Turning on the chat-room bridge without its own account", x-why-unsafe: "The room reads as the admin", safer_alternative: "Give the bridge a low-role account", x-provenance-basis: "§1.12 N-BRIDGE", provenance: {kind: documented, source: 'docs/chat-rooms.md "answers as the admin who turned it on"'}}
- {id: km-admin-for-python-tool, components: [assistant], pattern: "Making someone admin so they can write a Python tool", x-why-unsafe: "Admin equals host code execution", safer_alternative: "Use prompt, graph or http tools", x-provenance-basis: "§1.2 roles and trust; §1.13 'Make only fully trusted people admins'", provenance: {kind: maintainer, date: "2026-10"}}
- - {id: km-wide-web-networks, components: [outbound], pattern: "Wide LENS_WEB_NETWORKS", x-why-unsafe: "Editors, and iCal feed redirects, can then reach those networks", safer_alternative: "List single hosts", x-provenance-basis: "§1.6 web_networks row; §1.13 private networks row", provenance: {kind: documented, source: 'docs/configuration.md "loopback and cloud metadata addresses stay out"'}}
+ - {id: km-wide-web-networks, components: [outbound], pattern: "Wide LENS_WEB_NETWORKS", x-why-unsafe: "Editors, and iCal and news feed redirects, can then reach those networks", safer_alternative: "List single hosts", x-provenance-basis: "§1.6 web_networks row; §1.13 private networks row", provenance: {kind: documented, source: 'docs/configuration.md "loopback and cloud metadata addresses stay out"'}}
- {id: km-push-token-in-url, components: [optional], pattern: "Webhook push token in a URL that proxies log", x-why-unsafe: "The token is a credential in the path", safer_alternative: "Use the header form", x-provenance-basis: "§1.13 row 'Keep logs that record request paths private'", provenance: {kind: documented, source: 'docs/sensors.md "Ways in"'}}
- {id: km-hosted-model-sensitive, components: [assistant], pattern: "Pointing the assistant at a hosted model for sensitive archives", x-why-unsafe: "Assets' text leaves the machine", safer_alternative: "Use a local model server", x-provenance-basis: "§1.13 row 'Pick model and speech servers you trust'", provenance: {kind: maintainer, date: "2026-10"}}
From 98afecaa56600197c68f4cfccf4ec227ba4f1ff3 Mon Sep 17 00:00:00 2001
From: Adeel Ahmad
Date: Sat, 17 Oct 2026 05:56:56 +0000
Subject: [PATCH 2/2] docs(requirements): news feeds are built, with the checks
that prove it
Co-Authored-By: Claude Opus 5.5 (1M context)
Claude-Session: https://claude.ai/code/session_01SWH8a26M9wkAPq2eRrfWZK
---
README.md | 1 -
docs/requirements.md | 1 +
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index d47634a36..15df25827 100644
--- a/README.md
+++ b/README.md
@@ -207,7 +207,6 @@ These are open pull requests, not yet part of Lens:
- **The assistant** answering questions about Lens itself, from its docs and this server
([#400](https://github.com/adeelahmad/lense/pull/400)).
- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
-- **Sources:** RSS and Atom news feeds ([#453](https://github.com/adeelahmad/lense/pull/453)).
- **Pictures:** described on this machine with Florence-2 ([#477](https://github.com/adeelahmad/lense/pull/477)), and
low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
diff --git a/docs/requirements.md b/docs/requirements.md
index 41ecf6972..0a6603fd1 100644
--- a/docs/requirements.md
+++ b/docs/requirements.md
@@ -106,6 +106,7 @@ request, to another check that proves the same thing; loosening a requirement or
| A watched folder on each kind of storage connection, run against rclone in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| IMAP mail, Gmail and Google Workspace | [test_mail_calendar_sources.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mail_calendar_sources.py) `test_an_imap_account_as_a_source`, `test_an_imap_message_is_an_email_document_with_its_attachments`; [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_a_workspace_mailbox_through_the_gmail_api`, `test_a_connected_google_account` |
| iCal calendars and Google Calendar | [test_mail_calendar_sources.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mail_calendar_sources.py) `test_a_calendar_feed_as_a_source`; [test_google_calendar_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_google_calendar_source.py) `test_a_shared_calendar_read_as_a_service_account`, `test_a_connected_google_account` |
+| RSS and Atom news feeds: a feed or a site's page as a source, each post imported as text with its link, author and date; private networks only when allowed | [test_news_feed_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_news_feed_source.py) `test_a_news_feed_as_a_source`, `test_feed_addresses`, `test_what_a_feed_holds_stays_data`, `test_a_feed_file_imported_as_text` |
| A built-in MQTT broker, syslog and webhook streams | [test_sensors.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sensors.py) `test_hub_takes_devices_in_and_organises_them`, `test_syslog_senders_become_sensors`, `test_webhooks_and_one_list_of_sensors`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `sensor-webhook`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `sensor-webhook` ≤ 3 clicks |
| Import webhooks for scanners and phone shortcuts | [test_import_hooks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_hooks.py) `test_pushing_files_urls_and_text`, `test_owners_manage_hooks`; [import-hooks.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/import-hooks.test.tsx) |
| Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` |