diff --git a/CHANGELOG.md b/CHANGELOG.md index 74f6ab9a5..059e6d293 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,10 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version each run shows what its actions did. Try one on an activity without keeping or sending anything, or run it now; Settings → Automations switches them all off and sets how far they may go ([Running automations](docs/automations.md#running-automations)). +- **News feeds.** Subscribe to a blog's or a news site's RSS or Atom feed: add it on the Sources page (its address, + or just the site's) and watch it, and each new post comes into a namespace as text, with its link, author and date, + so answers cite it. Feeds are asked for again only when they've changed, and a private feed's address is kept like a + password ([News feeds](docs/processing.md#news-feeds-rss-and-atom)). - **Each answer says what produced it.** The *i* in an answer's header shows the model and the host it ran on, the tools it was offered and used, the scope, its limits and what it used, tokens, cost (an estimate from your prices), how long it took, how it ended and which tool results read like instructions to an AI, with the calls made for it. diff --git a/README.md b/README.md index bd05eb16d..0df6a9128 100644 --- a/README.md +++ b/README.md @@ -97,13 +97,14 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p [adding an MCP server](docs/assistant.md#adding-an-mcp-server), [the MCP runner](docs/assistant.md#the-mcp-runner); [checks](docs/requirements.md#extend-the-assistant)) - **Bring in everything.** Watched folders on S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV or local disks (a - local folder in four clicks); IMAP mail, Gmail and Google Workspace, iCal calendars and Google Calendar; a built-in - MQTT broker, syslog and webhook streams; import webhooks for scanners and phone shortcuts. Google accounts connect - with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google Drive can sign in through - rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs, Office files, photos, JSON - and zip or tar archives, and, with an Apache Tika server (an opt-in extra), documents nothing else here can read; - capture web pages one at a time, from a link list or a bookmarks file, by API or by asking the assistant, from private - networks you allow in Settings too; and import videos and podcasts by their address (off by default). + local folder in four clicks); IMAP mail, Gmail and Google Workspace, iCal calendars, Google Calendar and RSS and Atom + news feeds; a built-in MQTT broker, syslog and webhook streams; import webhooks for scanners and phone shortcuts. + Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google + Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs, + Office files, photos, JSON and zip or tar archives, and, with an Apache Tika server (an opt-in extra), documents + nothing else here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking + the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off + by default). ([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika), [import webhooks](docs/api.md#import-webhooks); [checks](docs/requirements.md#bring-in-everything)) - **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a @@ -207,7 +208,6 @@ These are open pull requests, not yet part of Lens: - **The assistant** answering questions about Lens itself, from its docs and this server ([#400](https://github.com/adeelahmad/lense/pull/400)). - **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)). -- **Sources:** RSS and Atom news feeds ([#453](https://github.com/adeelahmad/lense/pull/453)). - **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)). - **Web clipper:** signing in, or using your API token, not only an import webhook's ([#406](https://github.com/adeelahmad/lense/pull/406)). diff --git a/docs/configuration.md b/docs/configuration.md index d964a5ddb..ebe8c80fb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -292,7 +292,7 @@ text (Word, text, Markdown) are imported as text. Settings → Documents: | `documents.convert_seconds` | 300 | how long making one PDF may take before its job fails, 10–3600 seconds | | `documents.attachment_resources` | true | whether an email's attachments that Lens can read (documents, images, audio, video, emails) also become resources of their own; they're kept as its files either way | | `documents.video_urls` | false | whether videos and podcasts can be imported by their address ([API](api.md#videos-and-podcasts)): their media is downloaded with yt-dlp, which Lens fetches on first use | -| `documents.web_networks` | `[]` | private networks (CIDR, such as `192.168.1.0/24`) that web pages, calendar feeds and videos may be fetched from besides the public internet: for an intranet, or a calendar server at home. The web app can't list this machine (loopback) or link-local addresses, where clouds answer with the server's own credentials; archive.yaml or `LENS_WEB_NETWORKS` in `.env` (comma-separated, added to archive.yaml's) can. Those stay whatever Settings saves: the networks listed in Settings are added to them | +| `documents.web_networks` | `[]` | private networks (CIDR, such as `192.168.1.0/24`) that web pages, calendar and news feeds and videos may be fetched from besides the public internet: for an intranet, or a calendar server at home. The web app can't list this machine (loopback) or link-local addresses, where clouds answer with the server's own credentials; archive.yaml or `LENS_WEB_NETWORKS` in `.env` (comma-separated, added to archive.yaml's) can. Those stay whatever Settings saves: the networks listed in Settings are added to them | | `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it | | `documents.anytopdf_url` | none | a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work`, such as `https://convert.home:8640`. Set, it converts instead of the program here | | `documents.anytopdf_token` | none | the node's bearer token (its `ANYTOPDF_QUEUE_TOKEN`), kept encrypted | diff --git a/docs/processing.md b/docs/processing.md index 7e37803b1..421c63fb7 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -427,7 +427,8 @@ A folder scan and a routine run are each claimed first, so several processes doi ## Storage sources Admins add sources in the app (`/api/v1/sources`): S3 or S3-compatible, Dropbox, Google Drive, OneDrive, SFTP, SMB, -WebDAV, or a folder on this machine; or an email account (IMAP, or Gmail and Google Workspace), a calendar feed (iCal) or Google Calendar, below. A watched folder maps a path on a source to a namespace, with include/exclude +WebDAV, or a folder on this machine; or an email account (IMAP, or Gmail and Google Workspace), a calendar feed (iCal), +Google Calendar or a [news feed](#news-feeds-rss-and-atom) (RSS or Atom), below. A watched folder maps a path on a source to a namespace, with include/exclude patterns, audio and/or transcripts, a polling interval, how long a file must be unchanged before it is picked up, and whether files already there are imported (backfill). New audio is queued for the full pipeline; new transcripts are imported and analysed. Audio stays where it is: it is copied to a cache for processing and streamed from the source for @@ -479,8 +480,8 @@ files, then Import (`POST /api/v1/import/source`). The listing marks files that ### Email (IMAP), calendar feeds (iCal) and Google Calendar -Three kinds of source aren't storage and don't use rclone; their messages and events are shown as files, so browsing, -importing chosen ones and watching work as above. +These kinds of source aren't storage and don't use rclone; their messages and events are shown as files, so browsing, +importing chosen ones and watching work as above. [News feeds](#news-feeds-rss-and-atom) work the same way. - **Email (IMAP)**: host, port, security (SSL/TLS, STARTTLS or none), user and password (an app password where the provider has them). Mailboxes are the folders, and each message is a file `/-.eml`, @@ -548,6 +549,38 @@ importing chosen ones and watching work as above. a Google calendar's secret address in iCal format (Settings › Integrate calendar) works as a calendar feed. - `.eml` and `.ics` files in any source, and uploaded through Import, can be read as text this way too. +### News feeds (RSS and Atom) + +A news feed (type `rss`) is a source too: Settings › Sources › Add › News feed, then watch it. Watching a feed +subscribes to it: each new post becomes a text resource of the namespace, titled and dated by the post, so answers cite +it and its link. + +- **Address**: the feed's own (`https://`, or `feed://`), or a page of the site: when the page names its feed + (``, as blogs and news sites do), the first feed it names is read. + A user and password if the feed asks for one. The address is kept encrypted like a password and never shown again, + since a private feed's address often carries its key. It is fetched as calendar feeds are: public addresses only + (and the networks in `documents.web_networks`), on ports 80 and 443, and the password is never sent to another + server, including one that a page names as its feed's. +- **Posts**: the feed is one folder of posts, each a file `.atom` named by its date and title. RSS (0.9x, 1.0, + 2.0) and Atom (0.3, 1.0) are read by [feedparser](https://github.com/kurtmckee/feedparser) (BSD), which drops + scripts, styles and event handlers from the posts' HTML and makes their links absolute. Whatever the feed, each post + is kept as an Atom entry document of its own (RFC 4287): its title, the post's page, enclosures (a podcast's audio), + when it was published and updated, its authors, categories, summary and content, and the feed's title and site + (never its address). It comes in as text: the title, its link, who wrote it, when, the feed, its categories and + enclosures, then its text. Its content type is Feed post, so a namespace can give feed posts a pipeline of their own + (Settings › Content types). +- **New posts only, unless you backfill**: as for any watch, a new watch on a feed takes the posts published after it + starts; turn on backfill to bring in those the feed lists already. The newest 500 posts of a feed are listed. +- **Light on the network**: a feed read is reused for a minute, then asked for again with its `ETag` and + `Last-Modified`, so a feed that hasn't changed costs one small request. A post is known by its id (an RSS item's + `guid`), so a feed that reorders or retitles its posts doesn't bring them in twice; a post whose text changes is read + again into the resource it already is. A post the feed drops later keeps its resource and its file. +- `.atom` files uploaded through Import (a single post, or a whole Atom feed) are read as text the same way. + +Refine later: JSON Feed (feedparser reads RSS and Atom only); an option to capture each post's whole page, for feeds +that carry only a summary; podcast feeds bringing in each episode's audio to transcribe; OPML import of a feed list; +subscribing in one step (address and namespace) from Settings. + ### Gmail and Google Workspace A Gmail or Google Workspace mailbox (type `gmail`) is read through Google's Gmail API, read only (`gmail.readonly`), diff --git a/docs/requirements.md b/docs/requirements.md index 37741ba08..32ba79bfa 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -106,6 +106,7 @@ request, to another check that proves the same thing; loosening a requirement or | A watched folder on each kind of storage connection, run against rclone in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) | | IMAP mail, Gmail and Google Workspace | [test_mail_calendar_sources.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mail_calendar_sources.py) `test_an_imap_account_as_a_source`, `test_an_imap_message_is_an_email_document_with_its_attachments`; [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_a_workspace_mailbox_through_the_gmail_api`, `test_a_connected_google_account` | | iCal calendars and Google Calendar | [test_mail_calendar_sources.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mail_calendar_sources.py) `test_a_calendar_feed_as_a_source`; [test_google_calendar_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_google_calendar_source.py) `test_a_shared_calendar_read_as_a_service_account`, `test_a_connected_google_account` | +| RSS and Atom news feeds: a feed or a site's page as a source, each post imported as text with its link, author and date; private networks only when allowed | [test_news_feed_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_news_feed_source.py) `test_a_news_feed_as_a_source`, `test_feed_addresses`, `test_what_a_feed_holds_stays_data`, `test_a_feed_file_imported_as_text` | | A built-in MQTT broker, syslog and webhook streams | [test_sensors.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sensors.py) `test_hub_takes_devices_in_and_organises_them`, `test_syslog_senders_become_sensors`, `test_webhooks_and_one_list_of_sensors`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `sensor-webhook`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `sensor-webhook` ≤ 3 clicks | | Import webhooks for scanners and phone shortcuts | [test_import_hooks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_hooks.py) `test_pushing_files_urls_and_text`, `test_owners_manage_hooks`; [import-hooks.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/import-hooks.test.tsx) | | Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` | diff --git a/docs/sensors.md b/docs/sensors.md index 809c4a41c..7d9eabcca 100644 --- a/docs/sensors.md +++ b/docs/sensors.md @@ -4,7 +4,7 @@ A **sensor** is anything that feeds Lens. There are two families, in one list (` | Family | Kinds | Channels | What it reports becomes | | --- | --- | --- | --- | -| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP, Gmail and Google Workspace); calendar feeds (iCal); Google Calendar | the folders it watches | resources, run through their pipelines (as sources always have) | +| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP, Gmail and Google Workspace); calendar feeds (iCal); Google Calendar; news feeds (RSS and Atom) | the folders it watches | resources, run through their pipelines (as sources always have) | | Streams | MQTT devices, syslog senders, webhooks | streams: an MQTT topic, a syslog program, a webhook's stream name | readings, kept with retention, and hourly summaries of anything numeric | File sensors are the storage sources: they are added and changed as sources (`/api/v1/sources`), and nothing about diff --git a/fastapi_backend/app/api/v1/routes/imports.py b/fastapi_backend/app/api/v1/routes/imports.py index 494e5525b..691ed9f94 100644 --- a/fastapi_backend/app/api/v1/routes/imports.py +++ b/fastapi_backend/app/api/v1/routes/imports.py @@ -34,7 +34,7 @@ router = APIRouter(prefix="/import", tags=["imports"]) -IMPORT_EXT = {".txt", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".json", ".jsonl", ".srt", ".vtt", ".eml", ".ics"} +IMPORT_EXT = {".txt", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".json", ".jsonl", ".srt", ".vtt", ".eml", ".ics", ".atom"} def _upload_name(filename: str | None) -> str: diff --git a/fastapi_backend/app/domain/content_types.py b/fastapi_backend/app/domain/content_types.py index a9ad637fd..cc08a2daf 100644 --- a/fastapi_backend/app/domain/content_types.py +++ b/fastapi_backend/app/domain/content_types.py @@ -55,6 +55,7 @@ ("archive", "text", "Archive", "A zip or tar file, and the files in it", {"extensions": [".tar", ".tgz", ".zip"]}), ("email", "text", "Email", "A message, with its attachments", {"extensions": [".eml", ".msg"]}), ("calendar_event", "text", "Calendar event", "An invitation or a calendar entry", {"extensions": [".ics"]}), + ("feed_post", "text", "Feed post", "A post from a news feed (RSS or Atom)", {"extensions": [".atom"]}), ] # fmt: skip diff --git a/fastapi_backend/app/domain/feeds.py b/fastapi_backend/app/domain/feeds.py index bcbef94f9..1a2aff36a 100644 --- a/fastapi_backend/app/domain/feeds.py +++ b/fastapi_backend/app/domain/feeds.py @@ -1,6 +1,6 @@ -"""Sources that aren't file storage: IMAP mailboxes, Gmail and Google Workspace mailboxes, iCalendar feeds and Google -Calendar (gcal.py), shown as folders of files so browsing, choosing files, watching and importing work for them as they -do for rclone's storage (sources.py). +"""Sources that aren't file storage: IMAP mailboxes, Gmail and Google Workspace mailboxes, iCalendar feeds, Google +Calendar (gcal.py) and news feeds (RSS and Atom: rss.py), shown as folders of files so browsing, choosing files, +watching and importing work for them as they do for rclone's storage (sources.py). - An IMAP account's folders are its mailboxes, and each message is a file, `/-.eml`: a UID stands for one message only while the mailbox's UIDVALIDITY stays the same, so a mailbox the server rebuilt @@ -16,7 +16,7 @@ read as text: its title, when and where, who, and its description. The address is kept sealed, like a password (a private calendar's address is all it takes to read it), and fetched as web pages are captured (netguard.py): public addresses only, and the networks in documents.web_networks, on ports 80 and 443. Its password is never sent - on to another server it redirects to. + on to another server it redirects to (`get`, which news feeds are fetched with too). Both keep the names of their files (`name`), a `title` and `when` for the resource made from one, beside what rclone's listings have (path, rel, size, modified). @@ -38,10 +38,10 @@ import urllib.request from email import policy -from . import calendars, gcal, netguard +from . import calendars, gcal, netguard, rss # sources a module of their own handles: browse, list_files, fetch, immutable and test, as below -OWN = {"gcal": gcal} +OWN = {"gcal": gcal, "rss": rss} TYPES = { "imap": { "label": "Email (IMAP)", @@ -455,38 +455,50 @@ def _opener(guard): return o -def _download(cfg, src): +def get(cfg, src, url, what, accept, limit, agent, headers=None, auth=True): + """GET a source's address (a calendar's, a news feed's) as web pages are captured: through netguard, to public + addresses and the networks in documents.web_networks only, on ports 80 and 443. The source's user and password go + with it (unless `auth` is False), never on to another server it redirects to; `headers` too (a conditional GET's + If-None-Match, say). Returns (status, the response's headers, its bytes, the address it ended at): a 304 answers + no bytes. RuntimeError, in words naming `what` ("calendar"), for what went wrong.""" from . import webcapture # the networks an admin allowed besides public ones; it imports what imports this - url = _url(cfg, src) - hit = _FEEDS.get(src["id"]) - if hit and hit[0] == url and time.monotonic() - hit[1] < FEED_SECONDS: - return hit[2] try: webcapture.check_url(cfg, url) except ValueError as e: if "public address" in str(e): host = urllib.parse.urlsplit(url).hostname raise RuntimeError( - f"that calendar can't be fetched: {host} is on a private network. An admin can allow that network in " + f"that {what} can't be fetched: {host} is on a private network. An admin can allow that network in " "Settings → Documents (Private networks Lens may fetch from), e.g. 192.168.1.0/24" ) from None - raise RuntimeError(f"that calendar can't be fetched: {e}") from None - req = urllib.request.Request(url, headers={"User-Agent": "Lens calendar source", "Accept": "text/calendar, */*;q=0.5"}) - user, pw = _params(src).get("user", ""), _secret(cfg, src, "pass") + raise RuntimeError(f"that {what} can't be fetched: {e}") from None + req = urllib.request.Request(url, headers={"User-Agent": agent, "Accept": accept, **(headers or {})}) + user, pw = (_params(src).get("user", ""), _secret(cfg, src, "pass")) if auth else ("", "") if user or pw: req.add_header("Authorization", "Basic " + base64.b64encode(f"{user}:{pw}".encode()).decode("ascii")) - with netguard.Guard(forward=True, networks=webcapture.networks(cfg), max_bytes=MAX_CALENDAR + 65536) as guard: + with netguard.Guard(forward=True, networks=webcapture.networks(cfg), max_bytes=limit + 65536) as guard: try: with _opener(guard).open(req, timeout=60) as r: - raw = r.read(MAX_CALENDAR + 1) + status, got, raw, final = r.status, r.headers, r.read(limit + 1), r.geturl() except urllib.error.HTTPError as e: - raise RuntimeError(f"the calendar's server answered {e.code} {e.reason}") from None + if e.code == 304: + return 304, e.headers, b"", url + raise RuntimeError(f"the {what}'s server answered {e.code} {e.reason}") from None except (urllib.error.URLError, OSError) as e: refused = f"; refused: {guard.refused[0]}" if guard.refused else "" - raise RuntimeError(f"couldn't fetch the calendar: {getattr(e, 'reason', e)}{refused}") from None - if len(raw) > MAX_CALENDAR: - raise RuntimeError("the calendar is larger than 50 MB") + raise RuntimeError(f"couldn't fetch the {what}: {getattr(e, 'reason', e)}{refused}") from None + if len(raw) > limit: + raise RuntimeError(f"the {what} is larger than {limit // 2**20} MB") + return status, got, raw, final + + +def _download(cfg, src): + url = _url(cfg, src) + hit = _FEEDS.get(src["id"]) + if hit and hit[0] == url and time.monotonic() - hit[1] < FEED_SECONDS: + return hit[2] + _, _, raw, _ = get(cfg, src, url, "calendar", "text/calendar, */*;q=0.5", MAX_CALENDAR, "Lens calendar source") text = raw.decode("utf-8", "replace") if "BEGIN:VCALENDAR" not in text[:4096].upper(): raise RuntimeError("that address doesn't give a calendar (no BEGIN:VCALENDAR)") diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index 7a333e6bd..2eb1bf7c5 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -859,8 +859,16 @@ def read_calendar(path): return calendars.text(pathlib.Path(path).read_text(encoding="utf-8-sig", errors="replace")) +def read_feed(path): + """A news feed's post (.atom, as a feed source keeps each), or a whole feed, as Markdown, a section per post (rss.py).""" + from . import rss + + return rss.text(pathlib.Path(path).read_bytes()) + + DOC_READERS = {".docx": read_docx, ".doc": read_doc, ".pdf": read_pdf} -MARKDOWN_READERS = {".eml": read_email, ".ics": read_calendar} # their subject or event title is the title +# their subject, event title or post title is the title +MARKDOWN_READERS = {".eml": read_email, ".ics": read_calendar, ".atom": read_feed} FORMATS = ("auto", "text", "markdown", "mdx", "json", "jsonl", "srt", "vtt", "chat") @@ -1037,7 +1045,7 @@ def _store_import(db, cfg, ns, t, title, fp, src, st, audio, speaker_names, engi def import_transcript(db, cfg, ns, tpath, audio=None, title=None, speaker_names=None, fmt="auto", log=print, collection=None): - """A transcript file (txt, md, mdx, docx, doc, pdf, json, jsonl, srt, vtt, eml, ics), optionally with its audio, into a + """A transcript file (txt, md, mdx, docx, doc, pdf, json, jsonl, srt, vtt, eml, ics, atom), optionally with its audio, into a collection of the namespace (default: its default collection).""" src = pathlib.Path(audio or tpath) t = read_transcript(tpath, fmt) diff --git a/fastapi_backend/app/domain/rss.py b/fastapi_backend/app/domain/rss.py new file mode 100644 index 000000000..a8c0ded65 --- /dev/null +++ b/fastapi_backend/app/domain/rss.py @@ -0,0 +1,397 @@ +"""News feeds (RSS and Atom) as a source, a file sensor as feeds.py's calendar feeds are: a feed is one folder of +posts, each a file `.atom` of its own, so browsing, choosing posts, watching and importing work as they do for any +source. Watching a feed subscribes to it: each new post becomes a text resource of the namespace, titled and dated by +the post, with its address, author and feed written in, so answers cite it. + +The address can be the feed's own or a page of the site that names its feed (``, as blogs and news sites do): the first feed the page names is read. feedparser (BSD) +reads RSS 0.9x, 1.0 and 2.0 and Atom 0.3 and 1.0, drops scripts, styles and event handlers from the posts' HTML and +makes their links absolute. Whatever the feed was, each post is written as an Atom entry document (RFC 4287 +§4.1.2): its id, title, the post's page and its enclosures (a podcast's audio), when it was published and updated, its +authors, categories, summary and content, and the feed it came from (its title and site, never its address). `text` +reads one as Markdown, as ingest.py reads emails and calendar events. + +The address is kept sealed, like a password (a private feed's address often carries its key), and fetched as calendar +feeds are (feeds.get): public addresses only, and the networks in documents.web_networks, on ports 80 and 443; the +password, for a feed that asks for one, is never sent to another server, a feed that a page names elsewhere included. +A feed read is reused for FEED_SECONDS, then asked for again with its ETag and Last-Modified, so one that hasn't +changed costs a small request and no parsing. Its posts stay in memory for KEEP_SECONDS (a scan reads the new ones +right after listing them), its listing for as long as Lens runs: a post wanted later is read again, so many feeds +don't fill a small machine's memory. + +A post's file is named by its id (an RSS item's guid), so it stays the same file however the feed reorders or retitles +it. Its time is when it was published; a post whose text changes (its size) is read again into the resource it +already is. +""" + +from __future__ import annotations + +import calendar +import datetime as dt +import hashlib +import io +import re +import time +import urllib.parse +import xml.etree.ElementTree as ET +from html.parser import HTMLParser + +TYPE = { + "label": "News feed (RSS or Atom)", + "fields": {"user": ""}, + "secrets": ["url", "pass"], # the address is kept like a password: a private feed's carries its key +} +MAX_FEED = 20 * 2**20 +MAX_POSTS = 500 # a feed's first posts (its newest, as feeds list them) that are listed +FEED_SECONDS = 60 # a feed read for a listing is reused this long (a scan then reads each of its posts) +KEEP_SECONDS = 300 # how long a feed's posts stay in memory after it was read (its listing stays) +AGENT = "Lens feed reader" +ACCEPT = "application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.9, */*;q=0.5" +FEED_TYPES = ("application/rss+xml", "application/atom+xml", "application/rdf+xml") +ATOM = "http://www.w3.org/2005/Atom" +EPOCH = "1970-01-01T00:00:00+00:00" # for posts that don't say when: their size tells +_XML_BAD = re.compile("[\x00-\x08\x0b\x0c\x0e-\x1f￾￿]") +# source id -> the feed as last read: {url, found, at, read_at, etag, last_modified, title, site, entries, bodies} +_FEEDS: dict = {} + + +def _plain(v): + """Text on one line, without control characters.""" + return " ".join(re.sub(r"[\x00-\x1f\x7f]", " ", str(v or "")).split()) + + +def _href(v): + v = _plain(v) + return v if urllib.parse.urlsplit(v).scheme in ("http", "https") else None + + +def _xml(v): + return _XML_BAD.sub("", str(v)) + + +def _html_text(markup): + from . import mailbox # its reader of HTML mail bodies, a line per block + + return mailbox.html_text(markup) + + +def _url(cfg, src): + from . import feeds # it imports this + + url = feeds._secret(cfg, src, "url").strip() + url = re.sub(r"^feed:(?=https?://)", "", url, flags=re.I) # feed:https://…, as some sites link their feeds + url = re.sub(r"^feed://", "https://", url, flags=re.I) + if not re.match(r"^https?://[^\s/]+\S*$", url, re.I): + raise RuntimeError("the feed's address starts with https:// (or feed://)") + return url + + +def _server(url): + u = urllib.parse.urlsplit(url) + return u.hostname, u.port or (443 if u.scheme == "https" else 80) + + +# ---------- reading a feed ---------- +class _Alternates(HTMLParser): + """The feeds a web page names: its (or Atom's).""" + + def __init__(self): + super().__init__(convert_charrefs=True) + self.found = [] + + def handle_starttag(self, tag, attrs): + if tag != "link": + return + a = {k.lower(): (v or "").strip() for k, v in attrs} + kind = a.get("type", "").lower().split(";")[0].strip() + if "alternate" in a.get("rel", "").lower().split() and kind in FEED_TYPES and a.get("href"): + self.found.append(a["href"]) + + +def _named_feed(raw, base): + """The address of the first feed a page names, or None.""" + p = _Alternates() + p.feed(raw[: 2 * 2**20].decode("utf-8", "replace")) + p.close() + for href in p.found: + url = urllib.parse.urljoin(base, href) + if urllib.parse.urlsplit(url).scheme in ("http", "https"): + return url + return None + + +def _page(ctype, raw): + """Whether what came back is a web page rather than a feed (some servers call their feeds text/html).""" + head = raw[:1024].lstrip(b"\xef\xbb\xbf \t\r\n").lower() + if head.startswith((b" KEEP_SECONDS: + h["bodies"] = None + hit = _FEEDS.get(src["id"]) + if hit and hit["url"] != url: + hit = None + whole = hit is not None and (hit["bodies"] is not None or not bodies) # what's kept is all that's wanted + if whole and now - hit["at"] < FEED_SECONDS: + return hit + + def get(address, headers=None): + # the source's password goes only to the server its address is on + auth = _server(address) == _server(url) + return feeds.get(cfg, src, address, "feed", ACCEPT, MAX_FEED, AGENT, headers, auth=auth) + + target = hit["found"] if hit else url + cond = {k: v for k, v in (("If-None-Match", hit["etag"]), ("If-Modified-Since", hit["last_modified"])) if v} if whole else {} + status, headers, raw, final = get(target, cond) + if status == 304 and whole: + hit["at"] = now + return hit + if _page(headers.get("Content-Type"), raw): + named = _named_feed(raw, final) if target == url else None + if not named: + raise RuntimeError( + "that address is a web page that names no news feed: give the feed's own address (it often ends in /feed, /rss or .xml)" + ) + target = named + status, headers, raw, final = get(target) + if _page(headers.get("Content-Type"), raw): + raise RuntimeError("the feed that page names is a web page too, not a news feed") + feed = parse(raw, final, headers.get("Content-Type")) + out = { + "url": url, + "found": target, + "at": now, + "read_at": now, + "etag": headers.get("ETag"), + "last_modified": headers.get("Last-Modified"), + "title": feed["title"], + "site": feed["site"], + "entries": {k: e for k, (e, _) in feed["posts"].items()}, + "bodies": {k: b for k, (_, b) in feed["posts"].items()}, + } + _FEEDS[src["id"]] = out + return out + + +def _feedparser(raw, headers=None): + import feedparser # loaded when a feed is read: a server without feeds never needs it + + # from a stream: given a string, feedparser would fetch an address, or open a file of that name + return feedparser.parse(io.BytesIO(raw), response_headers=headers or {}) + + +def parse(raw, base, ctype=None): + """A feed's title, site and posts, {title, site, posts: {file: (entry, Atom entry document)}}, from its bytes; + `base` is the address it came from, which relative links are read against (without its query, so a private feed's + key stays out of what is kept).""" + u = urllib.parse.urlsplit(base) + base = urllib.parse.urlunsplit((u.scheme, u.netloc, u.path, "", "")) + d = _feedparser(raw, {"content-location": base, **({"content-type": ctype} if ctype else {})}) + if not d.get("version") and not d.get("entries"): + raise RuntimeError("that address doesn't give a news feed (RSS or Atom)") + feed = d.get("feed") or {} + title = _plain(feed.get("title")) or u.hostname or "News feed" + site = _href(feed.get("link")) + feed_id = _plain(feed.get("id")) or site + posts = {} + for e in d.get("entries") or []: + if len(posts) >= MAX_POSTS: + break + p = _post(e) + if p is None: + continue + rel = f"{hashlib.sha1(p['id'].encode()).hexdigest()[:20]}.atom" + if rel in posts: # the same post twice: the first (newest) one + continue + body = _atom(p, title, site, feed_id) + day = (p["published"] or p["updated"] or "")[:10] + name = re.sub(r"[\x00-\x1f\x7f/\\]", "", f"{day} {p['title']}".strip())[:150].strip() + entry = { + "path": rel, + "rel": rel, + "name": f"{name or 'Post'}.atom", + "dir": False, + "size": len(body), + "modified": p["published"] or p["updated"] or EPOCH, + "title": p["title"] or None, + "when": p["published"] or p["updated"], + "settled": True, # a post is whole once its feed lists it: no waiting for it to settle + } + posts[rel] = (entry, body) + return {"title": title, "site": site, "posts": posts} + + +def _iso(t): + try: + return dt.datetime.fromtimestamp(calendar.timegm(t), dt.UTC).isoformat(timespec="seconds") if t else None + except (OverflowError, ValueError, TypeError): + return None + + +def _post(e): + """What a feed's entry says of a post, tidied: None when it has nothing to tell it by.""" + links = [l for l in e.get("links") or [] if isinstance(l, dict)] + page = _href(e.get("link")) or next((_href(l.get("href")) for l in links if l.get("rel") == "alternate" and _href(l.get("href"))), None) + enclosures = [ + {"href": _href(l.get("href")), "type": _plain(l.get("type"))[:100], "length": _plain(l.get("length"))[:20]} + for l in links + if l.get("rel") == "enclosure" and _href(l.get("href")) + ][:20] + title = e.get("title") or "" + if "html" in str((e.get("title_detail") or {}).get("type") or ""): + title = _html_text(title) + title = _plain(title)[:300] + content = next((c for c in e.get("content") or [] if isinstance(c, dict) and c.get("value")), {}) + summary = e.get("summary") or "" + summary_type = str((e.get("summary_detail") or {}).get("type") or "text/html") + if summary == content.get("value"): # feedparser repeats an Atom entry's content as its summary + summary = "" + authors = [] + for a in e.get("authors") or ([{"name": e.get("author")}] if e.get("author") else []): + if isinstance(a, dict) and (a.get("name") or a.get("email")): + authors.append({"name": _plain(a.get("name"))[:200], "email": _plain(a.get("email"))[:200], "uri": _href(a.get("href"))}) + published, updated = _iso(e.get("published_parsed")), _iso(e.get("updated_parsed")) + text_of = content.get("value") or summary + pid = _plain(e.get("id")) or page + if not pid and (title or text_of): + pid = "sha1:" + hashlib.sha1(f"{title}\n{published}\n{text_of}".encode()).hexdigest() + if not pid: + return None + return { + "id": pid[:2000], + "title": title, + "page": page, + "enclosures": enclosures, + "published": published, + "updated": updated, + "authors": authors[:20], + "categories": [ + c for c in dict.fromkeys(_plain(t.get("label") or t.get("term"))[:100] for t in e.get("tags") or [] if isinstance(t, dict)) if c + ][:30], + "summary": summary, + "summary_type": "html" if "html" in summary_type else "text", + "content": content.get("value") or "", + "content_type": "html" if "html" in str(content.get("type") or "text/html") else "text", + } + + +def _atom(p, feed_title, site, feed_id): + """A post as an Atom entry document of its own (RFC 4287 §4.1.2), the feed it came from as its .""" + + def add(parent, tag, text=None, **attrs): + el = ET.SubElement(parent, tag, {k: _xml(v) for k, v in attrs.items() if v}) + if text: + el.text = _xml(text) + return el + + root = ET.Element("entry", {"xmlns": ATOM}) + add(root, "id", p["id"]) + add(root, "title", p["title"] or "(no title)", type="text") + if p["page"]: + add(root, "link", rel="alternate", href=p["page"]) + for x in p["enclosures"]: + add(root, "link", rel="enclosure", href=x["href"], type=x["type"], length=x["length"]) + if p["published"]: + add(root, "published", p["published"]) + add(root, "updated", p["updated"] or p["published"] or EPOCH) + for a in p["authors"]: + el = add(root, "author") + add(el, "name", a["name"] or a["email"]) + if a["email"]: + add(el, "email", a["email"]) + if a["uri"]: + add(el, "uri", a["uri"]) + for c in p["categories"]: + add(root, "category", term=c) + if p["summary"]: + add(root, "summary", p["summary"], type=p["summary_type"]) + if p["content"]: + add(root, "content", p["content"], type=p["content_type"]) + source = add(root, "source") + if feed_id: + add(source, "id", feed_id) + add(source, "title", feed_title, type="text") + if site: + add(source, "link", rel="alternate", href=site) + ET.indent(root) + return b'\n' + ET.tostring(root, encoding="utf-8") + b"\n" + + +# ---------- a post as text ---------- +def _who(a): + return f"{a['name']} ({a['email']})" if a["name"] and a["email"] and a["name"] != a["email"] else a["name"] or a["email"] + + +def text(raw): + """A post's file, or any RSS or Atom feed (one section per post), as Markdown: the post's title as its heading, + 'Label — value' lines (its address, who wrote it, when, the feed, its categories and enclosures), then its text.""" + d = _feedparser(raw) + feed = d.get("feed") or {} + out = [] + for e in d.get("entries") or []: + p = _post(e) + if p is None: + continue + source = e.get("source") or {} + name, site = _plain(source.get("title") or feed.get("title")), _href(source.get("link") or feed.get("link")) + rows = [ + ("Link", p["page"]), + ("By", ", ".join(_who(a) for a in p["authors"])), + ("Published", p["published"]), + ("Updated", p["updated"] if p["updated"] not in (p["published"], EPOCH) else None), + ("Feed", f"{name} ({site})" if name and site else name or site), + ("Categories", ", ".join(p["categories"])), + ("Enclosures", ", ".join(x["href"] + (f" ({x['type']})" if x["type"] else "") for x in p["enclosures"])), + ] + body = p["content"] or p["summary"] + kind = p["content_type"] if p["content"] else p["summary_type"] + body = _html_text(body) if kind == "html" else body.strip() + head = [f"{k} — {v}" for k, v in rows if v] + out.append("\n".join([f"# {p['title'] or '(no title)'}", "", *head, "", body]).strip()) + if not out: + raise SystemExit("no posts found: that isn't a news feed or a post of one (RSS or Atom)") + return "\n\n".join(out) + "\n" + + +# ---------- what feeds.py calls ---------- +def browse(cfg, src, path): + if path: + raise ValueError("a news feed has no folders") + return sorted(read(cfg, src)["entries"].values(), key=lambda e: e["when"] or "", reverse=True) + + +def list_files(cfg, src, path, cursor=None): + return browse(cfg, src, path) + + +def fetch(cfg, src, path): + body = read(cfg, src, bodies=True)["bodies"].get(path) + if body is None: + raise FileNotFoundError("that post isn't in the feed any more") + return body + + +def immutable(src): + return False # a post can be edited + + +def forget(src): + """Drop what was read of this source's feed, so the next call fetches it whole.""" + _FEEDS.pop(src["id"], None) + + +def test(cfg, src): + forget(src) + read(cfg, src) diff --git a/fastapi_backend/app/domain/sources.py b/fastapi_backend/app/domain/sources.py index 1b433fd15..8060f96d5 100644 --- a/fastapi_backend/app/domain/sources.py +++ b/fastapi_backend/app/domain/sources.py @@ -2,8 +2,8 @@ A source is one connection: S3 or S3-compatible, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, or a folder on this machine (only inside sources.local_roots); or, without rclone, an email account (IMAP, or Gmail and Google -Workspace), a calendar feed (iCal) or Google Calendar, whose messages and events are shown as files (feeds.py). A watch -maps a folder on a source to a namespace: new audio, video, documents (PDFs) and images become resources queued for +Workspace), a calendar feed (iCal), Google Calendar or a news feed (RSS or Atom), whose messages, events and posts are +shown as files (feeds.py). A watch maps a folder on a source to a namespace: new audio, video, documents (PDFs) and images become resources queued for the whole pipeline, their files staying on the source; new transcripts are imported and queued for analysis. A watch on an email account can carry routing rules that send some messages to other namespaces, or skip them (mail_routes.py). Credentials are stored encrypted and handed to rclone in a private temporary config file per call; OAuth tokens rclone refreshes are saved. @@ -55,7 +55,8 @@ OBSCURED = {"pass"} # rclone wants these obscured in its config file NOT_RCLONE = {"account"} # parameters Lens keeps for itself, not rclone options SKIPPED = "not audio, video, a document, an image or text" -TRANSCRIPT_EXT = {".txt", ".text", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".srt", ".vtt", ".json", ".jsonl", ".eml", ".ics"} +TRANSCRIPT_EXT = {".txt", ".text", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".srt", ".vtt", ".json", ".jsonl"} +TRANSCRIPT_EXT |= {".eml", ".ics", ".atom"} # emails, calendar events and news feeds' posts, read as text # what a watched folder picks up (its `kinds`): the first three are from before documents, and read PDFs as transcripts TAKES = { "audio": {"audio"}, @@ -316,7 +317,12 @@ def cached_copy(db, cfg, sid, path, space=None): tmp = dest.with_name(dest.name + ".part") try: if feeds.handles(src): - tmp.write_bytes(feeds.fetch(cfg, src, p)) + try: + tmp.write_bytes(feeds.fetch(cfg, src, p)) + except FileNotFoundError: + if dest.exists(): # an event or post gone from its feed since: the copy fetched last + return dest + raise else: run(db, cfg, src, lambda n: ["copyto", f"{n}:{p}", str(tmp)], timeout=6 * 3600) if encrypt: diff --git a/fastapi_backend/app/schemas/sources.py b/fastapi_backend/app/schemas/sources.py index 5e2aefa56..82b98d5c1 100644 --- a/fastapi_backend/app/schemas/sources.py +++ b/fastapi_backend/app/schemas/sources.py @@ -1,4 +1,4 @@ -"""Storage sources (where files live, through rclone; email accounts, Gmail and calendar feeds) and watched folders (admins).""" +"""Storage sources (where files live, through rclone; email accounts, Gmail, calendar feeds and news feeds) and watched folders (admins).""" from __future__ import annotations @@ -8,7 +8,7 @@ from app.schemas.common import Created, Ok, RequestModel, ResponseModel -SourceType = Literal["s3", "dropbox", "drive", "onedrive", "sftp", "smb", "webdav", "local", "imap", "gmail", "ical", "gcal"] +SourceType = Literal["s3", "dropbox", "drive", "onedrive", "sftp", "smb", "webdav", "local", "imap", "gmail", "ical", "gcal", "rss"] WatchKinds = Literal["audio", "transcripts", "both", "documents", "all"] KINDS_HELP = ( "what to pick up: audio (and video), transcripts, both of those (PDFs read as text), documents (PDFs and " diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml index 11f3ae338..a4119db34 100644 --- a/fastapi_backend/pyproject.toml +++ b/fastapi_backend/pyproject.toml @@ -25,6 +25,7 @@ dependencies = [ "opentelemetry-sdk>=1.27,<2", "opentelemetry-exporter-otlp-proto-http>=1.45.1,<2", "webauthn>=3.0.1,<4", + "feedparser>=6.0.11,<7", ] [project.optional-dependencies] diff --git a/fastapi_backend/tests/api/test_content_types.py b/fastapi_backend/tests/api/test_content_types.py index 6588358af..9cf854e02 100644 --- a/fastapi_backend/tests/api/test_content_types.py +++ b/fastapi_backend/tests/api/test_content_types.py @@ -105,6 +105,7 @@ def sub(path, media=None, source="audio"): assert sub("/in/meetup 2024.mp4")[1] == "video" assert sub("/in/My Podcast Ep 12.mp3")[1] == "podcast" assert sub("/in/invite.ics", source="document")[1] == "calendar_event" + assert sub("harbour news:9f2c.atom", source="transcript")[1] == "feed_post" assert sub("/in/re: contract.eml", source="document")[1] == "email" diff --git a/fastapi_backend/tests/api/test_news_feed_source.py b/fastapi_backend/tests/api/test_news_feed_source.py new file mode 100644 index 000000000..ead2c63f0 --- /dev/null +++ b/fastapi_backend/tests/api/test_news_feed_source.py @@ -0,0 +1,283 @@ +"""News feeds (RSS and Atom) as a source: a feed's posts shown as files, browsed, watched and imported like any +source's. Everything runs against a feed served over HTTP from this process.""" + +from __future__ import annotations + +import base64 +import http.server +import socket +import threading + +import pytest + +from app.domain import content_types, feeds, netguard, rss, sources, store +from tests.helpers import drain, login, make_user + +R = store.R + +POSTS = { + "night": ( + "Night crossing", + "Tue, 29 Sep 2026 09:30:00 +0100", + "<p>The ships <script>steal()</script>arrived <a href='/pier' onclick='x()'>at dawn</a>." + "</p><p>The lighthouse keeper wrote twice about the fog.</p>", + ), + "buoys": ("New buoys", "Wed, 30 Sep 2026 14:00:00 +0000", "The new buoys are in, painted red."), + "regatta": ("Autumn regatta", "Thu, 01 Oct 2026 08:00:00 +0000", "Two days on the water. Bring the logbook."), +} + + +def _rss(keys, edits=None): + items = [] + for k in keys: + title, date, text = POSTS[k] + text = (edits or {}).get(k, text) + items.append( + f"{title}https://harbour.example.org/posts/{k}harbour-{k}" + f"{date}Mara KeaneHarbour" + f"{text}" + ) + return ( + "" + "Harbour newshttps://harbour.example.org/From the pier" + + "".join(reversed(items)) + + "" + ) + + +PAGE = """Harbour + +

Welcome to the harbour.

""" + + +class _Site(http.server.BaseHTTPRequestHandler): + feed = "" + etag = '"v1"' + auth: str | None = None + feed_path = "/feed.xml?key=s3cret" # where the page says the feed is: a private feed's address carries its key + hits: list = [] + + def do_GET(self): + _Site.hits.append((self.path, self.headers.get("Host"), self.headers.get("Authorization"), self.headers.get("If-None-Match"))) + if self.path == "/blog": + self._send(PAGE.format(feed=_Site.feed_path).encode(), "text/html; charset=utf-8") + elif self.path == "/plain": + self._send(b"No feed here", "text/html") + elif self.path.split("?")[0] == "/feed.xml": + if _Site.auth and self.headers.get("Authorization") != _Site.auth: + self.send_response(401) + self.end_headers() + elif self.headers.get("If-None-Match") == _Site.etag: + self.send_response(304) + self.end_headers() + else: + self._send(_Site.feed.encode(), "application/rss+xml", _Site.etag) + else: + self.send_response(404) + self.end_headers() + + def _send(self, body, ctype, etag=None): + self.send_response(200) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(body))) + if etag: + self.send_header("ETag", etag) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *a): + pass + + +@pytest.fixture +def site(cfg, monkeypatch): + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Site) + threading.Thread(target=srv.serve_forever, daemon=True).start() + _Site.feed, _Site.etag, _Site.auth, _Site.feed_path, _Site.hits = _rss(["night", "buoys"]), '"v1"', None, "/feed.xml?key=s3cret", [] + rss._FEEDS.clear() + monkeypatch.setattr(netguard, "PORTS", netguard.PORTS | {srv.server_address[1]}) # the test server's port, as if 443 + cfg["documents"]["web_networks"] = ["127.0.0.0/8"] # a network an admin allowed: the test server's + yield f"http://127.0.0.1:{srv.server_address[1]}" + srv.shutdown() + + +@pytest.fixture +def admin(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +def _segments(db, rid): + return " ".join(s["text"] for s in db.rows("SELECT idx, text FROM segment WHERE recording = $r ORDER BY idx", r=rid)) + + +def _stale(): + for hit in rss._FEEDS.values(): # past FEED_SECONDS: the next listing asks the server again + hit["at"] = 0 + + +def test_a_news_feed_as_a_source(site, client, db, cfg, admin): # site first: it sets what the app reads + h = admin + assert client.get("/api/v1/sources/backends", headers=h).json()["rss"]["label"] == "News feed (RSS or Atom)" + # the blog's page, not its feed: the feed it names is read + r = client.post("/api/v1/sources", headers=h, json={"name": "harbour", "type": "rss", "secrets": {"url": f"{site}/blog"}}) + assert r.status_code == 200 and r.json()["health"]["ok"], r.text + sid = r.json()["id"] + src = client.get("/api/v1/sources", headers=h).json()[0] + assert (src["type"], src["label"], src["secrets"]["url"]) == ("rss", "News feed (RSS or Atom)", {"secret": True, "set": True}) + + entries = client.get(f"/api/v1/sources/{sid}/browse", headers=h).json() + assert [e["name"] for e in entries] == ["2026-09-30 New buoys.atom", "2026-09-29 Night crossing.atom"] + night = entries[1] + assert (night["title"], night["when"], night["modified"], night["dir"]) == ( + "Night crossing", + "2026-09-29T08:30:00+00:00", + "2026-09-29T08:30:00+00:00", + False, + ) + assert site not in client.get(f"/api/v1/sources/{sid}/browse", headers=h).text # the address stays sealed + assert client.get(f"/api/v1/sources/{sid}/browse", params={"path": "x"}, headers=h).status_code == 400 + + # a post's file is an Atom entry of its own: the feed's HTML cleaned, its links made absolute + atom = b"".join(sources.stream(db, cfg, sid, night["path"])) + assert atom.startswith(b'\n') + assert b"harbour-night" in atom and b'href="https://harbour.example.org/posts/night"' in atom + assert b"steal()" not in atom and b"onclick" not in atom and f"{site}/pier".encode() in atom # read against the feed's address + assert b'Harbour news' in atom and b"s3cret" not in atom # ... which isn't kept + + # watched with backfill: every post becomes a text resource, titled and dated by the post + r = client.post("/api/v1/watches", headers=h, json={"source": sid, "namespace": "pods", "stable_seconds": 600, "backfill": True}) + assert r.status_code == 200, r.text + wid = r.json()["id"] + assert client.post("/api/v1/watches/preview", headers=h, json={"source": sid}).json()["transcripts"] == 2 + assert sources.poll_watch(db, cfg, wid)["new"] == 2 # a post is whole once listed: no waiting for it to settle + drain(db, cfg) + recs = {r["title"]: r for r in db.rows("SELECT record::id(id) AS id, title, source, recorded_at, status, remote, path FROM recording")} + assert set(recs) == {"Night crossing", "New buoys"} + one = recs["Night crossing"] + assert (one["source"], one["recorded_at"], one["status"]) == ("transcript", "2026-09-29T08:30:00+00:00", "analyzed") + assert one["remote"] == {"source": sid, "path": night["path"]} and one["path"] == f"harbour:{night['path']}" + said = _segments(db, one["id"]) + assert "https://harbour.example.org/posts/night" in said and "Mara Keane" in said and "Harbour news" in said + assert "keeper wrote twice about the fog" in said and "steal" not in said + types = content_types.all_types(db) + rec = db.one("SELECT source, path, title, media FROM $r", r=R("recording", one["id"])) + assert content_types.recognise(db, rec, types)["key"] == "feed_post" + + # asked again once the copy read is old: it hasn't changed (304), so nothing is new and nothing is parsed + _stale() + _Site.hits = [] + assert sources.poll_watch(db, cfg, wid)["new"] == 0 + assert [(p, etag) for p, _, _, etag in _Site.hits] == [("/feed.xml?key=s3cret", '"v1"')] + # a new post, and a post edited since + _Site.feed, _Site.etag = _rss(["night", "buoys", "regatta"], {"buoys": "The new buoys are in, painted red and white."}), '"v2"' + _stale() + assert sources.poll_watch(db, cfg, wid)["new"] == 2 + drain(db, cfg) + titles = sorted(r["title"] for r in db.rows("SELECT title FROM recording")) + assert titles == ["Autumn regatta", "New buoys", "Night crossing"] # the edited post read into the one it was + assert "red and white" in _segments(db, recs["New buoys"]["id"]) + + # a post the feed has dropped since: its file is still the copy fetched last + _Site.feed, _Site.etag = _rss(["regatta"]), '"v3"' + _stale() + assert sources.poll_watch(db, cfg, wid)["new"] == 0 + with pytest.raises(FileNotFoundError): + rss.fetch(cfg, sources.get(db, sid), night["path"]) + assert b"Night crossing" in b"".join(sources.stream(db, cfg, sid, night["path"], space=store.ns_id(db, "pods"))) + + # chosen on purpose, from the listing: already there + regatta = client.get(f"/api/v1/sources/{sid}/browse", headers=h).json()[0] + r = client.post("/api/v1/import/source", headers=h, json={"source": sid, "paths": [regatta["path"]], "namespace": "pods"}) + assert r.json()["results"][0]["status"] == "already" + + +def test_feed_addresses(site, client, db, cfg, admin, monkeypatch): + with monkeypatch.context() as m: + m.setattr(feeds, "_secret", lambda cfg, src, k: src[k]) + assert rss._url(cfg, {"url": "feed://harbour.example.org/rss"}) == "https://harbour.example.org/rss" + assert rss._url(cfg, {"url": "feed:https://harbour.example.org/rss"}) == "https://harbour.example.org/rss" + for bad in ("", "ftp://x/feed.xml", "/etc/passwd", "file:///etc/passwd", "https:///feed.xml"): + with pytest.raises(RuntimeError): + rss._url(cfg, {"url": bad}) + + def health(url, password=None, **params): + body = {"type": "rss", "params": params, "secrets": {"url": url, **({"pass": password} if password else {})}} + sid = client.post("/api/v1/sources", headers=admin, json=body).json()["id"] + rss._FEEDS.clear() + _Site.hits = [] + return sources.test(db, cfg, sid) # with this cfg, as the test changes it + + # what isn't public is refused, unless an admin allowed its network + cfg["documents"]["web_networks"] = [] + refused = health(f"{site}/feed.xml") + assert not refused["ok"] and "127.0.0.1 is on a private network" in refused["error"] and "Settings → Documents" in refused["error"] + cfg["documents"]["web_networks"] = ["127.0.0.0/8"] + assert health(f"{site}/feed.xml")["ok"] + # a page that names no feed, and an address that is no feed at all + plain = health(f"{site}/plain") + assert not plain["ok"] and "names no news feed" in plain["error"] + assert "404" in health(f"{site}/missing.xml")["error"] + + # a feed that asks for a password gets it; a feed the page names on another server never does + _Site.auth = "Basic " + base64.b64encode(b"lens:feed password 1").decode() + assert health(f"{site}/blog", "feed password 1", user="lens")["ok"] + assert [a for p, _, a, _ in _Site.hits if p.startswith("/feed.xml")][-1] == _Site.auth + real = socket.getaddrinfo # where localhost is ::1 too (Ubuntu), only its IPv4 address: the test server's + monkeypatch.setattr( + netguard.socket, + "getaddrinfo", + lambda host, *a, **k: real(host, *a, **({**k, "family": socket.AF_INET} if host == "localhost" else k)), + ) + _Site.feed_path = site.replace("127.0.0.1", "localhost") + "/feed.xml" + elsewhere = health(f"{site}/blog", "feed password 1", user="lens") + assert not elsewhere["ok"] and "401" in elsewhere["error"] + assert [(h.split(":")[0], a) for p, h, a, _ in _Site.hits] == [("127.0.0.1", _Site.auth), ("localhost", None)] + + +def test_posts_leave_memory(site, client, db, cfg, admin): + """A feed's posts don't stay in memory: its listing does, and a post wanted after that reads the feed in full.""" + r = client.post("/api/v1/sources", headers=admin, json={"name": "harbour", "type": "rss", "secrets": {"url": f"{site}/feed.xml"}}) + sid = r.json()["id"] + first = client.get(f"/api/v1/sources/{sid}/browse", headers=admin).json()[0] + (hit,) = rss._FEEDS.values() + assert set(hit["bodies"]) == set(hit["entries"]) + hit["read_at"] -= rss.KEEP_SECONDS + 1 + _Site.hits = [] + assert client.get(f"/api/v1/sources/{sid}/browse", headers=admin).json()[0] == first # the listing, from memory + assert hit["bodies"] is None and _Site.hits == [] + # a post wanted now: read in full, without asking whether it changed (a 304 would bring no posts) + assert b'New buoys' in rss.fetch(cfg, sources.get(db, sid), first["path"]) + assert [(p, etag) for p, _, _, etag in _Site.hits] == [("/feed.xml", None)] + (hit,) = rss._FEEDS.values() + assert set(hit["bodies"]) == set(hit["entries"]) + + +def test_what_a_feed_holds_stays_data(tmp_path): + """An entity naming a file of this machine is never read in, and a post's HTML keeps no scripts.""" + secret = tmp_path / "secret.txt" + secret.write_text("TOP SECRET") + feed = ( + f"]>" + "Harbour newsLeak &leak;x" + "a <img src=x onerror=steal()> &leak; b" + ).encode() + posts = rss.parse(feed, "https://harbour.example.org/feed.xml")["posts"] + kept = b"".join(body for _, body in posts.values()) + "".join(e["name"] for e, _ in posts.values()).encode() + assert b"TOP SECRET" not in kept and b"onerror" not in kept + assert b"TOP SECRET" not in rss.text(feed).encode() + + +def test_a_feed_file_imported_as_text(client, db, admin): + atom = b""" +Harbour newsurn:harbour2026-10-01T08:00:00Z + +Autumn regattaurn:harbour:regatta2026-10-01T08:00:00Z +<p>Two days on the water.</p> +""" + r = client.post( + "/api/v1/import", headers=admin, json={"namespace": "pods", "filename": "harbour.atom", "data": base64.b64encode(atom).decode()} + ) + assert r.status_code == 200, r.text + said = _segments(db, r.json()["id"]) + assert "Two days on the water." in said and "https://harbour.example.org/posts/regatta" in said diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock index 5df6b2925..ddd9c7f27 100644 --- a/fastapi_backend/uv.lock +++ b/fastapi_backend/uv.lock @@ -1178,6 +1178,27 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/05/99/49ee85903dee060d9f08297b4a342e5e0bcfca2f027a07b4ee0a38ab13f9/faster_whisper-1.2.1-py3-none-any.whl", hash = "sha256:79a66ad50688c0b794dd501dc340a736992a6342f7f95e5811be60b5224a26a7", size = 1118909, upload-time = "2025-10-31T11:35:47.794Z" }, ] +[[package]] +name = "feedparser" +version = "6.0.14" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "feedparser-sgmllib" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/37/8a/a53da4a77352045d277978a2df322d5379369f9deb1707178899ff7e1121/feedparser-6.0.14.tar.gz", hash = "sha256:088679b0c4b543ee211a820dd544698c76a402122eae7473c04a43425f283d06", size = 286108, upload-time = "2026-07-30T14:07:40.491Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/61/f04912e63702e73fb2a378f9c0a1ad9eb17a334a11a6b3fe1daa593903c2/feedparser-6.0.14-py3-none-any.whl", hash = "sha256:e35e3f760151b0c3b22cac9684155cae186a233e16c49bcbc6c49e91e3131137", size = 80668, upload-time = "2026-07-30T14:07:39.175Z" }, +] + +[[package]] +name = "feedparser-sgmllib" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/da/df/38596299216e5c22d60ed7f97902bb2bc72cfb95f732400f4fa976fd2e62/feedparser_sgmllib-2.1.0.tar.gz", hash = "sha256:61facf2918c4389b5b00714f76c5e03431ffcd94cd1f51d657edd6cd7c396579", size = 26845, upload-time = "2026-08-02T21:27:53.891Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/85/a0/79a31f898092e145bd66e2b338fb0656979acb2bbbcae8220940fbfcd820/feedparser_sgmllib-2.1.0-py3-none-any.whl", hash = "sha256:2cab2d43b95a954f920f18aebce7a4dbbb3f539780b127e2aa114f579821e01d", size = 11652, upload-time = "2026-08-02T21:27:52.894Z" }, +] + [[package]] name = "filelock" version = "4.0.7" @@ -1907,6 +1928,7 @@ dependencies = [ { name = "cryptography" }, { name = "fastapi", extra = ["standard"] }, { name = "fastapi-mail" }, + { name = "feedparser" }, { name = "jinja2" }, { name = "markdown" }, { name = "numpy" }, @@ -2033,6 +2055,7 @@ requires-dist = [ { name = "fastapi", extras = ["standard"], specifier = ">=0.143.0,<0.144" }, { name = "fastapi-mail", specifier = ">=1.4.1,<2" }, { name = "faster-whisper", marker = "extra == 'whisper'" }, + { name = "feedparser", specifier = ">=6.0.11,<7" }, { name = "funasr", marker = "extra == 'sensevoice'" }, { name = "impacket", marker = "extra == 'smb'", specifier = ">=0.13" }, { name = "insightface", marker = "extra == 'insightface'" }, diff --git a/nextjs-frontend/__tests__/sources-model.test.ts b/nextjs-frontend/__tests__/sources-model.test.ts index 8aba1334a..8502d41fd 100644 --- a/nextjs-frontend/__tests__/sources-model.test.ts +++ b/nextjs-frontend/__tests__/sources-model.test.ts @@ -219,6 +219,27 @@ describe("connections", () => { expect(suggestName("ical", cal.params)).toBe("Calendar feed"); expect([fileKind("Harbour report.eml"), fileKind("2026-10-02 Stand-up.ics")]).toEqual(["document", "transcript"]); }); + it("takes news feeds by their address, or their site's", () => { + const RSS: BackendSpec = { label: "News feed (RSS or Atom)", fields: { user: "" }, secrets: ["url", "pass"] }; + expect(sourceSubtitle({ type: "rss", params: { user: "" } })).toBe("News feed · address kept secret"); + expect(validateForm("rss", RSS, emptyForm(RSS), false)).toHaveProperty("url"); + const feed = { ...emptyForm(RSS), secrets: { url: "https://harbour.example.org/feed.xml?key=s3cret" } }; + expect(validateForm("rss", RSS, feed, false)).toEqual({}); + for (const ok of [ + "feed://harbour.example.org/rss", + "feed:https://harbour.example.org/rss", + "http://harbour.example.org", + ]) + expect(validateForm("rss", RSS, { ...feed, secrets: { url: ok } }, false)).toEqual({}); + expect(validateForm("rss", RSS, { ...feed, secrets: { url: "harbour.example.org" } }, false)).toHaveProperty("url"); + expect(validateForm("rss", RSS, emptyForm(RSS), true, (k) => k === "url")).toEqual({}); + expect(buildPayload(RSS, feed, "rss").secrets).toEqual({ url: "https://harbour.example.org/feed.xml?key=s3cret" }); + // named by its server, never by the rest of its address + expect(suggestName("rss", feed.params, feed.secrets)).toBe("News feed · harbour.example.org"); + expect(suggestName("rss", {}, { url: "feed:https://news.example.org/atom" })).toBe("News feed · news.example.org"); + expect(suggestName("rss", {})).toBe("News feed"); + expect(fileKind("2026-09-29 Night crossing.atom")).toBe("transcript"); + }); it("signs Google Calendar in as a Google account or a service account", () => { const GCAL: BackendSpec = { label: "Google Calendar", diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 29b5fd8c8..ce455826b 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -16513,6 +16513,7 @@ export type RcloneSignInStart = { | "gmail" | "ical" | "gcal" + | "rss" | null; /** * Source @@ -20566,7 +20567,19 @@ export type Source = { * Type */ type: - "s3" | "dropbox" | "drive" | "onedrive" | "sftp" | "smb" | "webdav" | "local" | "imap" | "gmail" | "ical" | "gcal"; + | "s3" + | "dropbox" + | "drive" + | "onedrive" + | "sftp" + | "smb" + | "webdav" + | "local" + | "imap" + | "gmail" + | "ical" + | "gcal" + | "rss"; /** * Label */ @@ -20611,7 +20624,19 @@ export type SourceCreate = { * Type */ type: - "s3" | "dropbox" | "drive" | "onedrive" | "sftp" | "smb" | "webdav" | "local" | "imap" | "gmail" | "ical" | "gcal"; + | "s3" + | "dropbox" + | "drive" + | "onedrive" + | "sftp" + | "smb" + | "webdav" + | "local" + | "imap" + | "gmail" + | "ical" + | "gcal" + | "rss"; /** * Params */ diff --git a/nextjs-frontend/components/library/source-labels.ts b/nextjs-frontend/components/library/source-labels.ts index a0fc17ea5..304aeb7d2 100644 --- a/nextjs-frontend/components/library/source-labels.ts +++ b/nextjs-frontend/components/library/source-labels.ts @@ -11,6 +11,8 @@ export const SOURCE_TYPE_LABEL: Record = { imap: "Email", gmail: "Gmail", ical: "Calendar", + gcal: "Google Calendar", + rss: "News feed", }; export function sourceTypeLabel(type: string | null | undefined, fallback = "Source"): string { diff --git a/nextjs-frontend/components/library/sources-strip.tsx b/nextjs-frontend/components/library/sources-strip.tsx index daadf2e49..e314de38e 100644 --- a/nextjs-frontend/components/library/sources-strip.tsx +++ b/nextjs-frontend/components/library/sources-strip.tsx @@ -10,6 +10,7 @@ import { ListOrdered, Mail, Network, + Rss, Server, type LucideIcon, } from "lucide-react"; @@ -33,6 +34,8 @@ const ICON: Record = { imap: Mail, gmail: Mail, ical: CalendarDays, + gcal: CalendarDays, + rss: Rss, }; type ChipState = { meta: string; tone: "muted" | "red"; tip: string }; diff --git a/nextjs-frontend/components/sources/connection-dialog.tsx b/nextjs-frontend/components/sources/connection-dialog.tsx index f0ffaa0f6..6965ccc12 100644 --- a/nextjs-frontend/components/sources/connection-dialog.tsx +++ b/nextjs-frontend/components/sources/connection-dialog.tsx @@ -161,7 +161,7 @@ export function ConnectionDialog({ const body = buildPayload(backends[t], f, t); sent = body.secrets; if (id == null) { - name = f.name.trim() || suggestName(t, f.params); + name = f.name.trim() || suggestName(t, f.params, f.secrets); const r = await data( Sources.createSource({ client, @@ -241,7 +241,7 @@ export function ConnectionDialog({ let id = createdId; const body = buildPayload(backends[type], form, type); if (id == null) { - const name = form.name.trim() || suggestName(type, form.params); + const name = form.name.trim() || suggestName(type, form.params, form.secrets); id = (await data(Sources.createSource({ client, body: { type, name, ...body } }))).id; setCreatedId(id); setCreatedName(name); diff --git a/nextjs-frontend/components/sources/connection-fields.tsx b/nextjs-frontend/components/sources/connection-fields.tsx index 128c2152a..6e7db5189 100644 --- a/nextjs-frontend/components/sources/connection-fields.tsx +++ b/nextjs-frontend/components/sources/connection-fields.tsx @@ -566,6 +566,23 @@ export function ConnectionFields({

); + case "rss": + return ( +
+ {secret("url", { + label: "Feed address", + hint: "The RSS or Atom feed’s address, or the site’s: Lens finds the feed a page names. Kept like a password, since a private feed’s address carries its key", + })} +
+ {text("user", { optional: true })} + {secret("pass", { label: "Password", hint: "Only if the feed asks for one" })} +
+

+ Each post becomes text: its title, link, author, date and what the feed says of it. Watch the feed to bring + in new posts as they’re published. +

+
+ ); case "gmail": { const keyText = form.secrets.service_account_credentials; const key = keyText ? parseServiceAccountKey(keyText) : null; diff --git a/nextjs-frontend/components/sources/source-icons.tsx b/nextjs-frontend/components/sources/source-icons.tsx index aeb15f226..0f02ac99a 100644 --- a/nextjs-frontend/components/sources/source-icons.tsx +++ b/nextjs-frontend/components/sources/source-icons.tsx @@ -10,6 +10,7 @@ import { type LucideIcon, Mail, Network, + Rss, Server, } from "lucide-react"; @@ -28,6 +29,7 @@ export const TYPE_ICON: Record = { gmail: Mail, ical: CalendarDays, gcal: CalendarCheck, + rss: Rss, }; /** The connection's type in a 32px rounded tile. */ diff --git a/nextjs-frontend/components/sources/source-model.ts b/nextjs-frontend/components/sources/source-model.ts index 27e109d1d..3fd4f5a26 100644 --- a/nextjs-frontend/components/sources/source-model.ts +++ b/nextjs-frontend/components/sources/source-model.ts @@ -20,6 +20,7 @@ export const TYPE_ORDER: SourceType[] = [ "gmail", "ical", "gcal", + "rss", ]; /** Short names for the type picker and rows (the backend's labels are longer). */ @@ -36,6 +37,7 @@ export const TYPE_NAME: Record = { gmail: "Gmail", ical: "Calendar feed", gcal: "Google Calendar", + rss: "News feed", }; /** How to name the storage itself in sentences ("Files in the bucket aren’t touched"). */ @@ -52,6 +54,7 @@ export const STORAGE_NOUN: Record = { gmail: "the mailbox", ical: "the calendar", gcal: "Google Calendar", + rss: "the feed", }; /** Field labels for the backend's parameter and secret names. */ @@ -170,6 +173,8 @@ export function sourceSubtitle(s: Pick): string { return `Gmail · ${str(p.user) || "no address"} · ${str(p.via) === "imap" ? "IMAP" : "API"}`; case "ical": return "Calendar feed · address kept secret"; + case "rss": + return "News feed · address kept secret"; case "gcal": { const chosen = str(p.calendars) .split(/[,\s]+/) @@ -271,7 +276,7 @@ const AUDIO = [ ".m4v", ".avi", ]; -const TRANSCRIPT = [".srt", ".vtt", ".json", ".jsonl", ".ics"]; +const TRANSCRIPT = [".srt", ".vtt", ".json", ".jsonl", ".ics", ".atom"]; const DOCUMENT = [ ".pdf", ".doc", @@ -323,7 +328,8 @@ export type SourceFileKind = "audio" | "transcript" | "document" | "image" | "ot /** * What a watched folder would make of a file, by extension (the backend's default lists): PDFs, Office files, text, * Markdown, web pages and emails are documents, as they are for new watched folders (those set to audio and - * transcripts read PDFs, Word and text files as transcripts); subtitles, JSON and calendar events are transcripts. + * transcripts read PDFs, Word and text files as transcripts); subtitles, JSON, calendar events and feed posts are + * transcripts. */ export function fileKind(name: string): SourceFileKind { const n = name.toLowerCase(); @@ -526,10 +532,20 @@ export function emptyForm(spec: BackendSpec, source?: Pick): string { +/** A name for a new connection, from what was typed ("SFTP · calls-gw", "S3 · eu-central-1", "Dropbox", "News feed · + * harbour.example.org"). A feed's or calendar's address is a secret, but its server names it well. */ +export function suggestName( + type: SourceType, + params: Record, + secrets: Record = {}, +): string { if (type === "gmail") return params.user?.trim() ? `Gmail · ${params.user.trim()}` : "Gmail"; - const host = (params.host || params.url || "").replace(/^(https?|webcals?):\/\//i, "").split(/[/:]/)[0]; + const address = params.host || params.url || (type === "rss" || type === "ical" ? (secrets.url ?? "") : ""); + const host = address + .trim() + .replace(/^feed:(?=https?:)/i, "") + .replace(/^(https?|webcals?|feed):\/\//i, "") + .split(/[/:?#]/)[0]; const extra = type === "s3" ? [params.provider !== "AWS" ? params.provider : "", params.region].filter(Boolean).join(" ") : host; return extra @@ -605,6 +621,11 @@ export function validateForm( if (url === undefined ? !(editing && isSet("url")) : !/^(https?|webcals?):\/\/\S+$/i.test(url.trim())) e.url = "Enter the calendar’s address, starting with https:// or webcal://"; } + if (type === "rss") { + const url = form.secrets.url; + if (url === undefined ? !(editing && isSet("url")) : !/^(feed:)?(https?|feed):\/\/\S+$/i.test(url.trim())) + e.url = "Enter the feed’s address, or the site’s, starting with https://"; + } return e; } diff --git a/nextjs-frontend/components/storage/file-store.ts b/nextjs-frontend/components/storage/file-store.ts index deb765794..fe906eae6 100644 --- a/nextjs-frontend/components/storage/file-store.ts +++ b/nextjs-frontend/components/storage/file-store.ts @@ -1,7 +1,7 @@ /** Where Lens keeps its own files (Settings → Storage, the backend's `files` section: app/domain/blobs.py). */ /** Connections files can't be kept on: email accounts and calendars are read, not written (feeds.py). */ -export const NOT_STORAGE = new Set(["imap", "gmail", "ical"]); +export const NOT_STORAGE = new Set(["imap", "gmail", "ical", "gcal", "rss"]); export type FileStoreTry = { store: "local"; crypt: false } | { store: "connection"; connection: number | null; folder: string; crypt: boolean }; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index ec8c4b0f4..ffd031dcf 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -59444,7 +59444,8 @@ "imap", "gmail", "ical", - "gcal" + "gcal", + "rss" ] }, { @@ -66441,7 +66442,8 @@ "imap", "gmail", "ical", - "gcal" + "gcal", + "rss" ], "title": "Type" }, @@ -66538,7 +66540,8 @@ "imap", "gmail", "ical", - "gcal" + "gcal", + "rss" ], "title": "Type" },