Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Google Calendar as a sensor.** Add a Google Calendar connection on the Sources page (or `POST /api/v1/sources`
with type `gcal`): each calendar is a folder and each event becomes text, as an iCal feed's do, read-only through
Google's Calendar API. It signs in as a Google account (an OAuth client's authorized user JSON) or as a service
account, which reads the calendars shared with it or, in Google Workspace, a user's calendars through domain-wide
delegation. Calls to Google show in the activity ledger
([Google Calendar](docs/processing.md#email-imap-calendar-feeds-ical-and-google-calendar)).

## 0.10.0 <small>October 15, 2026</small> {id="0.10.0"}

- **The assistant always answers.** When a question used up its tool steps (`ai.max_steps`), the assistant used to stop
Expand Down Expand Up @@ -836,7 +843,7 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
`POST …/{tid}/secret` and `GET …/{tid}/deliveries` (docs/api.md).
- **Email and calendars as sources.** Sources can now be an email account (IMAP) or a calendar feed (iCal), next to
S3, Drive, SFTP and the rest. Browse them, import chosen messages or events, or watch them like a folder
(docs/processing.md#email-imap-and-calendar-feeds-ical).
(docs/processing.md#email-imap-calendar-feeds-ical-and-google-calendar).
- IMAP: mailboxes are folders and messages are `.eml` files named by their subjects. Each message becomes an
email document, titled by its subject and dated when it was sent, and its attachments are kept and become
resources of their own. Watching the whole account skips the bin, junk, drafts and Gmail's All Mail, Starred
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Everything below is built and merged. Work that is still in a pull request is li
([Extending the assistant](docs/assistant.md#extending-the-assistant-tools-skills-hooks-and-plugins),
[the MCP runner](docs/assistant.md#the-mcp-runner))
- **Bring in everything.** Watched folders on S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV or local disks
(a local folder in four clicks); IMAP mail and iCal calendars; a built-in MQTT broker, syslog and webhook streams;
(a local folder in four clicks); IMAP mail, iCal calendars and Google Calendar; a built-in MQTT broker, syslog and webhook streams;
import webhooks for scanners and phone shortcuts. Upload audio, video, PDFs, Office files, photos, JSON and zip or
tar archives, and, with an Apache Tika server (an opt-in extra), documents nothing else here can read; capture web
pages one at a time, from a link list or a bookmarks file, by API or by asking the assistant, from private networks
Expand Down
2 changes: 1 addition & 1 deletion docs/assistant-email.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ subject: newsletter = skip

A rule's conditions are on `from`, `to` (To, Cc and the address it arrived at), `subject`, `list` (List-Id) and
`mailbox`, joined by `;`, with patterns joined by `,`. Patterns work as in an email watch's routing rules
([Processing](processing.md#email-imap-and-calendar-feeds-ical)): one without `*` or `?` matches anywhere in the
([Processing](processing.md#email-imap-calendar-feeds-ical-and-google-calendar)): one without `*` or `?` matches anywhere in the
text. `skip` leaves the email unanswered. When no rule matches, or a rule names a namespace that doesn't exist, Lens
picks the namespace for each question, as in an open chat room ([Chat rooms](chat-rooms.md)).

Expand Down
29 changes: 26 additions & 3 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ A folder scan and a routine run are each claimed first, so several processes doi
## Storage sources

Admins add sources in the app (`/api/v1/sources`): S3 or S3-compatible, Dropbox, Google Drive, OneDrive, SFTP, SMB,
WebDAV, or a folder on this machine; or an email account (IMAP) or a calendar feed (iCal), below. A watched folder maps a path on a source to a namespace, with include/exclude
WebDAV, or a folder on this machine; or an email account (IMAP), a calendar feed (iCal) or Google Calendar, below. A watched folder maps a path on a source to a namespace, with include/exclude
patterns, audio and/or transcripts, a polling interval, how long a file must be unchanged before it is picked up, and
whether files already there are imported (backfill). New audio is queued for the full pipeline; new transcripts are
imported and analysed. Audio stays where it is: it is copied to a cache for processing and streamed from the source for
Expand All @@ -311,9 +311,9 @@ files, then Import (`POST /api/v1/import/source`). The listing marks files that
- Folders on this machine can only be watched inside `sources.local_roots`, and the rclone binary can only be set in
archive.yaml: the web app can neither open up the server's disk nor choose what runs.

### Email (IMAP) and calendar feeds (iCal)
### Email (IMAP), calendar feeds (iCal) and Google Calendar

Two kinds of source aren't storage and don't use rclone; their messages and events are shown as files, so browsing,
Three kinds of source aren't storage and don't use rclone; their messages and events are shown as files, so browsing,
importing chosen ones and watching work as above.

- **Email (IMAP)**: host, port, security (SSL/TLS, STARTTLS or none), user and password (an app password where the
Expand Down Expand Up @@ -344,6 +344,29 @@ importing chosen ones and watching work as above.
attendees, how it repeats, and its description), dated when it starts: an all-day event on its date, wherever you
are. Time zones are read as IANA names, Windows' names (as Outlook writes them), or the calendar's own VTIMEZONE. An event that changes (its LAST-MODIFIED,
or its size) is read again into the resource it already is, rather than made a second one.
- **Google Calendar** (`gcal`): read through Google's Calendar API with read-only access (`calendar.readonly`), so
nothing in the calendar is changed. Each calendar is a folder and each event a file `<calendar id>/<event id>.ics`,
named by its date and title, and read as text exactly as an iCal feed's events are; it also says which occurrences
of a repeating event were cancelled and the event's video call. A changed occurrence of a repeating event is an event
of its own, as Google keeps it; working-location entries ("Home", "Office") are left out. It signs in one of two ways,
each kept encrypted like a password and never shown again:

- **Google account** (`token`): the "authorized user" JSON (`client_id`, `client_secret`, `refresh_token`) of an
OAuth client of your own Google Cloud project, with the Calendar API turned on there. `gcloud auth
application-default login --client-id-file=client.json --scopes=https://www.googleapis.com/auth/calendar.readonly,https://www.googleapis.com/auth/cloud-platform`
writes one (to `application_default_credentials.json` in gcloud's folder).
- **Service account** (`service_account_credentials`, its .json key file). On its own it reads the calendars shared
with its address: share them, and give their IDs in `calendars` (Google Calendar › Settings › Integrate
calendar). In Google Workspace it can read a user's calendars instead (`impersonate`), once an admin gave it
domain-wide delegation for the `calendar.readonly` scope (Admin console › Security › API controls).

`calendars` (IDs, comma separated) limits the source to those; empty, it reads every calendar in the account's
list, and watching the whole account (no path) leaves out Google's own (holidays, birthdays, week numbers). Events
that ended more than `days_back` days ago (365) are left out; empty takes every event. A watch lists each calendar
again every poll (`events.list`, 2,500 events a call). Every call to Google is a row in the activity ledger
(`google.token`, `google.calendar`); Google throttling the account shows as working, slowly, and a scan it
interrupts is tried again at the next poll. The token endpoint is always Google's own, whatever a key file says. Without a Google Cloud project,
a Google calendar's secret address in iCal format (Settings › Integrate calendar) works as a calendar feed.
- `.eml` and `.ics` files in any source, and uploaded through Import, can be read as text this way too.

## Pipelines and templates
Expand Down
2 changes: 1 addition & 1 deletion docs/sensors.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A **sensor** is anything that feeds Lens. There are two families, in one list (`

| Family | Kinds | Channels | What it reports becomes |
| --- | --- | --- | --- |
| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP); calendar feeds (iCal) | the folders it watches | resources, run through their pipelines (as sources always have) |
| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP); calendar feeds (iCal); Google Calendar | the folders it watches | resources, run through their pipelines (as sources always have) |
| Streams | MQTT devices, syslog senders, webhooks | streams: an MQTT topic, a syslog program, a webhook's stream name | readings, kept with retention, and hourly summaries of anything numeric |

File sensors are the storage sources: they are added and changed as sources (`/api/v1/sources`), and nothing about
Expand Down
16 changes: 13 additions & 3 deletions fastapi_backend/app/domain/calendars.py
Original file line number Diff line number Diff line change
Expand Up @@ -350,18 +350,28 @@ def _span(event):
return f"{_shown(s)} to {_shown(end)}"


def _skipped(event):
"""The occurrences of a repeating event that don't happen (its EXDATEs), in order."""
out = [when(p, x, event.get("zones")) for p, v in all_of(event, "EXDATE") for x in (v or "").split(",")]
return sorted((t for t in out if t is not None), key=iso)


def event_text(event):
"""An event as Markdown: its summary as the heading, when and where, who, how it repeats, then its description.
The details are 'Label — value' lines, so the transcript reader doesn't take them for speakers."""
"""An event as Markdown: its summary as the heading, when and where, who, how it repeats (and the occurrences that
don't happen), its video call, then its description. The details are 'Label — value' lines, so the transcript
reader doesn't take them for speakers."""
title = summary(event) or "(no title)"
rows = [("When", _span(event))]
rows.append(("Where", unescape(first(event, "LOCATION")[1] or "").strip()))
p, v = first(event, "ORGANIZER")
rows.append(("Organizer", _person(p, v) if v else ""))
rows.append(("Attendees", ", ".join(_person(p, v) for p, v in all_of(event, "ATTENDEE"))))
rows.append(("Repeats", (first(event, "RRULE")[1] or "").strip()))
rule = (first(event, "RRULE")[1] or "").strip()
rows.append(("Repeats", rule))
rows.append(("Except", "; ".join(_shown(t) for t in _skipped(event)) if rule else ""))
status = (first(event, "STATUS")[1] or "").strip().lower()
rows.append(("Status", status if status and status != "confirmed" else ""))
rows.append(("Meeting", (first(event, "CONFERENCE")[1] or "").strip()))
rows.append(("Link", (first(event, "URL")[1] or "").strip()))
head = [f"{k} — {v}" for k, v in rows if v]
desc = unescape(first(event, "DESCRIPTION")[1] or "").strip()
Expand Down
19 changes: 16 additions & 3 deletions fastapi_backend/app/domain/feeds.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""Sources that aren't file storage: IMAP mailboxes and iCalendar feeds, shown as folders of files so browsing,
choosing files, watching and importing work for them as they do for rclone's storage (sources.py).
"""Sources that aren't file storage: IMAP mailboxes, iCalendar feeds and Google Calendar (gcal.py), shown as folders of
files so browsing, choosing files, watching and importing work for them as they do for rclone's storage (sources.py).

- An IMAP account's folders are its mailboxes, and each message is a file, `<mailbox>/<uidvalidity>-<uid>.eml`: a
UID stands for one message only while the mailbox's UIDVALIDITY stays the same, so a mailbox the server rebuilt
Expand Down Expand Up @@ -34,8 +34,10 @@
import urllib.request
from email import policy

from . import calendars, netguard
from . import calendars, gcal, netguard

# sources a module of their own handles: browse, list_files, fetch, immutable and test, as below
OWN = {"gcal": gcal}
TYPES = {
"imap": {
"label": "Email (IMAP)",
Expand All @@ -47,6 +49,7 @@
"fields": {"user": ""},
"secrets": ["url", "pass"],
},
**{k: m.TYPE for k, m in OWN.items()},
}
SECURITY = ("ssl", "starttls", "none")
# not taken when a whole account is watched: what can't be opened, the bin, junk and drafts, and the mailboxes that
Expand Down Expand Up @@ -487,27 +490,37 @@ def check_path(path):


def browse(cfg, src, path=""):
if src["type"] in OWN:
return OWN[src["type"]].browse(cfg, src, check_path(path))
return (_imap_browse if src["type"] == "imap" else _ical_browse)(cfg, src, check_path(path))


def list_files(cfg, src, path="", cursor=None):
"""Every file under `path`; for IMAP with a watch's `cursor` (a list of {mailbox, validity, uid}), only messages
after it."""
path = check_path(path)
if src["type"] in OWN:
return OWN[src["type"]].list_files(cfg, src, path, cursor)
if src["type"] == "imap":
return _imap_files(cfg, src, path, {c["mailbox"]: c for c in cursor or []})
return _ical_browse(cfg, src, path)


def fetch(cfg, src, path):
"""The file's bytes: a message as the server keeps it, an event as an .ics of its own."""
if src["type"] in OWN:
return OWN[src["type"]].fetch(cfg, src, check_path(path))
return (_imap_fetch if src["type"] == "imap" else _ical_fetch)(cfg, src, check_path(path))


def immutable(src):
"""Whether a file, once fetched, never changes (an IMAP message), so a cached copy can be kept."""
if src["type"] in OWN:
return OWN[src["type"]].immutable(src)
return src["type"] == "imap"


def test(cfg, src):
if src["type"] in OWN:
return OWN[src["type"]].test(cfg, src)
(_imap_test if src["type"] == "imap" else _ical_test)(cfg, src)
Loading
Loading