Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
account, which reads the calendars shared with it or, in Google Workspace, a user's calendars through domain-wide
delegation. Calls to Google show in the activity ledger
([Google Calendar](docs/processing.md#email-imap-calendar-feeds-ical-and-google-calendar)).
- **Gmail and Google Workspace as a sensor.** A new kind of email source reads a Gmail or Google Workspace mailbox
through Google's Gmail API (read only) or over IMAP. It signs in as a Google account (its authorized user JSON, as
for Google Calendar), or as a Workspace service account with domain-wide delegation, so nobody has to sign in.
Labels are its folders, and a watch follows Gmail's history: it reads only what arrived, and one that leaves what's
there reads nothing old. Connecting the account from the connection dialog is in progress
([Gmail and Google Workspace](docs/processing.md#gmail-and-google-workspace)).
- **Find a note.** The Notes page has a search box that finds notes by the words in their title, summary or text (the
pages of recordings and entities too), with the words around each match marked, and Enter opens the first. A filter
shows only Projects, Areas, Resources or Archive, and each note says which notes it's inside. Jump to (⌘K) finds notes
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,12 @@ Everything below is built and merged. Work that is still in a pull request is li
([Extending the assistant](docs/assistant.md#extending-the-assistant-tools-skills-hooks-and-plugins),
[the MCP runner](docs/assistant.md#the-mcp-runner))
- **Bring in everything.** Watched folders on S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV or local disks
(a local folder in four clicks); IMAP mail, iCal calendars and Google Calendar; a built-in MQTT broker, syslog and webhook streams;
import webhooks for scanners and phone shortcuts. Upload audio, video, PDFs, Office files, photos, JSON and zip or
tar archives, and, with an Apache Tika server (an opt-in extra), documents nothing else here can read; capture web
pages one at a time, from a link list or a bookmarks file, by API or by asking the assistant, from private networks
you allow in Settings too; and import videos and podcasts by their address (off by default).
(a local folder in four clicks); IMAP mail, Gmail and Google Workspace, iCal calendars and Google Calendar; a built-in
MQTT broker, syslog and webhook streams; import webhooks for scanners and phone shortcuts. Upload audio, video, PDFs,
Office files, photos, JSON and zip or tar archives, and, with an Apache Tika server (an opt-in extra), documents
nothing else here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking
the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off
by default).
([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika),
[import webhooks](docs/api.md#import-webhooks))
- **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a
Expand Down
3 changes: 2 additions & 1 deletion docs/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ its namespace, its pipeline and the routine. A request counts for the resources
/api/v1/routines/3` is `routine:3`), for what a `POST` made (from the `id` in its reply), and for who sent it.

Besides the resources, a row has its kind (`in`, `out` or `run`), what it was (`POST /api/v1/routines/{rid}`,
`model.chat`, `embeddings`, `decision`, `notify.webhook`, `tool.http`, `job.succeeded`, `routine.done`), the model,
`model.chat`, `embeddings`, `decision`, `notify.webhook`, `tool.http`, `google.token`, `gmail.list`, `job.succeeded`,
`routine.done`), the model,
tokens in and out, the estimated cost in USD, how long it took, and how it ended (an error's type or HTTP status,
never its message). It never holds prompt, reply, asset text or file content.

Expand Down
37 changes: 36 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ A folder scan and a routine run are each claimed first, so several processes doi
## Storage sources

Admins add sources in the app (`/api/v1/sources`): S3 or S3-compatible, Dropbox, Google Drive, OneDrive, SFTP, SMB,
WebDAV, or a folder on this machine; or an email account (IMAP), a calendar feed (iCal) or Google Calendar, below. A watched folder maps a path on a source to a namespace, with include/exclude
WebDAV, or a folder on this machine; or an email account (IMAP, or Gmail and Google Workspace), a calendar feed (iCal) or Google Calendar, below. A watched folder maps a path on a source to a namespace, with include/exclude
patterns, audio and/or transcripts, a polling interval, how long a file must be unchanged before it is picked up, and
whether files already there are imported (backfill). New audio is queued for the full pipeline; new transcripts are
imported and analysed. Audio stays where it is: it is copied to a cache for processing and streamed from the source for
Expand Down Expand Up @@ -335,6 +335,7 @@ importing chosen ones and watching work as above.
exists (rules never create one); mail for a namespace deleted since goes to the watch's. Rules apply to messages that
arrive after they're saved. `POST /api/v1/watches/routes/preview` shows where the latest messages would go before
saving, and a scan's stats count the messages `routed` elsewhere.
- **Gmail and Google Workspace**: see [below](#gmail-and-google-workspace).
- **Calendar feed (iCal)**: the calendar's iCal address (`https://` or `webcal://`; a user and password if it asks for
one). The address is kept encrypted like a password and never shown again, since a private calendar's address is
all it takes to read it. It is fetched the way web pages are captured: public addresses only (and the networks in
Expand Down Expand Up @@ -369,6 +370,40 @@ importing chosen ones and watching work as above.
a Google calendar's secret address in iCal format (Settings › Integrate calendar) works as a calendar feed.
- `.eml` and `.ics` files in any source, and uploaded through Import, can be read as text this way too.

### Gmail and Google Workspace

A Gmail or Google Workspace mailbox (type `gmail`) is read through Google's Gmail API, read only (`gmail.readonly`),
or over IMAP (`via: imap`). For a personal Gmail account, an email (IMAP) connection with an app password works too,
with nothing to set up at Google. A Gmail source signs in one of two ways:

- **Google account** (`token`): the "authorized user" JSON (`client_id`, `client_secret`, `refresh_token`) of an
OAuth client of your own Google Cloud project, with the Gmail API turned on there, kept encrypted like a password.
`gcloud auth application-default login --client-id-file=client.json --scopes=https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/cloud-platform`
writes one (with `https://mail.google.com/` instead to read over IMAP); an agent can give one it already has. Enter
the account's address as `user` to read over IMAP. Google reviews apps that read Gmail before people outside your
Workspace domain can use them without a warning: an app in testing works for its test users, whose sign-ins Google
ends after seven days. Connecting the account from the connection dialog, with no JSON to make, is in progress.
- **A service account** (Google Workspace). Upload the service account's JSON key and enter the address of the person
whose mail Lens reads; nobody signs in. A Workspace admin lets the account's client id use
`https://www.googleapis.com/auth/gmail.readonly` (or `https://mail.google.com/` to read over IMAP) under Admin
console › Security › API controls › Domain-wide delegation. One source reads one person's mailbox.

Labels are the folders: the inbox, sent, starred, important, the inbox's categories (Primary, Social, Promotions,
Updates, Forums) and your own labels. A message is a file `<label id>/<message id>.eml`, read as Gmail keeps it, so it
becomes an email document as an IMAP message does. Watching the whole mailbox (no path) takes everything but spam, the
bin, drafts and chats, each message once. A watch follows Gmail's history: a scan asks only for what arrived since the
last one (and, on a label, messages the label was put on since), and a watch that doesn't take what's there already
reads nothing old at all. Messages are taken as soon as Gmail lists them, with no wait for them to settle. When Gmail
no longer keeps the history a watch points into (it keeps about a week), the watch reads what arrived from an hour
before its last scan on, and skips what it already has. Browsing a label shows its newest 200 messages; Import → From a
source takes any message chosen. Routing rules work as for IMAP, and a message's `mailbox` is its labels.

Over IMAP, a Gmail source works as an IMAP account does, at imap.gmail.com with the Google sign-in (XOAUTH2). That
needs Google's all-of-Gmail scope, since Google has no read-only one for IMAP; Lens still only reads. Changing `via`
starts the source's watches again as new ones (taking what's there only if they backfill), since the two name folders
and messages differently: check their folders. Each access token Google gives, each listing and each message read is a
row in the activity ledger (`google.token`, `gmail.list`, `gmail.read`); none costs anything.

## Pipelines and templates

A pipeline is a named, versioned list of steps; each namespace can choose its default (otherwise: transcribe, diarize,
Expand Down
2 changes: 1 addition & 1 deletion docs/sensors.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A **sensor** is anything that feeds Lens. There are two families, in one list (`

| Family | Kinds | Channels | What it reports becomes |
| --- | --- | --- | --- |
| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP); calendar feeds (iCal); Google Calendar | the folders it watches | resources, run through their pipelines (as sources always have) |
| Files | S3, Dropbox, Google Drive, OneDrive, SFTP, SMB, WebDAV, a local folder (through rclone); email (IMAP, Gmail and Google Workspace); calendar feeds (iCal); Google Calendar | the folders it watches | resources, run through their pipelines (as sources always have) |
| Streams | MQTT devices, syslog senders, webhooks | streams: an MQTT topic, a syslog program, a webhook's stream name | readings, kept with retention, and hourly summaries of anything numeric |

File sensors are the storage sources: they are added and changed as sources (`/api/v1/sources`), and nothing about
Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/api/v1/routes/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def _watch(db: DB, wid: int) -> None:
@router.get("/sources/backends")
def list_backends(user: CurrentUser) -> dict[str, Backend]:
"""The kinds of storage a source can be, with their settings and secrets."""
keys = {"label", "fields", "secrets", "oauth"}
keys = {"label", "fields", "secrets", "oauth", "google"}
return {k: Backend.model_validate({f: x for f, x in v.items() if f in keys}) for k, v in sources.BACKENDS.items()}


Expand Down Expand Up @@ -134,7 +134,7 @@ def create_watch(body: WatchCreate, user: AdminWriter, db: Db, cfg: Cfg) -> Crea
def preview_watch(body: WatchPreviewRequest, user: AdminWriter, db: Db, cfg: Cfg) -> WatchPreview:
"""How many files a watched folder would pick up (the backfill count), before creating it."""
try:
files = sources.list_files(db, cfg, sources.get(db, body.source), body.path)
files = sources.list_files(db, cfg, sources.get(db, body.source), body.path, headers=False)
except KeyError:
raise HTTPException(400, "no such source") from None
except (ValueError, RuntimeError) as e:
Expand Down
Loading
Loading