Skip to content

feat(sources): Gmail and Google Workspace as an email sensor - #405

Merged
adeelahmad merged 7 commits into
mainfrom
claude/project-thread-yergon
Oct 15, 2026
Merged

adeelahmad merged 7 commits into
mainfrom
claude/project-thread-yergon

Conversation

@adeelahmad

@adeelahmad adeelahmad commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: Gmail and Google Workspace mail could reach Lens only through a plain IMAP source with an app password. Workspace domains that turn app passwords off had no way in.

After: a new gmail source type reads a Gmail or Google Workspace mailbox as a sensor, through the Gmail API (read only, the default) or over IMAP with XOAUTH2 (via). It signs in one of two ways, through the shared google_auth.py that Google Calendar (#399) uses:

  • As a Google account (OAuth). The token secret holds the account's authorized user JSON (refresh_token, client_id, client_secret), as for Google Calendar. An agent can hand over one it already has.
  • As a Workspace service account with domain-wide delegation. Upload the .json key as service_account_credentials and set user to the person it acts as. Google's refusals are explained in plain words, for example which client id an admin must allow, for which scopes, and where in the Admin console.

Connecting a Google account from the browser comes with Calendar's "Connect with Google" (#407), which Gmail joins in the next PR, together with the connection dialog's Gmail option.

Labels are the source's folders, and watches, routing rules and imports work as they do for IMAP. A watch follows Gmail's history ids, so its first scan of a large mailbox reads nothing old (unless it backfills), and later scans read only what arrived. When Gmail has forgotten the history (about a week), the watch falls back to a date search.

Motivation and Context

Adeel asked for email from Gmail and Google Workspace (OAuth and service account), IMAP or API, as a sensor. Google Calendar (#399) landed the shared google_auth.py first, and Gmail signs in through it. That module gains:

  • A check that a connected account allows what the source reads (the Gmail API needs gmail.readonly, IMAP needs https://mail.google.com/).
  • Plainer refusals: which client id and scopes a Workspace admin must allow, an address outside the domain, and a client Google no longer knows.

The backends listing also says which types sign in to Google (google: true). The Clicks journey's graph-merge wait is ported from #397: it times out now and then on main too.

Steps to reproduce (if appropriate):

  1. Make a Google account's authorized user JSON with gcloud auth application-default login --client-id-file=client.json --scopes=https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/cloud-platform.
  2. Run POST /api/v1/sources with {"type": "gmail", "name": "Mail", "secrets": {"token": "<that JSON>"}}.
  3. Add a watch on INBOX.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (ruff, mypy, prettier, tsc clean).
  • I added or updated tests. tests/api/test_gmail_source.py covers the API path, service account errors, a connected account (including IMAP over XOAUTH2, a scope it lacks, and revoked access) and agent tokens. It runs against tests/fake_gmail.py. The full backend suite passed locally: 1091 passed, 20 skipped.
  • My change requires documentation updates, and I have updated docs/ accordingly: processing.md (Gmail and Google Workspace), sensors.md, activity.md and README.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: This PR is the source, through the API. Connecting the account in the browser and the dialog's Gmail option come in the next PR, on feat(sources): Connect with Google for Google Calendar connections #407's flow; the docs and changelog label it in progress.
  • Agents: Everything goes through the API (/sources, watches, imports). An agent passes a token it already holds.
  • Zero setup: No new dependency (httpx and PyJWT are already in), and no .env setting.
  • Fewest clicks: A service account needs only its key and an address; via defaults to the API. One-click Google consent comes with feat(sources): Connect with Google for Google Calendar connections #407's button in the next PR.
  • Configurable: The source's settings (via, user, secrets) are edited like any other source's. Nothing is added to the setup wizard.
  • Light: Plain HTTP calls, and no new process.
  • Private and secure: Tokens and keys are sealed like passwords. Access tokens live in memory only. Gmail access is read only by default. Gmail is in threat-model.md/.yaml/.json.
  • Cost: Each call is logged in the activity ledger (google.token, gmail.list, gmail.read). Watches read history deltas, not whole mailboxes. No model calls.
  • User decides: The type is new and off until someone adds a source. Existing defaults are unchanged.
  • Upgrades: n/a. No stored-data shape changes: storage_source is schemaless, and the new type only adds rows.
  • Honest: The docs and changelog say connecting from the dialog is in progress.
  • Tests, docs and generated files updated: The OpenAPI schema and client were regenerated with make openapi.
  • Open: OAuth 2.0 refresh tokens, JWT bearer (RFC 7523), IMAP with XOAUTH2, RFC 822 .eml. MIT-compatible.
  • Reported: Refine later:
    • Gmail's batch endpoint instead of a thread pool for headers.
    • A Gmail search (q) filter on watches.
    • Push via Pub/Sub instead of polling.
    • Label changes on already-seen mail in whole-mailbox watches.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q

A new "gmail" source reads a Gmail or Google Workspace mailbox, by the
Gmail API (read only) or by IMAP with XOAUTH2. It signs in as a person
with the Google client from Settings › Sign-in (OAuth, refresh token
sealed on the source), or as a Workspace service account with
domain-wide delegation. Watches follow Gmail's history ids, so a first
scan of a big mailbox reads nothing and later scans read only new mail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q
@adeelahmad adeelahmad self-assigned this Oct 15, 2026
Sources live at /sources; /sensors/{id} is a different thing with its own
ids, so a Gmail source signed in there landed on the wrong page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q
Gmail now signs in through the shared google_auth.py that Google Calendar
brought (authorized-user tokens, one access-token cache), which gains
Sign in with Google for any Google source (Gmail and Google Calendar), a
check that a sign-in allows what the source reads, and plainer refusals.
The return from Google lands on /sources?source={id}.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q
Ported from #397: the journey fails now and then on main too with a 20 s
timeout while a slow runner is still reading import-file's episode.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q
… its own

Calendar's Connect with Google (#407) brings the browser consent for Google
sources; Gmail joins it in the next pull request. This one keeps the source:
a connected account's token, or a Workspace service account, read through
the Gmail API or over IMAP. The wording says to connect the Google account
again, as Calendar's does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzqrXPCMR7aJYyLPqmx41Q

Copy link
Copy Markdown
Owner Author

"Clicks and keystrokes per feature" failed on the earlier head because the graph-merge journey timed out (locator.waitFor: Timeout 20000ms, desktop only). The cause isn't this PR: the same journey is red on main at 3b0bc70. I ported #397's fix (commit 7577603, which waits for the imports' jobs before opening the graph) as 5ebb758. Once #397 merges, the port has no effect.


Generated by Claude Code

…-yergon

# Conflicts:
#	CHANGELOG.md
#	threat-model.yaml
@adeelahmad
adeelahmad merged commit bb75dce into main Oct 15, 2026
11 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-yergon branch October 15, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants