Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
was. Deleted notes wait in Recently deleted, at the bottom of the Notes page, for 30 days with their files and
history, to put back with one click or delete for good; then they go on their own
([Deleting and putting back](docs/notes.md#deleting-and-putting-back)).
- **Connect Gmail with Google.** Add connection › Gmail now has the Google account and Service account choice and
**Connect with Google**, as Google Calendar does: Google asks to read mail (read only; all of Gmail to read over
IMAP), the account's address becomes the connection's, and its labels open to pick one to watch. A service account
takes its key and the address it reads, and says which client ID and scope a Workspace admin allows. A Gmail
connection whose sign-in stopped working has **Connect Google account** on its row
([Gmail and Google Workspace](docs/processing.md#gmail-and-google-workspace)).
- **Connect with Google.** A Google Calendar connection signs in to a Google account from a button now: Google's consent
screen asks for read-only calendar access and sends you back to Sources with the account connected and its calendars
open. It uses the Google sign-in provider's OAuth client, an earlier connection's, or one entered in the form, which
Expand Down
17 changes: 10 additions & 7 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,13 +410,16 @@ A Gmail or Google Workspace mailbox (type `gmail`) is read through Google's Gmai
or over IMAP (`via: imap`). For a personal Gmail account, an email (IMAP) connection with an app password works too,
with nothing to set up at Google. A Gmail source signs in one of two ways:

- **Google account** (`token`): the "authorized user" JSON (`client_id`, `client_secret`, `refresh_token`) of an
OAuth client of your own Google Cloud project, with the Gmail API turned on there, kept encrypted like a password.
`gcloud auth application-default login --client-id-file=client.json --scopes=https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/cloud-platform`
writes one (with `https://mail.google.com/` instead to read over IMAP); an agent can give one it already has. Enter
the account's address as `user` to read over IMAP. Google reviews apps that read Gmail before people outside your
Workspace domain can use them without a warning: an app in testing works for its test users, whose sign-ins Google
ends after seven days. Connecting the account from the connection dialog, with no JSON to make, is in progress.
- **Google account** (`token`): **Connect with Google** in the connection form, as for
[Google Calendar](#email-imap-calendar-feeds-ical-and-google-calendar): Google's consent screen asks to read your mail
(read only; all of Gmail to read over IMAP, since Google has no read-only scope for it), then sends you back to
Sources with the account connected, its address as the source's `user`, and its labels open. Turn on the Gmail API in
the OAuth client's Google Cloud project and add the Gmail scope to its consent screen. Google reviews apps that read
Gmail before people outside your Workspace domain can use them without a warning: while the consent screen is in
Testing, it works for its test users and Google ends their sign-ins after seven days. Unticking the mail permission on
Google's page leaves the source unconnected, and says so. An agent saves the `token` directly instead (in the form:
Or paste an authorized user JSON): `gcloud auth application-default login --client-id-file=client.json --scopes=https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/cloud-platform`
writes one; enter the account's address as `user` to read over IMAP.
- **A service account** (Google Workspace). Upload the service account's JSON key and enter the address of the person
whose mail Lens reads; nobody signs in. A Workspace admin lets the account's client id use
`https://www.googleapis.com/auth/gmail.readonly` (or `https://mail.google.com/` to read over IMAP) under Admin
Expand Down
6 changes: 3 additions & 3 deletions fastapi_backend/app/api/v1/routes/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,9 @@ def google_client(user: AdminReader, request: Request, db: Db, cfg: Cfg) -> Goog
def connect_google(
sid: int, user: AdminWriter, request: Request, response: Response, db: Db, cfg: Cfg, body: GoogleConnect | None = None
) -> GoogleConnectStart:
"""Connect a Google account to a Google source (Google Calendar): open the returned address in this browser, consent
there, and Google sends you back to the Sources page with the account connected. Its refresh token is kept sealed on
the source. Without a client here, the Google sign-in provider's (Settings › Sign-in) or an earlier connection's is
"""Connect a Google account to a Google source (Gmail, Google Calendar): open the returned address in this browser,
consent there, and Google sends you back to the Sources page with the account connected. Its refresh token is kept
sealed on the source. Without a client here, the Google sign-in provider's (Settings › Sign-in) or an earlier connection's is
used. Only from the web app: the consent comes back to the browser that asked (an agent saves an authorized user
JSON it already has as the source's `token` instead). Audited as `source.google_connect`."""
if user.via != "access":
Expand Down
13 changes: 9 additions & 4 deletions fastapi_backend/app/domain/gmail.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,14 @@ def __init__(self, files=(), marks=()):
self.marks = list(marks)


def scopes(src):
"""What a source reads with: Gmail read only, or all of Gmail over IMAP (Google has no read-only scope for it)."""
return [MAIL] if str((src.get("params") or {}).get("via") or "").strip() == "imap" else [GMAIL_READ]
def scopes(params):
"""What a source reads with, by its params: Gmail read only, or all of Gmail over IMAP (Google has no read-only scope
for it)."""
return (MAIL,) if str((params or {}).get("via") or "").strip() == "imap" else (GMAIL_READ,)


SCOPES = scopes # what Connect with Google asks for (google_auth.scopes_for)
ACCOUNT = "user" # the parameter Connect with Google fills in with the account's address


def credentials(cfg, src):
Expand All @@ -86,7 +91,7 @@ def credentials(cfg, src):

def token(cfg, src):
"""An access token for the source's mailbox (IMAP takes it too)."""
return google_auth.access_token(cfg, credentials(cfg, src), scopes(src))
return google_auth.access_token(cfg, credentials(cfg, src), scopes(src.get("params")))


class _Expired(Exception):
Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/domain/google_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -251,9 +251,9 @@ def get(cfg, creds, scopes, url, params=None, api="api", what=None):


def _module(src):
from . import feeds # feeds imports gcal, which imports this
from . import feeds, gmail # they import this

mod = feeds.OWN.get(src["type"])
mod = feeds.OWN.get(src["type"]) or {"gmail": gmail}.get(src["type"])
if mod is None or not getattr(mod, "SCOPES", None):
raise ValueError(f"a {src['type']} source doesn't sign in with a Google account")
return mod
Expand Down
39 changes: 39 additions & 0 deletions fastapi_backend/tests/api/test_gmail_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from __future__ import annotations

import json
import urllib.parse

import pytest
from cryptography.hazmat.primitives import serialization
Expand Down Expand Up @@ -233,6 +234,44 @@ def test_a_connected_google_account(client, db, cfg, admin, google, monkeypatch)
assert not h["ok"] and "Connect the Google account again" in h["error"]


def _connect(client, google, sid, headers, **consent):
"""Connect with Google from the browser: Google's consent page, then its way back to Sources (the query)."""
r = client.post(f"/api/v1/sources/{sid}/google", headers=headers, json={})
assert r.status_code == 200, r.text
url = r.json()["url"]
code, state = google.consent(url, **consent)
back = client.get("/api/v1/auth/external/google/callback", params={"code": code, "state": state}, follow_redirects=False)
assert back.status_code == 303, back.text
u = urllib.parse.urlsplit(back.headers["location"])
assert u.path == "/sources"
return dict(urllib.parse.parse_qsl(urllib.parse.urlsplit(url).query)), dict(urllib.parse.parse_qsl(u.query))


def test_connecting_the_account_in_the_browser(client, db, cfg, admin, google):
sid = client.post("/api/v1/sources", headers=admin, json={"type": "gmail", "name": "My mail"}).json()["id"]
body = {"kind": "google", "client_id": google.client_id, "client_secret": google.client_secret, "enabled": False}
assert client.post("/api/v1/auth/providers", json=body, headers=admin).status_code == 200

# Google asks to read mail, read only; someone unticks the box there: not connected
asked, back = _connect(client, google, sid, admin, untick=[GMAIL_READ])
assert asked["scope"].split() == ["openid", "email", GMAIL_READ] and asked["access_type"] == "offline"
assert "wasn't given access" in back["google_error"] and back["source"] == str(sid)
assert not sources.get(db, sid).get("sealed")

# connected: the account's address is the source's user, its refresh token kept sealed
_, back = _connect(client, google, sid, admin)
assert back == {"google": "connected", "source": str(sid), "account": MARA}
src = client.get("/api/v1/sources", headers=admin).json()[0]
assert src["params"]["user"] == MARA and src["secrets"]["token"]["set"] and src["health"]["ok"]
assert json.loads(google_auth.secret(cfg, sources.get(db, sid), "token"))["client_id"] == google.client_id

# read over IMAP instead: Google is asked for all of Gmail
r = client.patch(f"/api/v1/sources/{sid}", headers=admin, json={"params": {"via": "imap"}})
assert r.json()["health"]["error"].endswith("connect the Google account again")
asked, back = _connect(client, google, sid, admin)
assert MAIL in asked["scope"].split() and back["google"] == "connected"


def test_an_agent_gives_a_token_it_has(client, admin, google):
google.refresh["1//agent"] = {"email": MARA, "scope": GMAIL_READ, "client_id": google.client_id}
token = {"refresh_token": "1//agent", "client_id": google.client_id, "client_secret": google.client_secret}
Expand Down
28 changes: 28 additions & 0 deletions nextjs-frontend/__tests__/add-connection.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ const BACKENDS = {
fields: { calendars: "", impersonate: "", days_back: "365", account: "" },
secrets: ["token", "service_account_credentials"],
},
gmail: {
label: "Gmail or Google Workspace",
fields: { user: "", via: "api" },
secrets: ["token", "service_account_credentials"],
},
};
const CALLBACK = "https://lens.example/api/v1/auth/external/google/callback";
const CONSENT = "https://accounts.google.com/o/oauth2/v2/auth?client_id=c";
Expand Down Expand Up @@ -162,4 +167,27 @@ describe("connecting a Google account", () => {
client_secret: "shh",
});
});

it("connects Gmail the same way, and asks a service account whose mail it reads", async () => {
(Sources.googleClient as jest.Mock).mockImplementation(() =>
ok({ origin: "sign-in", client_id: "c", redirect_uri: CALLBACK }),
);
show(jest.fn());
fireEvent.click(screen.getByRole("radio", { name: /Gmail/ }));
fireEvent.click(screen.getByRole("radio", { name: "Service account" }));
fireEvent.click(screen.getByRole("button", { name: "Test connection" }));
expect(await screen.findByText("Enter the address of the person whose mail Lens reads.")).toBeInTheDocument();
expect(Sources.createSource).not.toHaveBeenCalled();

fireEvent.click(screen.getByRole("radio", { name: "Google account" }));
fireEvent.click(await screen.findByRole("button", { name: "Connect with Google" }));
await waitFor(() => expect(navigate.assign).toHaveBeenCalledWith(CONSENT));
expect((Sources.createSource as jest.Mock).mock.calls[0][0].body).toEqual({
type: "gmail",
name: "Gmail",
params: { user: "", via: "api" },
secrets: {},
});
expect((Sources.connectGoogle as jest.Mock).mock.calls[0][0]).toMatchObject({ path: { sid: 7 }, body: {} });
});
});
46 changes: 45 additions & 1 deletion nextjs-frontend/__tests__/sources-model.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {
confirms,
emptyForm,
fileKind,
gmailScope,
healthOf,
imapHost,
nextScan,
Expand Down Expand Up @@ -154,9 +155,14 @@ describe("connections", () => {
const key = JSON.stringify({
type: "service_account",
client_email: "lens@p.iam.gserviceaccount.com",
client_id: "1234567890",
private_key: "K",
});
expect(parseServiceAccountKey(key)).toEqual({ ok: true, email: "lens@p.iam.gserviceaccount.com" });
expect(parseServiceAccountKey(key)).toEqual({
ok: true,
email: "lens@p.iam.gserviceaccount.com",
clientId: "1234567890",
});
expect(parseServiceAccountKey("nope").ok).toBe(false);
expect(parseServiceAccountKey('{"access_token":"a"}').ok).toBe(false);

Expand Down Expand Up @@ -269,6 +275,44 @@ describe("connections", () => {
);
expect(suggestName("gcal", blank.params)).toBe("Google Calendar");
});
it("signs Gmail in as a Google account or as a service account acting for one person", () => {
const GMAIL: BackendSpec = {
label: "Gmail or Google Workspace",
fields: { user: "", via: "api" },
secrets: ["token", "service_account_credentials"],
};
const blank = emptyForm(GMAIL);
expect(Object.keys(validateForm("gmail", GMAIL, blank, false))).toEqual(["token"]); // Connect with Google
expect(validateForm("gmail", GMAIL, blank, true, (k) => k === "token")).toEqual({});
const account = '{"client_id":"c","client_secret":"s","refresh_token":"r"}';
const pasted = { ...blank, params: { user: "mara@x.io", via: "imap" }, secrets: { token: account } };
expect(validateForm("gmail", GMAIL, pasted, false)).toEqual({});
expect(buildPayload(GMAIL, pasted, "gmail")).toEqual({
params: { user: "mara@x.io", via: "imap" },
secrets: { token: account, service_account_credentials: null },
});
const key = JSON.stringify({
type: "service_account",
client_email: "lens@p.iam.gserviceaccount.com",
private_key: "K",
});
const sa = { ...blank, driveAuth: "service_account" as const, secrets: { service_account_credentials: key } };
expect(Object.keys(validateForm("gmail", GMAIL, sa, false))).toEqual(["user"]);
const acting = { ...sa, params: { ...sa.params, user: "mara@x.io" } };
expect(validateForm("gmail", GMAIL, acting, false)).toEqual({});
expect(buildPayload(GMAIL, acting, "gmail")).toEqual({
params: { user: "mara@x.io", via: "api" },
secrets: { service_account_credentials: key, token: null },
});
expect(gmailScope("imap")).toBe("https://mail.google.com/");
expect(sourceSubtitle({ type: "gmail", params: { user: "mara@x.io", via: "api" } })).toBe(
"Gmail · mara@x.io · API",
);
expect([suggestName("gmail", blank.params), suggestName("gmail", acting.params)]).toEqual([
"Gmail",
"Gmail · mara@x.io",
]);
});
it("confirms deletes by name, loosely", () => {
expect(confirms(" dropbox · callrecorder", "Dropbox · CallRecorder")).toBe(true);
expect(confirms("Dropbox", "Dropbox · CallRecorder")).toBe(false);
Expand Down
6 changes: 3 additions & 3 deletions nextjs-frontend/app/openapi-client/sdk.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8323,9 +8323,9 @@ export class Sources {
/**
* Connect Google
*
* Connect a Google account to a Google source (Google Calendar): open the returned address in this browser, consent
* there, and Google sends you back to the Sources page with the account connected. Its refresh token is kept sealed on
* the source. Without a client here, the Google sign-in provider's (Settings › Sign-in) or an earlier connection's is
* Connect a Google account to a Google source (Gmail, Google Calendar): open the returned address in this browser,
* consent there, and Google sends you back to the Sources page with the account connected. Its refresh token is kept
* sealed on the source. Without a client here, the Google sign-in provider's (Settings › Sign-in) or an earlier connection's is
* used. Only from the web app: the consent comes back to the browser that asked (an agent saves an authorized user
* JSON it already has as the source's `token` instead). Audited as `source.google_connect`.
*/
Expand Down
Loading
Loading