Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
was. Deleted notes wait in Recently deleted, at the bottom of the Notes page, for 30 days with their files and
history, to put back with one click or delete for good; then they go on their own
([Deleting and putting back](docs/notes.md#deleting-and-putting-back)).
- **Connect with Google.** A Google Calendar connection signs in to a Google account from a button now: Google's consent
screen asks for read-only calendar access and sends you back to Sources with the account connected and its calendars
open. It uses the Google sign-in provider's OAuth client, an earlier connection's, or one entered in the form, which
also shows the redirect URI to register. A connection whose sign-in stopped working has **Connect Google account**
on its row; pasting the authorized user JSON still works
([Google Calendar](docs/processing.md#email-imap-calendar-feeds-ical-and-google-calendar)).
- **Every entity has its own page.** An entity's page is named by its name, as a wiki names pages:
`/wiki/<namespace>/<Name>` (spaces as underscores), so Ada Lovelace in `work` is `/wiki/work/Ada_Lovelace`. It shows
what the entity is, the notes people and the assistant keep on it in the BlockSuite editor, every line that says it,
Expand Down
4 changes: 3 additions & 1 deletion docs/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,9 @@ OpenID Connect provider (Authentik, Keycloak, Okta, ...; give its issuer address
`/.well-known/openid-configuration` there). Each needs a client id and secret from the provider, and the redirect URI
the dialog shows: `<the web app's address>/api/v1/auth/external/<name>/callback`, registered for every address people
open Lens at. The secret is kept encrypted (like other settings secrets) and never shown again. The sign-in page then
has a "Continue with ..." button for each provider that's turned on.
has a "Continue with ..." button for each provider that's turned on. Google's client also connects Google accounts to
Google Calendar sources (Connect with Google, [Google Calendar](processing.md#email-imap-calendar-feeds-ical-and-google-calendar)),
through the same redirect URI.

* **Who it signs in.** An outside account that's connected signs in its Lens account. One that isn't yet is connected
to the account with the same email, when the provider vouches for the email: Google and OpenID Connect providers say
Expand Down
16 changes: 14 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,8 +374,20 @@ importing chosen ones and watching work as above.
of its own, as Google keeps it; working-location entries ("Home", "Office") are left out. It signs in one of two ways,
each kept encrypted like a password and never shown again:

- **Google account** (`token`): the "authorized user" JSON (`client_id`, `client_secret`, `refresh_token`) of an
OAuth client of your own Google Cloud project, with the Calendar API turned on there. `gcloud auth
- **Google account** (`token`): **Connect with Google** in the connection form saves the connection and opens
Google's consent screen in this browser. Once you allow read-only calendar access, Google sends you back to
Sources with the account connected (its address shows on the connection) and its calendars open, to pick one to
watch. It goes through an OAuth client of a Google Cloud project with the Calendar API turned on: the Google
sign-in provider's (Settings › Sign-in) when there is one, else an earlier Google connection's, else one you
enter in the form: a web application client whose authorized redirect URIs include
`<your Lens address>/api/v1/auth/external/google/callback` (the address Google sign-in uses too; the form shows
it). While the client's OAuth consent screen is in Testing, Google ends its refresh tokens after 7 days and warns
that the app isn't verified; publish it to keep the connection. A connection whose sign-in stopped working shows
**Connect Google account** on its row. Connecting happens in the web app (`POST /api/v1/sources/{id}/google`
returns the consent address, and refuses API tokens), because the consent itself is a person's and comes back to
the browser that asked. What is kept is the "authorized user" JSON (`client_id`, `client_secret`,
`refresh_token`), which an agent or a person can also save directly as the `token` (Or paste an authorized user
JSON): `gcloud auth
application-default login --client-id-file=client.json --scopes=https://www.googleapis.com/auth/calendar.readonly,https://www.googleapis.com/auth/cloud-platform`
writes one (to `application_default_credentials.json` in gcloud's folder).
- **Service account** (`service_account_credentials`, its .json key file). On its own it reads the calendars shared
Expand Down
30 changes: 29 additions & 1 deletion fastapi_backend/app/api/v1/routes/external.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from fastapi.responses import RedirectResponse

from app.api.deps import AdminWriter, Cfg, CurrentUser, Db, Writer, domain_errors, web_origin
from app.domain import auth, external_login, passkeys
from app.domain import auth, external_login, google_auth, passkeys, sources
from app.schemas.auth import (
ExternalIdentity,
ExternalProvider,
Expand Down Expand Up @@ -96,6 +96,8 @@ def external_callback(
flow = external_login.claim(db, state, request.cookies.get(COOKIE))
except ValueError as e:
return _back(web_origin(request), error=str(e), next="/login")
if flow.get("mode") == "source":
return _source_connected(db, cfg, flow, code, state, error, error_description)
origin, connect = flow["origin"], flow.get("mode") == "connect"
away = "/account" if connect else "/login"
if error or not code:
Expand All @@ -114,6 +116,32 @@ def external_callback(
return _back(origin, ticket=passkeys.issue_ticket(db, u["id"], f"external:{key}"), next=flow.get("next") or "/")


def _source_connected(db: Db, cfg: Cfg, flow: dict, code: str, state: str, error: str, error_description: str) -> RedirectResponse:
"""A Google account connected to a source (google_auth.connect_start): ends on the Sources page, which shows how it
went and offers the source's calendars to watch."""
sid = flow.get("source")
if error or not code:
why = "you cancelled" if error == "access_denied" else (error_description or error or "no code came back")
q = {"google_error": f"connecting the Google account didn't finish: {why}", "source": sid}
elif not (u := auth.active_account(db, flow.get("account"))) or not u.get("admin"):
q = {"google_error": "only admins can connect a Google account", "source": sid}
else:
try:
sid, email = google_auth.connect_finish(db, cfg, flow, code, state)
sources.test(db, cfg, sid)
auth.audit(db, u, "source.google_connect", f"storage_source:{sid}", [email] if email else None)
q = {"google": "connected", "source": sid, "account": email}
except KeyError:
q = {"google_error": "that connection was removed meanwhile"}
except (ValueError, RuntimeError) as e:
q = {"google_error": str(e), "source": sid}
r = RedirectResponse(
f"{flow['origin']}/sources?{urllib.parse.urlencode({k: v for k, v in q.items() if v not in (None, '')})}", status_code=303
)
r.delete_cookie(COOKIE, path="/api/v1/auth/external")
return r


# ---------- your outside accounts ----------
def _signed_in(user: CurrentUser) -> None:
if user.via != "access":
Expand Down
49 changes: 46 additions & 3 deletions fastapi_backend/app/api/v1/routes/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,20 @@
import logging
from collections import Counter

from fastapi import APIRouter, BackgroundTasks, HTTPException, Request
from fastapi import APIRouter, BackgroundTasks, HTTPException, Request, Response

from app.api.deps import Acl, AdminReader, AdminWriter, Cfg, CurrentUser, Db
from app.domain import auth, rclone_signin, sources, store
from app.api.deps import Acl, AdminReader, AdminWriter, Cfg, CurrentUser, Db, web_origin
from app.api.v1.routes.external import _cookie
from app.domain import auth, google_auth, rclone_signin, sources, store
from app.domain.store import DB
from app.schemas.admin import Started
from app.schemas.common import Created, Ok
from app.schemas.sources import (
Backend,
BrowseEntry,
GoogleClient,
GoogleConnect,
GoogleConnectStart,
RcloneSignIn,
RcloneSignInFinish,
RcloneSignInStart,
Expand Down Expand Up @@ -145,6 +149,45 @@ def test_source(sid: int, user: AdminWriter, db: Db, cfg: Cfg) -> SourceHealth:
return sources.test(db, cfg, sid)


@router.get("/sources/google-client")
def google_client(user: AdminReader, request: Request, db: Db, cfg: Cfg) -> GoogleClient:
"""The OAuth client a Google account would be connected through (never its secret), and the redirect URI it needs."""
try:
c = google_auth.client(db, cfg)
except ValueError:
c = {"origin": None, "client_id": None}
return GoogleClient(origin=c["origin"], client_id=c["client_id"], redirect_uri=web_origin(request) + google_auth.CALLBACK)


@router.post("/sources/{sid}/google")
def connect_google(
sid: int, user: AdminWriter, request: Request, response: Response, db: Db, cfg: Cfg, body: GoogleConnect | None = None
) -> GoogleConnectStart:
"""Connect a Google account to a Google source (Google Calendar): open the returned address in this browser, consent
there, and Google sends you back to the Sources page with the account connected. Its refresh token is kept sealed on
the source. Without a client here, the Google sign-in provider's (Settings › Sign-in) or an earlier connection's is
used. Only from the web app: the consent comes back to the browser that asked (an agent saves an authorized user
JSON it already has as the source's `token` instead). Audited as `source.google_connect`."""
if user.via != "access":
raise HTTPException(
403,
"connect Google from the web app: the consent comes back to the browser that asked, so API tokens and apps "
"can't (save an authorized user JSON as the source's token instead)",
)
body = body or GoogleConnect()
try:
src = sources.get(db, sid)
except KeyError:
raise HTTPException(404, "not found") from None
origin = web_origin(request)
try:
url, browser = google_auth.connect_start(db, cfg, src, origin, (body.client_id, body.client_secret), user.id)
except ValueError as e:
raise HTTPException(400, str(e)) from None
_cookie(response, origin, browser)
return GoogleConnectStart(url=url)


@router.get("/sources/{sid}/browse")
def browse_source(sid: int, user: AdminReader, db: Db, cfg: Cfg, path: str = "") -> list[BrowseEntry]:
"""The folders and files at one path of a source (for local sources, no path lists the allowed roots), with the
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/gcal.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
SCOPES = ("https://www.googleapis.com/auth/calendar.readonly",)
TYPE = {
"label": "Google Calendar",
"fields": {"calendars": "", "impersonate": "", "days_back": "365"},
"fields": {"calendars": "", "impersonate": "", "days_back": "365", "account": ""}, # account: the one connected
"secrets": ["token", "service_account_credentials"],
"google": True, # signs in through google_auth.py
}
Expand Down
148 changes: 148 additions & 0 deletions fastapi_backend/app/domain/google_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,13 @@

from __future__ import annotations

import base64
import datetime as dt
import hashlib
import json
import secrets
import time
import urllib.parse

import httpx
import jwt
Expand Down Expand Up @@ -237,3 +241,147 @@ def get(cfg, creds, scopes, url, params=None, api="api", what=None):
continue
return _answer(r, creds)
raise GoogleError("Google didn't accept a fresh sign-in") # not reached: the second attempt returns or raises


# ---------- connecting a Google account to a source (OAuth consent in the browser) ----------
AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth"
IDENTITY = ("openid", "email") # asked for too, to show which account is connected
FLOW_MINUTES = 10
CALLBACK = "/api/v1/auth/external/google/callback" # the Google sign-in provider's own: one redirect URI for both


def _module(src):
from . import feeds # feeds imports gcal, which imports this

mod = feeds.OWN.get(src["type"])
if mod is None or not getattr(mod, "SCOPES", None):
raise ValueError(f"a {src['type']} source doesn't sign in with a Google account")
return mod


def scopes_for(src):
"""The scopes a source's type reads with: its module's SCOPES, or SCOPES(params) when they depend on its settings."""
s = _module(src).SCOPES
return tuple(s(src.get("params") or {}) if callable(s) else s)


def client(db, cfg, client_id=None, client_secret=None):
"""The OAuth client a Google account is connected through: {client_id, client_secret, origin}. The one given, else
the Google sign-in provider's (Settings › Sign-in), else the one a Google source was connected through last.
ValueError when there's none."""
if client_id or client_secret:
if not (client_id and client_secret):
raise ValueError("enter both the client ID and the client secret of the Google OAuth client")
return {"client_id": str(client_id).strip(), "client_secret": str(client_secret).strip(), "origin": "given"}
from . import external_login, settings, sources, store

row = db.one("SELECT client_id, client_secret FROM $r", r=store.R("login_provider", "google"))
if row and row.get("client_id") and row.get("client_secret"):
secret_ = settings.unseal(cfg, row["client_secret"], external_login._context("google"))
return {"client_id": row["client_id"], "client_secret": secret_, "origin": "sign-in"}
google_types = [k for k, v in sources.BACKENDS.items() if v.get("google")]
rows = db.rows(
"SELECT record::id(id) AS id, created_at FROM storage_source WHERE type IN $t AND sealed.token != NONE ORDER BY created_at DESC",
t=google_types,
)
for sid in [r["id"] for r in rows]:
user = json.loads(secret(cfg, sources.get(db, sid), "token") or "{}")
if user.get("client_id") and user.get("client_secret"):
return {"client_id": user["client_id"], "client_secret": user["client_secret"], "origin": "connection"}
raise ValueError(
"Lens needs a Google OAuth client to connect an account: add Google in Settings › Sign-in, or enter a client "
"ID and secret (Google Cloud › APIs & Services › Credentials, a web application client)"
)


def connect_start(db, cfg, src, origin, given=None, account=None):
"""Google's consent page for connecting an account to source `src`, and the browser secret to keep in a cookie
(the sign-in cookie: the flow is an external_flow, used once, ten minutes, bound to this browser)."""
from . import auth, settings, store

scopes = scopes_for(src)
c = client(db, cfg, *(given or (None, None)))
state, verifier, browser = secrets.token_urlsafe(24), secrets.token_urlsafe(48), secrets.token_urlsafe(24)
challenge = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).decode().rstrip("=")
redirect_uri = origin.rstrip("/") + CALLBACK
expires = (dt.datetime.now(dt.timezone.utc) + dt.timedelta(minutes=FLOW_MINUTES)).isoformat(timespec="seconds")
db.q("DELETE external_flow WHERE expires_at < $n", n=store.now())
db.q(
"CREATE $r CONTENT $d",
r=store.R("external_flow", auth.sha(state)),
d={
"provider": "google",
"mode": "source",
"source": src["id"],
"scopes": list(scopes),
"client_id": c["client_id"],
"client_secret": settings.seal(cfg, c["client_secret"], f"external_flow:{auth.sha(state)}"),
"verifier": verifier,
"browser": auth.sha(browser),
"redirect_uri": redirect_uri,
"origin": origin.rstrip("/"),
"account": account, # the admin who started it: still one when Google sends them back
"expires_at": expires,
},
)
q = {
"response_type": "code",
"client_id": c["client_id"],
"redirect_uri": redirect_uri,
"scope": " ".join([*IDENTITY, *scopes]),
"state": state,
"code_challenge": challenge,
"code_challenge_method": "S256",
"access_type": "offline", # a refresh token, so the source reads on its own
"prompt": "consent", # Google gives a refresh token only when it asks
"include_granted_scopes": "false",
}
return AUTH_URL + "?" + urllib.parse.urlencode(q), browser


def connect_finish(db, cfg, flow, code, state):
"""Swap the code Google sent back for the account's refresh token and keep it on the source (its `token`; a
service account key it had is dropped). Returns the source's id and the account's address."""
from . import auth, settings, sources

src = sources.get(db, flow["source"]) # KeyError: removed meanwhile
client_secret = settings.unseal(cfg, flow["client_secret"], f"external_flow:{auth.sha(state)}")
form = {
"grant_type": "authorization_code",
"code": code,
"redirect_uri": flow["redirect_uri"],
"client_id": flow["client_id"],
"client_secret": client_secret,
"code_verifier": flow["verifier"],
}
with activity.call("google.token", cfg, detail={"as": "consent"}):
try:
r = httpx.post(TOKEN_URL, data=form, headers={"Accept": "application/json"}, timeout=TIMEOUT)
except httpx.HTTPError as e:
raise GoogleError(f"couldn't reach Google: {e}") from None
try:
body = r.json()
except ValueError:
body = {}
if r.status_code >= 400 or not isinstance(body, dict):
said = (body.get("error_description") or body.get("error")) if isinstance(body, dict) else None
raise GoogleError(f"Google refused: {said or r.status_code}; check the client ID and secret")
if not body.get("refresh_token"):
raise GoogleError("Google gave no refresh token; remove Lens from your Google account's third-party access and connect again")
granted = set(str(body.get("scope") or "").split())
missing = [s for s in flow.get("scopes") or [] if granted and s not in granted]
if missing:
raise GoogleError("Google wasn't given access to what this connection reads: tick it on Google's consent screen")
email = ""
if body.get("id_token"):
try: # straight from Google's token endpoint over TLS: no signature to check (OpenID Connect §3.1.3.7)
email = str(jwt.decode(body["id_token"], options={"verify_signature": False}).get("email") or "")
except jwt.PyJWTError:
email = ""
user = json.dumps(
{"type": "authorized_user", "client_id": flow["client_id"], "client_secret": client_secret, "refresh_token": body["refresh_token"]}
)
field = getattr(_module(src), "ACCOUNT", "account") # the parameter that names the connected account
params = {field: email} if email and field in sources.BACKENDS[src["type"]]["fields"] else None
sources.update(db, cfg, src["id"], params=params, secret_values={"token": user, "service_account_credentials": None})
return src["id"], email
18 changes: 18 additions & 0 deletions fastapi_backend/app/schemas/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,24 @@ class SourceUpdated(Ok):
health: SourceHealth


class GoogleClient(ResponseModel):
origin: Literal["sign-in", "connection"] | None = Field(
description="where the OAuth client for connecting a Google account comes from: the Google sign-in provider, "
"an earlier Google connection, or none (enter one to connect)"
)
client_id: str | None = None
redirect_uri: str = Field(description="the address to add to the OAuth client's authorized redirect URIs")


class GoogleConnect(RequestModel):
client_id: str | None = Field(default=None, max_length=300, description="an OAuth client of your own (else the one Lens has)")
client_secret: str | None = Field(default=None, max_length=300)


class GoogleConnectStart(ResponseModel):
url: str = Field(description="Google's consent page; open it in this browser")


class ImportedAs(ResponseModel):
recording: int
namespace: str
Expand Down
Loading
Loading