Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
also shows the redirect URI to register. A connection whose sign-in stopped working has **Connect Google account**
on its row; pasting the authorized user JSON still works
([Google Calendar](docs/processing.md#email-imap-calendar-feeds-ical-and-google-calendar)).
- **Google Drive connects with Google too.** When Lens has a Google client (Google sign-in's, or an earlier
connection's), a Drive connection signs in with **Connect with Google** in one click and shows the account it is
connected as; Read only asks Google for read-only Drive access. With no client it signs in through rclone, as before,
or through a Google app of your own ([Google Drive](docs/processing.md#storage-sources)).
- **Every entity has its own page.** An entity's page is named by its name, as a wiki names pages:
`/wiki/<namespace>/<Name>` (spaces as underscores), so Ada Lovelace in `work` is `/wiki/work/Ada_Lovelace`. It shows
what the entity is, the notes people and the assistant keep on it in the BlockSuite editor, every line that says it,
Expand Down
10 changes: 10 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,16 @@ files, then Import (`POST /api/v1/import/source`). The listing marks files that
they end on to `POST /api/v1/sources/rclone-signin/{id}` (`{"address": …}`), which answers `{connected: <id>}`
(400: not this sign-in's address, still waiting; 410: cancelled, refused or ran out). `DELETE` stops it. Audited as
`source.signin`.
- **Google Drive can connect with Google** as Google Calendar does: when Lens has a Google OAuth client (the Google
sign-in provider's, or an earlier Google connection's), a new Google account connection uses it, so **Connect with
Google** saves the connection, opens Google's consent screen in this browser and comes back with the account
connected (its address shows on the connection). Read only asks Google for `drive.readonly`, Full access for all of
Drive. With no client Lens signs in through rclone; Connect through your own Google app instead takes a web
application client of a Google Cloud project with the Drive API turned on (the form shows the redirect URI to
register). The connection keeps that client with its token (`client_id`, and the secret encrypted) so rclone renews
the token with it; a token from rclone (signing in through rclone, or pasted) drops it. A connection whose sign-in
stopped working shows **Connect Google account** on its row. As for Google Calendar, connecting happens in the web
app (`POST /api/v1/sources/{id}/google`).
- Google Drive can also sign in as a service account: choose Service account and upload its JSON key file
(`service_account_credentials`). Share the folders Lens should read with the account's address, or give a shared
drive ID (`team_drive`); in Google Workspace, `impersonate` reads a user's own Drive through domain-wide delegation.
Expand Down
37 changes: 37 additions & 0 deletions fastapi_backend/app/domain/gdrive.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
"""Google Drive connected as a Google account through a Google OAuth client of Lens's (Connect with Google,
google_auth.connect_start and connect_finish), for rclone, which reads Drive: what to ask Google for, and how the
account is kept so rclone can renew it.

rclone keeps a Google account as its own token JSON ({access_token, token_type, refresh_token, expiry}) and renews it
with the client it was given (`client_id`, `client_secret` in its config), so the source keeps those three: the token
and the client's secret sealed, the client's id as a parameter. A token pasted from `rclone authorize`, or one made by
signing in through rclone on the server (rclone_signin.py), renews with rclone's own client instead, so it comes
without one (sources.update drops a client left from before).
"""

from __future__ import annotations

import datetime as dt
import json

READONLY = "https://www.googleapis.com/auth/drive.readonly"
FULL = "https://www.googleapis.com/auth/drive"
ACCOUNT = "account" # the parameter that names the connected account


def SCOPES(params): # noqa: N802 (google_auth reads a module's SCOPES)
"""What Google is asked for: the connection's own scope (read only unless it was set to full)."""
return (FULL,) if str(params.get("scope") or "").strip() == "drive" else (READONLY,)


def keep(client_id, client_secret, body, email):
"""The connected account as the source keeps it, from what Google's token endpoint answered: (params, secrets)."""
lasts = int(body.get("expires_in") or 0)
expiry = dt.datetime.now(dt.timezone.utc) + dt.timedelta(seconds=max(lasts - 60, 0))
token = {
"access_token": body.get("access_token") or "",
"token_type": body.get("token_type") or "Bearer",
"refresh_token": body["refresh_token"],
"expiry": expiry.isoformat(timespec="seconds"),
}
return {"client_id": client_id, ACCOUNT: email}, {"token": json.dumps(token), "client_secret": client_secret}
25 changes: 17 additions & 8 deletions fastapi_backend/app/domain/google_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@
"https://www.googleapis.com/auth/gmail.readonly": "View your email messages and settings",
"https://mail.google.com/": "Read, compose, send and permanently delete all your email from Gmail",
"https://www.googleapis.com/auth/calendar.readonly": "See and download any calendar you can access using your Google Calendar",
"https://www.googleapis.com/auth/drive.readonly": "See and download all your Google Drive files",
"https://www.googleapis.com/auth/drive": "See, edit, create, and delete all of your Google Drive files",
}
TIMEOUT = 30
_TOKENS: dict[str, tuple[float, str]] = {} # who and for what -> (when it ends, monotonic; the access token)
Expand Down Expand Up @@ -251,9 +253,9 @@ def get(cfg, creds, scopes, url, params=None, api="api", what=None):


def _module(src):
from . import feeds, gmail # they import this
from . import feeds, gdrive, gmail # they import this

mod = feeds.OWN.get(src["type"]) or {"gmail": gmail}.get(src["type"])
mod = feeds.OWN.get(src["type"]) or {"gmail": gmail, "drive": gdrive}.get(src["type"])
if mod is None or not getattr(mod, "SCOPES", None):
raise ValueError(f"a {src['type']} source doesn't sign in with a Google account")
return mod
Expand Down Expand Up @@ -378,10 +380,17 @@ def connect_finish(db, cfg, flow, code, state):
email = str(jwt.decode(body["id_token"], options={"verify_signature": False}).get("email") or "")
except jwt.PyJWTError:
email = ""
user = json.dumps(
{"type": "authorized_user", "client_id": flow["client_id"], "client_secret": client_secret, "refresh_token": body["refresh_token"]}
)
field = getattr(_module(src), "ACCOUNT", "account") # the parameter that names the connected account
params = {field: email} if email and field in sources.BACKENDS[src["type"]]["fields"] else None
sources.update(db, cfg, src["id"], params=params, secret_values={"token": user, "service_account_credentials": None})
mod = _module(src)
if hasattr(mod, "keep"): # kept another way (Drive: as rclone keeps it)
params, kept = mod.keep(flow["client_id"], client_secret, body, email)
else:
user = {
"type": "authorized_user",
"client_id": flow["client_id"],
"client_secret": client_secret,
"refresh_token": body["refresh_token"],
}
field = getattr(mod, "ACCOUNT", "account") # the parameter that names the connected account
params, kept = ({field: email} if email and field in sources.BACKENDS[src["type"]]["fields"] else None), {"token": json.dumps(user)}
sources.update(db, cfg, src["id"], params=params, secret_values={**kept, "service_account_credentials": None})
return src["id"], email
15 changes: 12 additions & 3 deletions fastapi_backend/app/domain/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,10 @@
"dropbox": {"label": "Dropbox", "fields": {}, "secrets": ["token"], "oauth": "rclone authorize dropbox"},
"drive": {
"label": "Google Drive",
"fields": {"scope": "drive.readonly", "root_folder_id": "", "team_drive": "", "impersonate": ""},
"secrets": ["token", "service_account_credentials"], # signs in with a token, or as a service account
# client_id and account: the Google OAuth client and account connected with Connect with Google (gdrive.py)
"fields": {"scope": "drive.readonly", "root_folder_id": "", "team_drive": "", "impersonate": "", "client_id": "", "account": ""},
# signs in with a token (renewed with client_secret when connected with Google), or as a service account
"secrets": ["token", "service_account_credentials", "client_secret"],
"oauth": "rclone authorize drive",
},
"onedrive": {
Expand All @@ -51,6 +53,7 @@
**feeds.TYPES,
}
OBSCURED = {"pass"} # rclone wants these obscured in its config file
NOT_RCLONE = {"account"} # parameters Lens keeps for itself, not rclone options
SKIPPED = "not audio, video, a document, an image or text"
TRANSCRIPT_EXT = {".txt", ".text", ".md", ".markdown", ".mdx", ".docx", ".doc", ".pdf", ".srt", ".vtt", ".json", ".jsonl", ".eml", ".ics"}
# what a watched folder picks up (its `kinds`): the first three are from before documents, and read PDFs as transcripts
Expand Down Expand Up @@ -124,7 +127,7 @@ def _config(cfg, src):
name, spec = f"src{src['id']}", BACKENDS[src["type"]]
lines = [f"[{name}]", f"type = {src['type']}"]
for k, v in (src.get("params") or {}).items():
if k in spec["fields"] and v not in (None, ""):
if k in spec["fields"] and k not in NOT_RCLONE and v not in (None, ""):
lines.append(f"{k} = {_one_line(v)}")
for k, sealed in (src.get("sealed") or {}).items():
if k in spec["secrets"]:
Expand Down Expand Up @@ -403,6 +406,12 @@ def update(db, cfg, sid, name=None, params=None, secret_values=None):
src = get(db, sid)
_check_params(src["type"], params, secret_values)
params = tidy(src["type"], params)
given = secret_values or {}
if "client_secret" in BACKENDS[src["type"]]["secrets"] and given.get("token") and not given.get("client_secret"):
# a token on its own is rclone's (pasted, or signed in through rclone): it renews with rclone's own client, so
# the client and account a Google connection left are dropped
secret_values = {**secret_values, "client_secret": None}
params = {**(params or {}), "client_id": "", "account": ""}
sealed = dict(src.get("sealed") or {})
for k, v in (secret_values or {}).items():
if v is None or v == "":
Expand Down
106 changes: 106 additions & 0 deletions fastapi_backend/tests/api/test_drive_google.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
"""Google Drive connected with Connect with Google (google_auth.connect_start and connect_finish, gdrive.py): the
account is kept as rclone keeps it, with the client that renews it, against tests/fake_google.py, never Google."""

from __future__ import annotations

import json
import urllib.parse

import pytest

from app.domain import gdrive, google_auth, sources
from tests import fake_google
from tests.helpers import login, make_user

MARA = "mara@harbour.example"


@pytest.fixture
def google(monkeypatch):
g = fake_google.Google()
monkeypatch.setattr(google_auth, "TOKEN_URL", g.base + "/token")
monkeypatch.setattr(google_auth, "AUTH_URL", g.base + "/auth")
monkeypatch.setattr(sources, "test", lambda *a, **k: {"ok": True}) # rclone isn't run in tests
g.clients.add(("lens-web", "web secret 1"))
yield g
g.close()


@pytest.fixture
def admin(client, db):
make_user(db, "root@x.io", "root password 1", admin=True)
h = login(client, "root@x.io", "root password 1")
r = client.post("/api/v1/auth/providers", headers=h, json={"kind": "google", "client_id": "lens-web", "client_secret": "web secret 1"})
assert r.status_code == 200, r.text
return h


def _connect(client, google, h, sid):
url = client.post(f"/api/v1/sources/{sid}/google", headers=h, json={}).json()["url"]
r = client.get(google.consent(url, MARA), follow_redirects=False)
assert r.status_code == 303, r.text
u = urllib.parse.urlsplit(r.headers["location"])
return url, u.path, dict(urllib.parse.parse_qsl(u.query))


def _conf(cfg, db, sid):
_, conf = sources._config(cfg, sources.get(db, sid))
return dict(line.split(" = ", 1) for line in conf.splitlines()[1:])


def test_drive_connects_with_google_for_rclone(google, client, db, cfg, admin):
h = admin
body = {"type": "drive", "name": "Mara's Drive", "params": {"root_folder_id": "https://drive.google.com/drive/folders/1AbC"}}
sid = client.post("/api/v1/sources", headers=h, json=body).json()["id"]
url, path, back = _connect(client, google, h, sid)
q = dict(urllib.parse.parse_qsl(urllib.parse.urlsplit(url).query))
assert q["client_id"] == "lens-web" and q["scope"].split() == ["openid", "email", gdrive.READONLY]
assert path == "/sources" and back == {"google": "connected", "source": str(sid), "account": MARA}

src = next(s for s in client.get("/api/v1/sources", headers=h).json() if s["id"] == sid)
assert src["params"]["account"] == MARA and src["params"]["client_id"] == "lens-web"
assert src["secrets"]["token"]["set"] and src["secrets"]["client_secret"]["set"]
assert "web secret 1" not in json.dumps(db.rows("SELECT * FROM storage_source")) # sealed
# rclone gets its own token shape and the client to renew it with; the account is Lens's, not an rclone option
conf = _conf(cfg, db, sid)
tok = json.loads(conf["token"])
assert tok["refresh_token"] in google.users and tok["token_type"] == "Bearer" and tok["access_token"] and tok["expiry"]
assert conf["client_id"] == "lens-web" and conf["client_secret"] == "web secret 1"
assert conf["root_folder_id"] == "1AbC" and "account" not in conf

# full access asks Google for all of Drive
client.patch(f"/api/v1/sources/{sid}", headers=h, json={"params": {"scope": "drive"}})
url, _, back = _connect(client, google, h, sid)
assert dict(urllib.parse.parse_qsl(urllib.parse.urlsplit(url).query))["scope"].split()[-1] == gdrive.FULL
assert back["google"] == "connected"


def test_a_token_from_rclone_renews_with_rclones_own_client(google, client, db, cfg, admin):
"""Pasting a token (or signing in through rclone) after connecting with Google: the token is rclone's, so the
Google client and account it was connected with are dropped, or rclone would renew it with the wrong client."""
h = admin
key = {"type": "service_account", "private_key": "k", "client_email": "lens@x.iam.gserviceaccount.com"}
body = {"type": "drive", "secrets": {"service_account_credentials": json.dumps(key)}}
sid = client.post("/api/v1/sources", headers=h, json=body).json()["id"]
_connect(client, google, h, sid)
src = sources.get(db, sid)
assert set(src["sealed"]) == {"token", "client_secret"} # the service account gave way to the connected account

pasted = json.dumps({"access_token": "ya29.x", "token_type": "Bearer", "refresh_token": "1//rclone", "expiry": "2030-01-01T00:00:00Z"})
r = client.patch(f"/api/v1/sources/{sid}", headers=h, json={"secrets": {"token": pasted}})
assert r.status_code == 200, r.text
src = next(s for s in client.get("/api/v1/sources", headers=h).json() if s["id"] == sid)
assert src["params"]["client_id"] == "" and src["params"]["account"] == ""
assert not src["secrets"]["client_secret"]["set"]
conf = _conf(cfg, db, sid)
assert json.loads(conf["token"])["refresh_token"] == "1//rclone" and "client_id" not in conf and "client_secret" not in conf
# options alone keep the connection as it is
_connect(client, google, h, sid)
client.patch(f"/api/v1/sources/{sid}", headers=h, json={"params": {"root_folder_id": "1Other"}})
assert _conf(cfg, db, sid)["client_id"] == "lens-web"


def test_drive_scopes():
assert google_auth.scopes_for({"type": "drive", "params": {"scope": "drive.readonly"}}) == (gdrive.READONLY,)
assert google_auth.scopes_for({"type": "drive", "params": {"scope": "drive"}}) == (gdrive.FULL,)
assert google_auth.scopes_for({"type": "drive", "params": {}}) == (gdrive.READONLY,)
Loading
Loading