From 244e4f8c4788f788edec43ffbdfa85b363671d86 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 10 Oct 2026 21:27:27 +0000 Subject: [PATCH 1/2] feat(clipper): ask the Lens assistant from the extension's side panel The web clipper gets a side panel with the Lens assistant: Ask Lens in the popup, on the right-click menu or by a shortcut. It signs in through Lens's own consent page (OAuth with PKCE and launchWebAuthFlow, nothing to copy), or with an API token where the browser can't show that page (Safari, which opens it in a tab). Conversations are the same ones the web app shows: cited answers rendered as Markdown through DOMPurify, the steps taken, approvals to approve or decline, Stop, and Open in Lens. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ --- CHANGELOG.md | 5 + DEVLOG.md | 5 +- docs/browser-assistant.md | 28 +- docs/web-clipper.md | 3 + extensions/browser/README.md | 3 +- extensions/browser/edge/smoke.mjs | 100 +++++++- extensions/browser/package-lock.json | 40 +++ extensions/browser/package.json | 3 + extensions/browser/scripts/build.mjs | 6 +- extensions/browser/src/background.js | 14 +- extensions/browser/src/core/account.js | 226 ++++++++++++++++ extensions/browser/src/core/assistant.js | 111 ++++++++ extensions/browser/src/manifest.json | 6 +- extensions/browser/src/ui/popup.html | 5 +- extensions/browser/src/ui/popup.js | 12 + extensions/browser/src/ui/sidepanel.html | 75 ++++++ extensions/browser/src/ui/sidepanel.js | 298 ++++++++++++++++++++++ extensions/browser/targets/safari.json | 13 +- extensions/browser/test/account.test.js | 173 +++++++++++++ extensions/browser/test/assistant.test.js | 108 ++++++++ threat-model.json | 12 +- threat-model.md | 1 + threat-model.yaml | 11 +- 23 files changed, 1241 insertions(+), 17 deletions(-) create mode 100644 extensions/browser/src/core/account.js create mode 100644 extensions/browser/src/core/assistant.js create mode 100644 extensions/browser/src/ui/sidepanel.html create mode 100644 extensions/browser/src/ui/sidepanel.js create mode 100644 extensions/browser/test/account.test.js create mode 100644 extensions/browser/test/assistant.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c7ab2c96..ea8a740e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version month. **Costs** in the left menu shows spend by day, model and resource, and sets the namespace's budget ([Budgets](docs/budgets.md#the-meter-and-the-costs-page)). +- **Ask Lens from your browser.** The web clipper extension has the Lens assistant in a side panel: **Ask Lens** in + its popup or on the right-click menu. It signs in with one click through Lens's own sign-in page (OAuth, nothing to + copy), and the conversations are the same ones Lens's chat shows, with cited answers, the steps taken and approvals. + Letting the assistant use the browser itself is in progress ([The assistant in your browser](docs/browser-assistant.md)). + ## 0.7.0 October 10, 2026 {id="0.7.0"} - **Prometheus metrics.** Lens can serve `/metrics` for Prometheus and Grafana: jobs by status and pipeline, imports, diff --git a/DEVLOG.md b/DEVLOG.md index 33d9ea7c4..fb9b74fb2 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -18,13 +18,14 @@ action log. Page text reaches the model as tool results (third-party text, untru PRs: - [x] Server: sessions, actions, the browser tools, `ai.browser` and `ai.browser_steps`, the API for agents -- [ ] Extension: side panel chat, OAuth sign-in +- [x] Extension: side panel chat, OAuth sign-in (the Edge smoke test signs in through a stand-in consent page) - [ ] Extension: browser control (read with refs, click, type, keys, download into the conversation), site permissions, asking first, the action log Refine later: the browser from WhatsApp/Slack while the side panel is open (a setting, off); keep-alives in the chat stream while an action waits; browser tools over MCP; real input events (Chrome's debugger) for sites that ignore -synthetic ones. +synthetic ones; asking about the page or selection from the right-click menu; the model and namespaces picker in the +side panel. ## 2026-10-10 · Similar content diff --git a/docs/browser-assistant.md b/docs/browser-assistant.md index 726e3d9af..feb4f1715 100644 --- a/docs/browser-assistant.md +++ b/docs/browser-assistant.md @@ -1,7 +1,8 @@ # The assistant in your browser -**Status: in progress.** Built: the server side, below: sessions, the assistant's browser tools and the API. In -progress: the Lens extension's side panel, where you chat with the assistant and it uses your browser. +**Status: in progress.** Built: the server side (sessions, the assistant's browser tools and the API) and the Lens +extension's side panel, where you sign in and chat with the assistant. In progress: the side panel carrying out the +assistant's browser actions, with the sites you allow, your yes for anything consequential and the action log. Ask the assistant from your browser, and let it do things there for you: "check my email and import the important attachments", "find last month's invoice on the supplier's site and put it in Accounts", "save this page". It opens @@ -12,6 +13,27 @@ Lens looks in what it already has first. Mail from an account connected as a sou pages you captured need no browsing; the browser is for what Lens can't reach itself, such as webmail without IMAP, or a site with no API. +## The side panel + +Open it from the extension's popup (**Ask Lens**), by right-clicking any page (**Ask Lens…**), or with a shortcut you +give it in your browser's extension shortcuts (browsers suggest at most four per extension, and the clipper has them). + +**Signing in** takes one click. The address is filled in from the clipper's Lens; press **Sign in**, let the extension +reach that address when the browser asks, and allow the "Lens browser extension" on Lens's own sign-in page, where you +are signed in with your passkey. Nothing to copy or paste: the extension signs in like any app (OAuth, see +[Authentication](authentication.md#oauth)), so it needs **Let apps and AI assistants sign in** on, in Settings → API +keys (on by default). Its tokens stay on this device and renew themselves; **Sign out** hands them back to Lens, and +you can also take its access away under API tokens → Apps with access. + +**Asking** works as in Lens's chat, and it's the same conversation: **Open in Lens** picks it up there, and **New** +starts another. Answers cite their sources (a citation opens the moment it came from), the steps the assistant took +are listed above the answer, and anything it proposes (an import, say) waits in the panel for **Approve** or +**Decline**. Enter sends, Shift+Enter starts a new line and Esc stops the +answer. + +Safari has no side panel and doesn't let extensions show a sign-in page, so there **Ask Lens** opens in a tab, and you +sign in with a write API token you make in Lens (your account's API tokens). Firefox's sidebar comes with its build. + ## How it works The Lens extension's side panel signs in to your Lens and connects a **session**. While the side panel is open, a @@ -59,6 +81,8 @@ A question sent with `browser` (the session's id) on `POST /api/v1/chats/{id}/me ## Refine later +- Ask about the page you're on, or the text you selected, straight from the right-click menu. +- Pick the model, or the namespaces to look in, in the side panel (the web app's chat has both). - The assistant's browser for conversations elsewhere (WhatsApp, Slack, Telegram) while the side panel is open, as a setting that's off by default. - Keep-alive events in the chat stream while an action waits for you, for proxies that cut quiet streams short. diff --git a/docs/web-clipper.md b/docs/web-clipper.md index 95a9fc9e7..b178d849f 100644 --- a/docs/web-clipper.md +++ b/docs/web-clipper.md @@ -42,6 +42,9 @@ network. With **Right-click and shortcut clips open the popup** on, those open the popup to look over first instead. +The same extension has the Lens assistant in a side panel: **Ask Lens** in the popup, or **Ask Lens…** on the +right-click menu. See [The assistant in your browser](browser-assistant.md). + ## Templates What's sent is the template with the clip's variables filled in. The default puts the metadata in YAML front matter diff --git a/extensions/browser/README.md b/extensions/browser/README.md index 52f4c952e..3ad1c2c7f 100644 --- a/extensions/browser/README.md +++ b/extensions/browser/README.md @@ -8,6 +8,7 @@ npm ci && npm test && npm run build # then Load unpacked: dist/chrome ``` - `src/core/`: the shared code every browser uses: `clip.js` (page, selection, element), `metadata.js`, - `markdown.js` (Turndown), `template.js`, `picker.js`, `lens.js` (pushing to Lens), `settings.js` + `markdown.js` (Turndown), `template.js`, `picker.js`, `lens.js` (pushing to Lens), `settings.js`; for the assistant's + side panel, `account.js` (signing in with OAuth) and `assistant.js` (the chat API, answers as safe HTML) - `src/background.js`, `src/content.js`, `src/ui/`: the extension around it - `targets/.json`: what each browser's manifest changes from `src/manifest.json` diff --git a/extensions/browser/edge/smoke.mjs b/extensions/browser/edge/smoke.mjs index fa383667c..16b40ca42 100644 --- a/extensions/browser/edge/smoke.mjs +++ b/extensions/browser/edge/smoke.mjs @@ -1,6 +1,9 @@ // Edge smoke test: loads dist/edge (npm run build in ..) into Microsoft Edge, opens the popup and options, clips an -// article page and checks a stand-in Lens import webhook got its Markdown. Run under a display (xvfb-run in CI). +// article page and checks a stand-in Lens import webhook got its Markdown. Then it signs the side panel in to the +// stand-in Lens through its OAuth consent page, asks the assistant and approves what it proposes. Run under a display +// (xvfb-run in CI). // EDGE_CHANNEL=chromium runs the same test in Playwright's Chromium, for machines without Edge. +import { createHash } from "node:crypto"; import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { createServer } from "node:http"; import { tmpdir } from "node:os"; @@ -22,8 +25,20 @@ ${"

The Lens clipper turns what you read into Markdown with its metadata and i

  • first point
  • second point
const x = 1;
footer links
`; -// the article page and a stand-in for Lens's import webhook, on one local port +// the article page and a stand-in for Lens (its import webhook, app sign-in and assistant), on one local port const pushes = []; +const asked = []; +const decided = []; +const ACCESS = "lens-smoke-access-token"; +let challenge = ""; +const sse = (events) => events.map(([name, data]) => `event: ${name}\ndata: ${JSON.stringify(data)}\n\n`).join(""); +const ANSWER = sse([ + ["step", { tool: "search", args: {}, summary: "Searched for “renewal”" }], + ["approval", { id: 41, tool: "import_files", summary: "Import invoice.pdf into Personal" }], + ["passages", [{ n: 1, recording_id: 7, title: "Renewal call", time: "12:34", t0: 754000, text: "renewal" }]], + ["token", { text: "The renewal is **due in March** [1]." }], + ["done", { message: 12 }], +]); const server = createServer((req, res) => { const cors = { "Access-Control-Allow-Origin": "*", @@ -31,6 +46,7 @@ const server = createServer((req, res) => { "Access-Control-Allow-Methods": "POST, OPTIONS", }; if (req.method === "OPTIONS") return res.writeHead(204, cors).end(); + if (lens(req, res)) return; if (req.url === "/article") return res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" }).end(ARTICLE); if (req.url === "/api/v1/hooks/import" && req.method === "POST") { let body = ""; @@ -44,6 +60,60 @@ const server = createServer((req, res) => { } res.writeHead(404, cors).end(); }); + +function lens(req, res) { + const url = new URL(req.url, origin); + const reply = (status, body, headers = {}) => res.writeHead(status, { "Content-Type": "application/json", ...headers }).end(JSON.stringify(body)); + const read = (then) => { + let body = ""; + req.on("data", (c) => (body += c)); + req.on("end", () => then(body)); + }; + if (url.pathname === "/.well-known/oauth-authorization-server") { + reply(200, { + issuer: origin, + authorization_endpoint: `${origin}/oauth/authorize`, + token_endpoint: `${origin}/api/v1/oauth/token`, + registration_endpoint: `${origin}/api/v1/oauth/register`, + revocation_endpoint: `${origin}/api/v1/oauth/revoke`, + }); + } else if (url.pathname === "/api/v1/oauth/register") { + read(() => reply(201, { client_id: "smoke-client" })); + } else if (url.pathname === "/oauth/authorize") { + // the consent page, with the person saying yes at once: back to the extension with a code + challenge = url.searchParams.get("code_challenge"); + const back = new URL(url.searchParams.get("redirect_uri")); + back.searchParams.set("code", "smoke-code"); + back.searchParams.set("state", url.searchParams.get("state")); + res.writeHead(302, { Location: back.toString() }).end(); + } else if (url.pathname === "/api/v1/oauth/token") { + read((body) => { + const form = new URLSearchParams(body); + const ok = form.get("code") === "smoke-code" && createHash("sha256").update(form.get("code_verifier")).digest("base64url") === challenge; + reply(ok ? 200 : 400, ok ? { access_token: ACCESS, refresh_token: "smoke-refresh", expires_in: 3600 } : { error: "invalid_grant" }); + }); + } else if (url.pathname.startsWith("/api/v1/") && req.headers.authorization !== `Bearer ${ACCESS}` && !url.pathname.startsWith("/api/v1/hooks/")) { + reply(401, { detail: "sign in" }); + } else if (url.pathname === "/api/v1/auth/me") { + reply(200, { user: { email: "ada@example.com", name: "Ada" } }); + } else if (url.pathname === "/api/v1/chats" && req.method === "POST") { + read(() => reply(200, { id: 5 })); + } else if (url.pathname === "/api/v1/chats/5/messages" && req.method === "POST") { + read((body) => { + asked.push(JSON.parse(body)); + res.writeHead(200, { "Content-Type": "text/event-stream" }).end(ANSWER); + }); + } else if (url.pathname === "/api/v1/approvals/41" && req.method === "POST") { + read((body) => { + decided.push(JSON.parse(body)); + reply(200, { status: "done" }); + }); + } else { + return false; + } + return true; +} + await new Promise((ok) => server.listen(0, "127.0.0.1", ok)); const origin = `http://127.0.0.1:${server.address().port}`; @@ -78,7 +148,7 @@ try { // a first install opens the options page; it and the popup render without errors const errors = []; - for (const page of ["options.html", "popup.html"]) { + for (const page of ["options.html", "popup.html", "sidepanel.html"]) { const p = await context.newPage(); p.on("pageerror", (e) => errors.push(`${page}: ${e.message}`)); await p.goto(`chrome-extension://${id}/${page}`); @@ -116,6 +186,30 @@ try { const push = pushes.at(-1); check(push?.auth === `Bearer ${TOKEN}`, "Lens got it with the hook's token"); check(push && JSON.parse(push.body).text === md, "and the Markdown as text"); + + // the side panel: sign in through the consent page, ask, see the cited answer, approve what it proposes + check(manifest.side_panel?.default_path === "sidepanel.html" && manifest.permissions.includes("sidePanel"), "the manifest has the side panel"); + const panel = await context.newPage(); + panel.on("pageerror", (e) => errors.push(`side panel: ${e.message}`)); + await panel.goto(`chrome-extension://${id}/sidepanel.html`); + check((await panel.inputValue("#server")) === origin, "the side panel offers the clipper's Lens to sign in to"); + await panel.click("#signinButton"); + await panel.waitForSelector("#q:visible", { timeout: 20000 }).catch(() => null); + check(await panel.isVisible("#q"), `the side panel signs in through Lens's consent page${(await panel.isVisible("#q")) ? "" : `: ${await panel.textContent("#signinStatus")}`}`); + const kept = await panel.evaluate(async () => (await chrome.storage.local.get("account")).account); + check(kept?.accessToken === ACCESS && kept.email === "ada@example.com", "and keeps the token and who signed in"); + await panel.fill("#q", "When is the renewal?"); + await panel.press("#q", "Enter"); + await panel.waitForSelector(".msg.assistant a.cite", { timeout: 10000 }).catch(() => null); + check(asked.at(-1)?.content === "When is the renewal?", "Enter asks the question"); + check((await panel.textContent(".msg.assistant .steps")).includes("Searched for"), "the steps show"); + const cite = await panel.getAttribute(".msg.assistant a.cite", "href").catch(() => null); + check(cite === `${origin}/resources/7?t=754`, "the answer cites the moment it came from"); + check((await panel.textContent(".msg.assistant strong").catch(() => "")) === "due in March", "the answer is Markdown"); + await panel.click(".approval button.primary"); + await panel.waitForFunction(() => document.querySelector(".approval")?.textContent.includes("Done"), null, { timeout: 5000 }).catch(() => null); + check(decided.at(-1)?.decision === "approve", "approving sends the person's pick to Lens"); + check(errors.length === 0, `no page errors${errors.length ? `: ${errors.join("; ")}` : ""}`); } finally { await context.close(); server.close(); diff --git a/extensions/browser/package-lock.json b/extensions/browser/package-lock.json index 218aefcc1..0cce984a4 100644 --- a/extensions/browser/package-lock.json +++ b/extensions/browser/package-lock.json @@ -10,6 +10,9 @@ "license": "MIT", "dependencies": { "@mozilla/readability": "0.6.0", + "dompurify": "3.4.16", + "eventsource-parser": "4.1.1", + "marked": "18.0.14", "turndown": "7.2.4", "turndown-plugin-gfm": "1.0.2" }, @@ -604,6 +607,13 @@ "node": ">=14.0.0" } }, + "node_modules/@types/trusted-types": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz", + "integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==", + "license": "MIT", + "optional": true + }, "node_modules/agent-base": { "version": "7.1.4", "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", @@ -667,6 +677,15 @@ "dev": true, "license": "MIT" }, + "node_modules/dompurify": { + "version": "3.4.16", + "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.16.tgz", + "integrity": "sha512-sqo+pNp3qRhCIpbgRi1y8Tgk27Bo2Ry7w0dC1NBeNTdZChWjz9Xb/KOoZbRP/R6pQZ80Qw8YhXw13hWWBbMRnQ==", + "license": "(MPL-2.0 OR Apache-2.0)", + "optionalDependencies": { + "@types/trusted-types": "^2.0.7" + } + }, "node_modules/entities": { "version": "6.0.1", "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz", @@ -722,6 +741,15 @@ "@esbuild/win32-x64": "0.28.2" } }, + "node_modules/eventsource-parser": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-4.1.1.tgz", + "integrity": "sha512-zaWNqeSLeyzEAAt2Q6aDBRDRk/VnPmsKqvToyGvv6aahvLv71A4Jw7jG93Hckf11zXurbqgsffvpSgKPoeU1Qw==", + "license": "MIT", + "engines": { + "node": ">=22.12" + } + }, "node_modules/html-encoding-sniffer": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-4.0.0.tgz", @@ -830,6 +858,18 @@ "dev": true, "license": "ISC" }, + "node_modules/marked": { + "version": "18.0.14", + "resolved": "https://registry.npmjs.org/marked/-/marked-18.0.14.tgz", + "integrity": "sha512-mBHK6FBHuBAlhgRe88w9F0O1AbwwXJUcQibUbC/QcdTbVGAD7aWza+xt3N6oT/jCZx3/OMeS+8rnuiHZcQ9s7A==", + "license": "MIT", + "bin": { + "marked": "bin/marked.js" + }, + "engines": { + "node": ">= 20" + } + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", diff --git a/extensions/browser/package.json b/extensions/browser/package.json index d4cab0cff..243ea7aff 100644 --- a/extensions/browser/package.json +++ b/extensions/browser/package.json @@ -12,6 +12,9 @@ }, "dependencies": { "@mozilla/readability": "0.6.0", + "dompurify": "3.4.16", + "eventsource-parser": "4.1.1", + "marked": "18.0.14", "turndown": "7.2.4", "turndown-plugin-gfm": "1.0.2" }, diff --git a/extensions/browser/scripts/build.mjs b/extensions/browser/scripts/build.mjs index ce1c2c126..eb6f5576f 100644 --- a/extensions/browser/scripts/build.mjs +++ b/extensions/browser/scripts/build.mjs @@ -30,7 +30,8 @@ function merge(base, over) { function referenced(m) { const files = new Set([...Object.values(m.icons ?? {}), ...Object.values(m.action?.default_icon ?? {})]); - for (const f of [m.action?.default_popup, m.options_ui?.page, m.background?.service_worker, ...(m.background?.scripts ?? [])]) if (f) files.add(f); + const pages = [m.action?.default_popup, m.options_ui?.page, m.side_panel?.default_path, m.sidebar_action?.default_panel]; + for (const f of [...pages, m.background?.service_worker, ...(m.background?.scripts ?? [])]) if (f) files.add(f); return [...files]; } @@ -51,6 +52,7 @@ for (const target of targets) { content: join(root, "src/content.js"), popup: join(root, "src/ui/popup.js"), options: join(root, "src/ui/options.js"), + sidepanel: join(root, "src/ui/sidepanel.js"), }, outdir: out, bundle: true, @@ -61,7 +63,7 @@ for (const target of targets) { define: { "process.env.NODE_ENV": '"production"' }, logLevel: "warning", }); - for (const f of ["popup.html", "options.html", "style.css"]) cpSync(join(root, "src/ui", f), join(out, f)); + for (const f of ["popup.html", "options.html", "sidepanel.html", "style.css"]) cpSync(join(root, "src/ui", f), join(out, f)); cpSync(join(root, "src/icons"), join(out, "icons"), { recursive: true }); const manifest = merge({ ...base, version }, JSON.parse(readFileSync(join(root, "targets", `${target}.json`), "utf8"))); writeFileSync(join(out, "manifest.json"), `${JSON.stringify(manifest, null, 2)}\n`); diff --git a/extensions/browser/src/background.js b/extensions/browser/src/background.js index aa8d22c42..b93851b6d 100644 --- a/extensions/browser/src/background.js +++ b/extensions/browser/src/background.js @@ -1,5 +1,5 @@ // The extension's background: context menus, shortcuts, and every push to Lens (so a send finishes even when the -// popup closes). +// popup closes). The assistant lives in the side panel (ui/sidepanel.js); this only opens it. import { fetchImages, push } from "./core/lens.js"; import * as settingsStore from "./core/settings.js"; @@ -11,6 +11,7 @@ const MENU = [ { id: "pick", title: "Pick part of the page to clip…", contexts: ["page", "selection"] }, { id: "clip-link", title: "Send this link to Lens", contexts: ["link"] }, { id: "clip-image", title: "Send this image to Lens", contexts: ["image"] }, + { id: "open-assistant", title: "Ask Lens…", contexts: ["page", "selection", "link", "image"] }, ]; ext.runtime.onInstalled.addListener(async ({ reason }) => { @@ -22,6 +23,15 @@ ext.runtime.onInstalled.addListener(async ({ reason }) => { } }); +// ---------- the assistant ---------- +// Called straight from the person's click or shortcut (browsers open a side panel only then): beside the page where +// the browser has a side panel or sidebar, else in a tab of its own. +function openAssistant(tab) { + if (ext.sidePanel?.open && tab?.windowId != null) return ext.sidePanel.open({ windowId: tab.windowId }); + if (ext.sidebarAction?.open) return ext.sidebarAction.open(); + return ext.tabs.create({ url: ext.runtime.getURL("sidepanel.html") }); +} + // ---------- running the clipper in a tab ---------- async function inject(tabId) { await ext.scripting.executeScript({ target: { tabId }, files: ["content.js"] }); @@ -112,6 +122,7 @@ async function quick(tab, mode) { ext.contextMenus.onClicked.addListener(async (info, tab) => { try { + if (info.menuItemId === "open-assistant") return await openAssistant(tab); if (info.menuItemId === "clip-page") return quick(tab, "page"); if (info.menuItemId === "clip-selection") return quick(tab, "selection"); if (info.menuItemId === "pick") return startPicker(tab.id, "quick"); @@ -136,6 +147,7 @@ ext.contextMenus.onClicked.addListener(async (info, tab) => { }); ext.commands.onCommand.addListener(async (command, tab) => { + if (command === "open-assistant") return openAssistant(tab); tab ??= (await ext.tabs.query({ active: true, currentWindow: true }))[0]; if (!tab) return; if (command === "clip-page") return quick(tab, "page"); diff --git a/extensions/browser/src/core/account.js b/extensions/browser/src/core/account.js new file mode 100644 index 000000000..460588fea --- /dev/null +++ b/extensions/browser/src/core/account.js @@ -0,0 +1,226 @@ +// Signing in to a Lens as its person, for the side panel: OAuth 2.1 with PKCE (docs/authentication.md#oauth), the way +// an MCP client does. The extension registers itself with the Lens, the browser shows Lens's own consent page (signed +// in with a passkey there), and the code comes back to the extension's redirect address: nothing to copy or paste. +// The tokens act as the person and stay on this device; they renew themselves, and signing out hands them back. +// +// A browser without launchWebAuthFlow (Safari) signs in with an API token the person makes in Lens instead. + +const ext = globalThis.browser ?? globalThis.chrome; +const KEY = "account"; +const SCOPE = "read write"; +const EARLY_MS = 60_000; // renew this long before the access token runs out + +export const canOAuth = () => Boolean(ext?.identity?.launchWebAuthFlow); + +const b64url = (bytes) => + btoa(String.fromCharCode(...new Uint8Array(bytes))) + .replace(/\+/g, "-") + .replace(/\//g, "_") + .replace(/=+$/, ""); + +/** A PKCE verifier and its S256 challenge (RFC 7636). */ +export async function pkce() { + const verifier = b64url(crypto.getRandomValues(new Uint8Array(32))); + const challenge = b64url(await crypto.subtle.digest("SHA-256", new TextEncoder().encode(verifier))); + return { verifier, challenge }; +} + +async function json(res, what) { + let body = {}; + try { + body = await res.json(); + } catch { + // not JSON + } + if (!res.ok) { + const why = body.error_description || (typeof body.detail === "string" ? body.detail : "") || `${res.status} ${res.statusText}`; + throw Object.assign(new Error(`${what}: ${why}`), { status: res.status }); + } + return body; +} + +/** Where a Lens signs apps in (RFC 8414); an error that says so when apps can't sign in there. */ +export async function discover(server, { fetchImpl = fetch } = {}) { + let res; + try { + res = await fetchImpl(`${server}/.well-known/oauth-authorization-server`, { credentials: "omit" }); + } catch (e) { + throw new Error(`couldn't reach ${server}: ${e.message}`); + } + if (res.status === 404) throw new Error("this Lens doesn't let apps sign in: an admin turns it on in Settings → API keys"); + return json(res, "this address didn't answer as a Lens"); +} + +export async function load() { + return (await ext.storage.local.get(KEY))[KEY] ?? null; +} + +async function store(account) { + if (account) await ext.storage.local.set({ [KEY]: account }); + else await ext.storage.local.remove(KEY); + return account; +} + +function tokens(account, got) { + return { + ...account, + accessToken: got.access_token, + refreshToken: got.refresh_token ?? account.refreshToken, + expiresAt: Date.now() + (got.expires_in ?? 3600) * 1000, + }; +} + +/** + * Sign in to the Lens at `server` (its address, normalised). Registers the extension there the first time, shows the + * consent page, and swaps the code for tokens. `launch` is identity.launchWebAuthFlow (tests pass their own). + */ +export async function signIn(server, { fetchImpl = fetch, launch, redirectUri } = {}) { + launch ??= (opts) => ext.identity.launchWebAuthFlow(opts); + redirectUri ??= ext.identity.getRedirectURL("lens"); + const meta = await discover(server, { fetchImpl }); + const before = await load(); + let clientId = before?.server === server && before.redirectUri === redirectUri ? before.clientId : null; + if (!clientId) { + const reg = await json( + await fetchImpl(meta.registration_endpoint, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + client_name: "Lens browser extension", + redirect_uris: [redirectUri], + token_endpoint_auth_method: "none", + }), + credentials: "omit", + }), + "Lens didn't register the extension", + ); + clientId = reg.client_id; + } + const { verifier, challenge } = await pkce(); + const state = b64url(crypto.getRandomValues(new Uint8Array(16))); + const url = new URL(meta.authorization_endpoint); + for (const [k, v] of Object.entries({ + response_type: "code", + client_id: clientId, + redirect_uri: redirectUri, + code_challenge: challenge, + code_challenge_method: "S256", + scope: SCOPE, + state, + })) + url.searchParams.set(k, v); + let back; + try { + back = new URL(await launch({ url: url.toString(), interactive: true })); + } catch (e) { + throw new Error(/did not approve|cancel|closed/i.test(e.message) ? "signing in was cancelled" : e.message); + } + if (back.searchParams.get("state") !== state) throw new Error("the sign-in came back for another request; try again"); + if (back.searchParams.get("error")) throw new Error(back.searchParams.get("error") === "access_denied" ? "you said no in Lens" : back.searchParams.get("error")); + const code = back.searchParams.get("code"); + if (!code) throw new Error("Lens didn't send a sign-in code back"); + const got = await json( + await fetchImpl(meta.token_endpoint, { + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body: new URLSearchParams({ grant_type: "authorization_code", code, redirect_uri: redirectUri, code_verifier: verifier, client_id: clientId }), + credentials: "omit", + }), + "Lens didn't sign the extension in", + ); + const account = tokens( + { server, clientId, redirectUri, tokenEndpoint: meta.token_endpoint, revocationEndpoint: meta.revocation_endpoint }, + got, + ); + return store(await withMe(account, fetchImpl)); +} + +/** Sign in with an API token (la_…) made in Lens, for a browser that can't show the consent page. */ +export async function signInWithToken(server, token, { fetchImpl = fetch } = {}) { + const account = { server, accessToken: token.trim(), apiToken: true }; + return store(await withMe(account, fetchImpl)); +} + +async function withMe(account, fetchImpl) { + const me = await json( + await fetchImpl(`${account.server}/api/v1/auth/me`, { headers: { Authorization: `Bearer ${account.accessToken}` }, credentials: "omit" }), + "Lens didn't accept the sign-in", + ); + return { ...account, email: me.user?.email ?? null, name: me.user?.name ?? null }; +} + +let renewing = null; + +/** The access token, renewed first when it's about to run out (one renewal at a time). */ +export async function accessToken({ fetchImpl = fetch, force = false } = {}) { + const account = await load(); + if (!account) throw new SignedOut(); + if (account.apiToken || (!force && account.expiresAt - EARLY_MS > Date.now())) return account.accessToken; + renewing ??= (async () => { + try { + const res = await fetchImpl(account.tokenEndpoint, { + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body: new URLSearchParams({ grant_type: "refresh_token", refresh_token: account.refreshToken, client_id: account.clientId }), + credentials: "omit", + }); + if (res.status === 400 || res.status === 401) { + await store(null); // the grant ended (revoked in Lens, or it lapsed): sign in again + throw new SignedOut(); + } + return (await store(tokens(account, await json(res, "Lens didn't renew the sign-in")))).accessToken; + } finally { + renewing = null; + } + })(); + return renewing; +} + +export class SignedOut extends Error { + constructor() { + super("sign in to Lens first"); + this.name = "SignedOut"; + } +} + +/** fetch() to the signed-in Lens's API as its person; a 401 renews the token once and tries again. */ +export async function api(path, init = {}, { fetchImpl = fetch } = {}) { + const account = await load(); + if (!account) throw new SignedOut(); + const go = async (token) => + fetchImpl(`${account.server}/api/v1${path}`, { + ...init, + headers: { Accept: "application/json", ...(init.headers ?? {}), Authorization: `Bearer ${token}` }, + credentials: "omit", + }); + let res = await go(await accessToken({ fetchImpl })); + if (res.status === 401 && !account.apiToken) res = await go(await accessToken({ fetchImpl, force: true })); + if (res.status === 401) { + await store(null); + throw new SignedOut(); + } + return res; +} + +/** api() for JSON answers: the body, or an Error with Lens's reason. */ +export async function apiJson(path, init = {}, opts = {}) { + const res = await api(path, init, opts); + return json(res, "Lens said no"); +} + +/** Sign out: the tokens are handed back to Lens (RFC 7009) and forgotten here. */ +export async function signOut({ fetchImpl = fetch } = {}) { + const account = await load(); + await store(null); + if (!account?.revocationEndpoint) return; + try { + await fetchImpl(account.revocationEndpoint, { + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body: new URLSearchParams({ token: account.refreshToken, client_id: account.clientId }), + credentials: "omit", + }); + } catch { + // signed out here either way; the grant lapses in Lens or is removed there + } +} diff --git a/extensions/browser/src/core/assistant.js b/extensions/browser/src/core/assistant.js new file mode 100644 index 000000000..3fb543d1a --- /dev/null +++ b/extensions/browser/src/core/assistant.js @@ -0,0 +1,111 @@ +// Talking to the Lens assistant from the extension, through the same API as the web app's chat (docs/api.md, chats): +// conversations, questions that stream their answer as server-sent events, stopping, and approvals. +import createDOMPurify from "dompurify"; +import { createParser } from "eventsource-parser"; +import { marked } from "marked"; +import { api, apiJson } from "./account.js"; + +export const newChat = async (opts) => (await apiJson("/chats", { method: "POST", headers: { "Content-Type": "application/json" }, body: "{}" }, opts)).id; + +export const getChat = (cid, opts) => apiJson(`/chats/${cid}`, {}, opts); + +export const pendingApprovals = async (cid, opts) => + (await apiJson(`/approvals?chat_id=${cid}`, {}, opts)).filter((a) => a.status === "pending"); + +export const stop = (cid, opts) => apiJson(`/chats/${cid}/stop`, { method: "POST" }, opts); + +export const decide = (aid, decision, opts) => + apiJson(`/approvals/${aid}`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ decision }) }, opts); + +/** + * Ask in conversation `cid`: `onEvent(name, data)` gets each event as it streams (step, approval, notice, passages, + * token, error, stopped, done, ...). `browser` is this browser's session, so the assistant can use it. Resolves when + * the answer is done. + */ +export async function ask(cid, content, { browser, onEvent, signal, ...opts } = {}) { + const res = await api( + `/chats/${cid}/messages`, + { + method: "POST", + headers: { "Content-Type": "application/json", Accept: "text/event-stream" }, + body: JSON.stringify(browser ? { content, browser } : { content }), + signal, + }, + opts, + ); + if (!res.ok) { + let why = `Lens answered ${res.status}`; + try { + const body = await res.json(); + if (typeof body.detail === "string") why = body.detail; + } catch { + // not JSON + } + throw new Error(why); + } + await readEvents(res.body, onEvent); +} + +/** Feed a server-sent event stream (a fetch body) to `onEvent(name, data)`, data parsed as JSON. */ +export async function readEvents(body, onEvent) { + const parser = createParser({ + onEvent(e) { + let data = e.data; + try { + data = JSON.parse(e.data); + } catch { + // plain text + } + onEvent(e.event ?? "message", data); + }, + }); + const reader = body.getReader(); + const text = new TextDecoder(); + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + parser.feed(text.decode(value, { stream: true })); + } + parser.feed(text.decode()); +} + +/** Where a cited moment opens in Lens. */ +export function passageLink(server, p) { + const t = typeof p.t0 === "number" && p.t0 > 0 ? `?t=${Math.floor(p.t0 / 1000)}` : ""; + return `${server}/resources/${p.recording_id}${t}`; +} + +const escapeAttr = (s) => String(s ?? "").replace(/["\\]/g, ""); + +/** + * An answer as HTML, safe to put in the page: Markdown, with each [n] a link to the moment it cites, cleaned of + * anything that could run or fetch on its own (DOMPurify). Links open in a new tab. + */ +export function answerHtml(text, passages = [], server = "", win = globalThis.window) { + const byN = new Map(passages.map((p) => [p.n, p])); + const cite = (n) => { + const p = byN.get(n); + if (!p || !server) return `[${n}]`; + const title = escapeAttr([p.title, p.time].filter(Boolean).join(" · ")); + return `[[${n}]](${passageLink(server, p)} "${title}")`; + }; + // [1], [1, 2] and [1][2] all cite, as in the web app + const linked = String(text ?? "").replace(/\[(\d{1,3}(?:\s*,\s*\d{1,3})*)\](?!\()/g, (_all, ns) => + ns.split(",").map((n) => cite(Number(n.trim()))).join(""), + ); + const html = marked.parse(linked, { async: false, gfm: true, breaks: true }); + const purify = createDOMPurify(win); + const clean = purify.sanitize(html, { USE_PROFILES: { html: true }, FORBID_TAGS: ["img", "style", "form", "input", "button"] }); + const box = win.document.createElement("div"); + box.innerHTML = clean; + for (const a of box.querySelectorAll("a")) { + if (!/^https?:/i.test(a.getAttribute("href") ?? "")) { + a.replaceWith(...a.childNodes); + continue; + } + a.target = "_blank"; + a.rel = "noopener noreferrer"; + if (/^\[\d+\]$/.test(a.textContent)) a.className = "cite"; + } + return box.innerHTML; +} diff --git a/extensions/browser/src/manifest.json b/extensions/browser/src/manifest.json index 3abfd5f85..244a1e00b 100644 --- a/extensions/browser/src/manifest.json +++ b/extensions/browser/src/manifest.json @@ -12,13 +12,15 @@ }, "background": { "service_worker": "background.js" }, "options_ui": { "page": "options.html", "open_in_tab": true }, - "permissions": ["activeTab", "scripting", "contextMenus", "storage"], + "side_panel": { "default_path": "sidepanel.html" }, + "permissions": ["activeTab", "scripting", "contextMenus", "storage", "sidePanel", "identity"], "optional_host_permissions": ["https://*/*", "http://*/*"], "commands": { "_execute_action": { "suggested_key": { "default": "Alt+Shift+L" }, "description": "Open the clipper" }, "clip-page": { "suggested_key": { "default": "Alt+Shift+P" }, "description": "Clip this page to Lens" }, "clip-selection": { "suggested_key": { "default": "Alt+Shift+S" }, "description": "Clip the selection to Lens" }, - "pick": { "suggested_key": { "default": "Alt+Shift+K" }, "description": "Pick part of the page to clip" } + "pick": { "suggested_key": { "default": "Alt+Shift+K" }, "description": "Pick part of the page to clip" }, + "open-assistant": { "description": "Ask Lens (opens the assistant beside the page)" } }, "content_security_policy": { "extension_pages": "script-src 'self'; object-src 'self'" } } diff --git a/extensions/browser/src/ui/popup.html b/extensions/browser/src/ui/popup.html index c29fd981f..f0832d85b 100644 --- a/extensions/browser/src/ui/popup.html +++ b/extensions/browser/src/ui/popup.html @@ -15,7 +15,10 @@

Clip to Lens

- + + + +
diff --git a/extensions/browser/src/ui/popup.js b/extensions/browser/src/ui/popup.js index 9a7e0ca0a..1570b4988 100644 --- a/extensions/browser/src/ui/popup.js +++ b/extensions/browser/src/ui/popup.js @@ -98,6 +98,18 @@ $("download").addEventListener("click", () => { }); $("options").addEventListener("click", () => ext.runtime.openOptionsPage()); +$("assistant").addEventListener("click", () => { + // opened straight from the click: browsers open a side panel only from the person's own action + const opened = ext.sidePanel?.open && tab + ? ext.sidePanel.open({ windowId: tab.windowId }) + : ext.sidebarAction?.open + ? ext.sidebarAction.open() + : ext.tabs.create({ url: ext.runtime.getURL("sidepanel.html") }); + Promise.resolve(opened).then( + () => window.close(), + (e) => status(e.message, "bad"), + ); +}); for (const b of document.querySelectorAll(".seg button")) b.addEventListener("click", () => run(b.dataset.mode)); (async () => { diff --git a/extensions/browser/src/ui/sidepanel.html b/extensions/browser/src/ui/sidepanel.html new file mode 100644 index 000000000..5a84213b0 --- /dev/null +++ b/extensions/browser/src/ui/sidepanel.html @@ -0,0 +1,75 @@ + + + + + + Ask Lens + + + + +
+

Ask Lens

+ + + + +
+
+ + +
+ + + diff --git a/extensions/browser/src/ui/sidepanel.js b/extensions/browser/src/ui/sidepanel.js new file mode 100644 index 000000000..4f14ffd5b --- /dev/null +++ b/extensions/browser/src/ui/sidepanel.js @@ -0,0 +1,298 @@ +// The side panel: the Lens assistant next to any page (docs/browser-assistant.md). Sign in once, through Lens's own +// sign-in page, then ask. The conversation is the same one the web app shows, so it can be picked up there. +import * as account from "../core/account.js"; +import * as assistant from "../core/assistant.js"; +import * as settingsStore from "../core/settings.js"; + +const ext = globalThis.browser ?? globalThis.chrome; +const $ = (id) => document.getElementById(id); +const CHAT_KEY = "assistantChat"; // {server, id}: the conversation the side panel continues + +let me = null; // the signed-in account +let chatId = null; +let answering = null; // {controller} while an answer streams + +function el(tag, props = {}, ...children) { + const node = document.createElement(tag); + Object.assign(node, props); + node.append(...children.filter((c) => c != null)); + return node; +} + +// ---------- signing in ---------- +function signinStatus(text, kind = "") { + $("signinStatus").className = `status ${kind}`; + $("signinStatus").textContent = text; +} + +async function showSignIn(why = "") { + me = null; + chatId = null; + for (const id of ["new", "open", "signout", "chat"]) $(id).hidden = true; + $("signin").hidden = false; + $("tokenRow").hidden = account.canOAuth(); + $("token").required = !account.canOAuth(); + if (!$("server").value) { + const dest = settingsStore.destination(await settingsStore.load()); + if (dest) $("server").value = dest.server; + } + signinStatus(why, why ? "bad" : ""); + // the first thing left to fill has focus; with the address known, Enter signs in + const next = !$("server").value ? $("server") : !$("tokenRow").hidden ? $("token") : $("signinButton"); + next.focus(); +} + +// browsers grant an extension a site only from the person's own click, so this is asked before anything is awaited +function askForSite(server) { + if (!ext.permissions?.request) return Promise.resolve(true); + return ext.permissions.request({ origins: [`${new URL(server).origin}/*`] }); +} + +$("signinForm").addEventListener("submit", (e) => { + e.preventDefault(); + let server; + try { + server = settingsStore.normalizeServer($("server").value); + } catch (err) { + return signinStatus(err.message, "bad"); + } + const allowed = askForSite(server); + (async () => { + $("signinButton").disabled = true; + signinStatus("Signing in…"); + try { + if (!(await allowed)) throw new Error(`allow the extension to reach ${new URL(server).host} when the browser asks`); + me = account.canOAuth() ? await account.signIn(server) : await account.signInWithToken(server, $("token").value); + $("token").value = ""; + signinStatus(""); + await showChat(); + } catch (err) { + signinStatus(`Couldn't sign in: ${err.message}`, "bad"); + } finally { + $("signinButton").disabled = false; + } + })(); +}); + +$("signout").addEventListener("click", async () => { + if (answering) await stopAnswer(); + await ext.storage.local.remove(CHAT_KEY); + await account.signOut(); + $("log").replaceChildren(); + await showSignIn(""); +}); + +// ---------- the conversation ---------- +function scrollDown(force = false) { + const log = $("log"); + if (force || log.scrollHeight - log.scrollTop - log.clientHeight < 80) log.scrollTop = log.scrollHeight; +} + +function emptyHint() { + if (!$("log").childElementCount) { + $("log").append(el("p", { className: "empty", textContent: "Ask about anything in your Lens: it answers with its sources." })); + } +} + +function setChat(id) { + chatId = id; + $("open").href = id ? `${me.server}/chat/${id}` : `${me.server}/chat`; + return ext.storage.local.set({ [CHAT_KEY]: { server: me.server, id } }); +} + +function addUser(text) { + $("log").querySelector(".empty")?.remove(); + $("log").append(el("div", { className: "msg user", textContent: text })); + scrollDown(true); +} + +/** An answer in the log, filled in as it streams. */ +function addAnswer() { + $("log").querySelector(".empty")?.remove(); + const steps = el("ul", { className: "steps", hidden: true }); + const thinking = el("p", { className: "thinking", textContent: "Thinking…" }); + const body = el("div", { className: "body" }); + const root = el("div", { className: "msg assistant" }, steps, thinking, body); + $("log").append(root); + return { + root, + step(text) { + steps.hidden = false; + steps.append(el("li", { textContent: text })); + scrollDown(); + }, + answer(text, passages) { + thinking.remove(); + body.innerHTML = assistant.answerHtml(text, passages, me.server, window); + scrollDown(); + }, + note(text, kind = "") { + root.append(el("p", { className: `note ${kind}`, textContent: text })); + scrollDown(); + }, + approval(a) { + root.append(approvalCard(a)); + scrollDown(); + }, + done() { + thinking.remove(); + }, + }; +} + +const DECIDED = { done: "Done", declined: "Declined" }; + +/** Work the assistant proposed, waiting for the person's pick (it never does it on its own). */ +function approvalCard(a) { + const status = el("span", { className: "status" }); + const buttons = el("p", { className: "row" }); + const decide = async (decision) => { + for (const b of buttons.querySelectorAll("button")) b.disabled = true; + status.className = "status"; + status.textContent = "…"; + try { + const out = await assistant.decide(a.id, decision); + buttons.replaceChildren(el("span", { className: "status ok", textContent: DECIDED[out.status] ?? out.status })); + } catch (err) { + if (err instanceof account.SignedOut) return showSignIn(err.message); + status.className = "status bad"; + status.textContent = err.message; + for (const b of buttons.querySelectorAll("button")) b.disabled = false; + } + }; + const approve = el("button", { className: "primary", type: "button", textContent: "Approve" }); + const decline = el("button", { type: "button", textContent: "Decline" }); + approve.addEventListener("click", () => decide("approve")); + decline.addEventListener("click", () => decide("decline")); + buttons.append(approve, decline, status); + return el("div", { className: "approval" }, el("p", { textContent: a.summary || a.tool }), buttons); +} + +function showMessage(m) { + if (m.role === "user") return addUser(m.content); + const view = addAnswer(); + for (const s of m.steps ?? []) view.step(s.summary || s.tool); + if (m.content && m.content !== "(no answer)") view.answer(m.content, m.passages ?? []); + if (m.notice) view.note(m.notice); + if (m.stopped) view.note("Stopped"); + if (m.error) view.note(m.error, "bad"); + view.done(); +} + +async function showChat() { + $("signin").hidden = true; + for (const id of ["new", "open", "signout", "chat"]) $(id).hidden = false; + $("signout").title = `Sign out of ${me.email ?? "Lens"} at ${new URL(me.server).host}`; + $("log").replaceChildren(); + const { [CHAT_KEY]: saved } = await ext.storage.local.get(CHAT_KEY); + await setChat(saved?.server === me.server ? saved.id : null); + if (chatId) { + try { + const c = await assistant.getChat(chatId); + for (const m of c.messages ?? []) showMessage(m); + const waiting = await assistant.pendingApprovals(chatId); + if (waiting.length) { + const view = addAnswer(); + view.done(); + for (const a of waiting.reverse()) view.approval(a); + } + } catch (err) { + if (err instanceof account.SignedOut) return showSignIn(err.message); + if (err.status === 404) await setChat(null); // removed in Lens: start afresh + else $("log").append(el("p", { className: "note bad", textContent: `Couldn't load the conversation: ${err.message}` })); + } + } + emptyHint(); + scrollDown(true); + $("q").focus(); +} + +$("new").addEventListener("click", async () => { + if (answering) await stopAnswer(); + await setChat(null); + $("log").replaceChildren(); + emptyHint(); + $("q").focus(); +}); + +function setAnswering(on) { + $("send").textContent = on ? "Stop" : "Send"; + $("send").classList.toggle("primary", !on); + $("send").title = on ? "Stop the answer (Esc)" : "Send (Enter)"; +} + +async function stopAnswer() { + if (!answering) return; + try { + await assistant.stop(chatId); // Lens keeps what came before and ends the stream + } catch { + answering?.controller.abort(); + } +} + +async function send(text) { + const view = { current: null }; + try { + if (!chatId) await setChat(await assistant.newChat()); + addUser(text); + view.current = addAnswer(); + answering = { controller: new AbortController() }; + setAnswering(true); + let answer = ""; + let passages = []; + await assistant.ask(chatId, text, { + signal: answering.controller.signal, + onEvent(name, data) { + const v = view.current; + if (name === "step") v.step(data.summary || data.tool); + else if (name === "approval") v.approval(data); + else if (name === "notice") v.note(data.message); + else if (name === "passages") passages = Array.isArray(data) ? data : []; + else if (name === "token") v.answer((answer += data.text ?? ""), passages); + else if (name === "error") v.note(data.message, "bad"); + else if (name === "stopped") v.note("Stopped"); + else if (name === "cleared") v.note("The assistant starts afresh from here"); + else if (name === "compacted") v.note("The conversation so far is folded into a summary"); + }, + }); + } catch (err) { + if (err instanceof account.SignedOut) return showSignIn(err.message); + if (err.name === "AbortError") view.current?.note("Stopped"); + else if (view.current) view.current.note(err.message, "bad"); + else $("log").append(el("p", { className: "note bad", textContent: err.message })); + } finally { + view.current?.done(); + answering = null; + setAnswering(false); + } +} + +$("ask").addEventListener("submit", (e) => { + e.preventDefault(); + if (answering) return stopAnswer(); + const text = $("q").value.trim(); + if (!text) return; + $("q").value = ""; + send(text).finally(() => $("q").focus()); +}); + +$("q").addEventListener("keydown", (e) => { + if (e.key === "Enter" && !e.shiftKey && !e.isComposing) { + e.preventDefault(); + if (!answering) $("ask").requestSubmit(); // the next question waits for this answer; Esc stops it + } +}); + +document.addEventListener("keydown", (e) => { + if (e.key === "Escape" && answering) { + e.preventDefault(); + stopAnswer(); + } +}); + +// ---------- start ---------- +(async () => { + me = await account.load(); + if (me) await showChat(); + else await showSignIn(); +})().catch((err) => showSignIn(err.message)); diff --git a/extensions/browser/targets/safari.json b/extensions/browser/targets/safari.json index f4769ad24..c9149877c 100644 --- a/extensions/browser/targets/safari.json +++ b/extensions/browser/targets/safari.json @@ -1,5 +1,16 @@ { "host_permissions": ["https://*/*", "http://*/*"], "optional_host_permissions": null, - "browser_specific_settings": { "safari": { "strict_min_version": "16.4" } } + "side_panel": null, + "permissions": [ + "activeTab", + "scripting", + "contextMenus", + "storage" + ], + "browser_specific_settings": { + "safari": { + "strict_min_version": "16.4" + } + } } diff --git a/extensions/browser/test/account.test.js b/extensions/browser/test/account.test.js new file mode 100644 index 000000000..50ab5b3d0 --- /dev/null +++ b/extensions/browser/test/account.test.js @@ -0,0 +1,173 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { beforeEach, test } from "node:test"; + +// the extension's storage, as the browser keeps it +const area = new Map(); +globalThis.chrome = { + storage: { + local: { + async get(key) { + return area.has(key) ? { [key]: structuredClone(area.get(key)) } : {}; + }, + async set(items) { + for (const [k, v] of Object.entries(items)) area.set(k, structuredClone(v)); + }, + async remove(key) { + area.delete(key); + }, + }, + }, +}; +const account = await import("../src/core/account.js"); + +const SERVER = "https://lens.test"; +const REDIRECT = "https://abc.chromiumapp.org/lens"; +const json = (status, body) => new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } }); +const b64url = (buf) => buf.toString("base64").replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, ""); + +/** A stand-in Lens: discovery, registration, consent, tokens and /auth/me, counting what it was asked. */ +function fakeLens({ discovery = 200, refresh = 200 } = {}) { + const lens = { registered: 0, refreshed: 0, revoked: [], codes: new Map(), calls: [], access: "at-1", refreshToken: "rt-1" }; + lens.fetchImpl = async (url, init = {}) => { + lens.calls.push({ url, init }); + const body = init.body instanceof URLSearchParams ? Object.fromEntries(init.body) : init.body ? JSON.parse(init.body) : null; + if (url === `${SERVER}/.well-known/oauth-authorization-server`) { + if (discovery !== 200) return json(discovery, { detail: "not found" }); + return json(200, { + authorization_endpoint: `${SERVER}/oauth/authorize`, + token_endpoint: `${SERVER}/api/v1/oauth/token`, + registration_endpoint: `${SERVER}/api/v1/oauth/register`, + revocation_endpoint: `${SERVER}/api/v1/oauth/revoke`, + }); + } + if (url.endsWith("/oauth/register")) { + lens.registered += 1; + assert.deepEqual(body.redirect_uris, [REDIRECT]); + assert.equal(body.token_endpoint_auth_method, "none"); + return json(201, { client_id: `client-${lens.registered}` }); + } + if (url.endsWith("/oauth/token") && body.grant_type === "authorization_code") { + const challenge = lens.codes.get(body.code); + assert.ok(challenge, "a code Lens gave out"); + assert.equal(b64url(createHash("sha256").update(body.code_verifier).digest()), challenge, "the PKCE verifier matches"); + return json(200, { access_token: lens.access, refresh_token: lens.refreshToken, expires_in: 3600 }); + } + if (url.endsWith("/oauth/token") && body.grant_type === "refresh_token") { + lens.refreshed += 1; + if (refresh !== 200) return json(refresh, { error: "invalid_grant" }); + assert.equal(body.refresh_token, lens.refreshToken); + lens.access = `at-${lens.refreshed + 1}`; + lens.refreshToken = `rt-${lens.refreshed + 1}`; + return json(200, { access_token: lens.access, refresh_token: lens.refreshToken, expires_in: 3600 }); + } + if (url.endsWith("/oauth/revoke")) { + lens.revoked.push(body.token); + return json(200, { ok: true }); + } + if (url === `${SERVER}/api/v1/auth/me`) { + const ok = init.headers.Authorization === `Bearer ${lens.access}` || init.headers.Authorization === "Bearer la_token"; + return ok ? json(200, { user: { email: "ada@example.com", name: "Ada" } }) : json(401, { detail: "sign in" }); + } + if (url === `${SERVER}/api/v1/chats`) { + return init.headers.Authorization === `Bearer ${lens.access}` || init.headers.Authorization === "Bearer la_token" + ? json(200, []) + : json(401, { detail: "sign in" }); + } + return json(404, { detail: "not found" }); + }; + // the consent page: the person says yes, and the browser comes back to the extension with a code + lens.launch = async ({ url, interactive }) => { + assert.equal(interactive, true); + const u = new URL(url); + assert.equal(u.searchParams.get("code_challenge_method"), "S256"); + assert.equal(u.searchParams.get("redirect_uri"), REDIRECT); + assert.equal(u.searchParams.get("scope"), "read write"); + const code = `code-${lens.codes.size + 1}`; + lens.codes.set(code, u.searchParams.get("code_challenge")); + return `${REDIRECT}?code=${code}&state=${u.searchParams.get("state")}`; + }; + lens.signIn = () => account.signIn(SERVER, { fetchImpl: lens.fetchImpl, launch: lens.launch, redirectUri: REDIRECT }); + return lens; +} + +beforeEach(() => area.clear()); + +test("signing in registers the extension once, uses PKCE and keeps who signed in", async () => { + const lens = fakeLens(); + const me = await lens.signIn(); + assert.equal(me.email, "ada@example.com"); + assert.equal(me.accessToken, "at-1"); + assert.equal(me.clientId, "client-1"); + assert.equal((await account.load()).refreshToken, "rt-1"); + await lens.signIn(); // again: the registration is reused + assert.equal(lens.registered, 1); +}); + +test("a sign-in that comes back for another request, or says no, is refused", async () => { + const lens = fakeLens(); + const elsewhere = async ({ url }) => `${REDIRECT}?code=x&state=not-${new URL(url).searchParams.get("state")}`; + await assert.rejects(account.signIn(SERVER, { fetchImpl: lens.fetchImpl, launch: elsewhere, redirectUri: REDIRECT }), /another request/); + const no = async ({ url }) => `${REDIRECT}?error=access_denied&state=${new URL(url).searchParams.get("state")}`; + await assert.rejects(account.signIn(SERVER, { fetchImpl: lens.fetchImpl, launch: no, redirectUri: REDIRECT }), /you said no/); + const closed = async () => { + throw new Error("The user did not approve access."); + }; + await assert.rejects(account.signIn(SERVER, { fetchImpl: lens.fetchImpl, launch: closed, redirectUri: REDIRECT }), /cancelled/); + assert.equal(await account.load(), null); +}); + +test("a Lens with app sign-in off says where to turn it on", async () => { + const lens = fakeLens({ discovery: 404 }); + await assert.rejects(lens.signIn(), /Settings → API keys/); +}); + +test("an access token about to run out is renewed once, however many ask", async () => { + const lens = fakeLens(); + await lens.signIn(); + await chrome.storage.local.set({ account: { ...(await account.load()), expiresAt: Date.now() + 1000 } }); + const tokens = await Promise.all([1, 2, 3].map(() => account.accessToken({ fetchImpl: lens.fetchImpl }))); + assert.deepEqual(tokens, ["at-2", "at-2", "at-2"]); + assert.equal(lens.refreshed, 1); + assert.equal((await account.load()).refreshToken, "rt-2"); +}); + +test("a 401 renews the token and tries once more; an ended grant signs out", async () => { + const lens = fakeLens(); + await lens.signIn(); + lens.access = "at-revoked-elsewhere"; // Lens no longer takes the token the extension holds + const res = await account.api("/chats", {}, { fetchImpl: lens.fetchImpl }); + assert.equal(res.status, 200); + assert.equal(lens.refreshed, 1); + + const ended = fakeLens({ refresh: 400 }); + await ended.signIn(); + ended.access = "gone"; + await assert.rejects(account.api("/chats", {}, { fetchImpl: ended.fetchImpl }), account.SignedOut); + assert.equal(await account.load(), null); +}); + +test("signing out hands the refresh token back to Lens", async () => { + const lens = fakeLens(); + await lens.signIn(); + await account.signOut({ fetchImpl: lens.fetchImpl }); + assert.deepEqual(lens.revoked, ["rt-1"]); + assert.equal(await account.load(), null); + await assert.rejects(account.api("/chats", {}, { fetchImpl: lens.fetchImpl }), account.SignedOut); +}); + +test("a browser without the sign-in page signs in with an API token, which is never renewed", async () => { + const lens = fakeLens(); + const me = await account.signInWithToken(SERVER, " la_token ", { fetchImpl: lens.fetchImpl }); + assert.equal(me.apiToken, true); + assert.equal(me.name, "Ada"); + assert.equal((await account.api("/chats", {}, { fetchImpl: lens.fetchImpl })).status, 200); + assert.equal(lens.refreshed, 0); + await assert.rejects(account.signInWithToken(SERVER, "la_wrong", { fetchImpl: lens.fetchImpl }), /didn't accept/); +}); + +test("Lens's own reason and status come with an error", async () => { + const lens = fakeLens(); + await lens.signIn(); + await assert.rejects(account.apiJson("/chats/99", {}, { fetchImpl: lens.fetchImpl }), (e) => e.status === 404 && /not found/.test(e.message)); +}); diff --git a/extensions/browser/test/assistant.test.js b/extensions/browser/test/assistant.test.js new file mode 100644 index 000000000..8d702f122 --- /dev/null +++ b/extensions/browser/test/assistant.test.js @@ -0,0 +1,108 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { JSDOM } from "jsdom"; + +const area = new Map([["account", { server: "https://lens.test", accessToken: "la_token", apiToken: true }]]); +globalThis.chrome = { + storage: { + local: { + async get(key) { + return area.has(key) ? { [key]: structuredClone(area.get(key)) } : {}; + }, + async set(items) { + for (const [k, v] of Object.entries(items)) area.set(k, v); + }, + async remove(key) { + area.delete(key); + }, + }, + }, +}; +const assistant = await import("../src/core/assistant.js"); + +/** A streamed answer, cut into uneven chunks the way a network delivers it. */ +function stream(text, size = 7) { + const bytes = new TextEncoder().encode(text); + return new ReadableStream({ + start(c) { + for (let i = 0; i < bytes.length; i += size) c.enqueue(bytes.slice(i, i + size)); + c.close(); + }, + }); +} + +test("asking posts the question and hands over each streamed event", async () => { + let seen; + const fetchImpl = async (url, init) => { + seen = { url, init }; + const sse = [ + 'event: step\ndata: {"tool":"search","args":{},"summary":"Searched for “invoice”"}\n\n', + 'event: passages\ndata: [{"n":1,"recording_id":7,"text":"x"}]\n\n', + 'event: token\ndata: {"text":"Your invoice [1]."}\n\n', + 'event: done\ndata: {"message":12}\n\n', + ].join(""); + return new Response(stream(sse), { status: 200, headers: { "content-type": "text/event-stream" } }); + }; + const events = []; + await assistant.ask(5, "Where's my invoice?", { fetchImpl, onEvent: (name, data) => events.push([name, data]) }); + assert.equal(seen.url, "https://lens.test/api/v1/chats/5/messages"); + assert.equal(seen.init.headers.Authorization, "Bearer la_token"); + assert.deepEqual(JSON.parse(seen.init.body), { content: "Where's my invoice?" }); + assert.deepEqual( + events.map(([n]) => n), + ["step", "passages", "token", "done"], + ); + assert.equal(events[2][1].text, "Your invoice [1]."); + + await assistant.ask(5, "and now?", { browser: "browser_session:abc", fetchImpl, onEvent() {} }); + assert.equal(JSON.parse(seen.init.body).browser, "browser_session:abc"); +}); + +test("a refused question says Lens's reason", async () => { + const fetchImpl = async () => new Response(JSON.stringify({ detail: "ask something" }), { status: 400 }); + await assert.rejects(assistant.ask(5, " ", { fetchImpl, onEvent() {} }), /ask something/); +}); + +test("approvals are decided by their id, and only pending ones wait", async () => { + const calls = []; + const fetchImpl = async (url, init = {}) => { + calls.push([url, init.method ?? "GET", init.body]); + if (url.includes("/approvals?")) return Response.json([{ id: 1, status: "pending" }, { id: 2, status: "done" }]); + return Response.json({ status: "done" }); + }; + assert.deepEqual(await assistant.pendingApprovals(5, { fetchImpl }), [{ id: 1, status: "pending" }]); + assert.deepEqual(await assistant.decide(1, "decline", { fetchImpl }), { status: "done" }); + assert.deepEqual(calls[1], ["https://lens.test/api/v1/approvals/1", "POST", '{"decision":"decline"}']); +}); + +test("answers render as Markdown with their citations linked, and nothing that runs", () => { + const { window } = new JSDOM(""); + const passages = [ + { n: 1, recording_id: 7, title: "Renewal call", time: "12:34", t0: 754000, text: "x" }, + { n: 2, recording_id: 9, title: 'Odd "title"', text: "y" }, + ]; + const text = [ + "**Paid** on time [1, 2] and see [3].", + "", + "", + "[click](javascript:alert(1)) [site](https://example.com)", + ].join("\n"); + const box = window.document.createElement("div"); + box.innerHTML = assistant.answerHtml(text, passages, "https://lens.test", window); + assert.equal(box.querySelector("strong").textContent, "Paid"); + const cites = [...box.querySelectorAll("a.cite")]; + assert.deepEqual( + cites.map((a) => [a.textContent, a.href]), + [ + ["[1]", "https://lens.test/resources/7?t=754"], + ["[2]", "https://lens.test/resources/9"], + ], + ); + assert.equal(cites[0].title, "Renewal call · 12:34"); + assert.match(box.textContent, /see \[3\]/, "a citation with no passage stays as text"); + assert.equal(box.querySelector("script"), null); + assert.equal(box.querySelector("img"), null); + const links = [...box.querySelectorAll("a")]; + assert.ok(links.every((a) => a.href.startsWith("https://") && a.target === "_blank" && a.rel === "noopener noreferrer")); + assert.match(box.textContent, /click/, "an unsafe link keeps its text"); +}); diff --git a/threat-model.json b/threat-model.json index 5a165bd1c..e257b48a1 100644 --- a/threat-model.json +++ b/threat-model.json @@ -110,7 +110,8 @@ "entry_points": [ "chat, /graph/ask, MCP tools", "extensions (prompt, graph and http tools, hooks), custom nodes including template nodes", - "Python code tools, shipped and admin-created templates, workflows, pipelines, routines" + "Python code tools, shipped and admin-created templates, workflows, pipelines, routines", + "Lens extension side panel" ], "touches": [ "network", @@ -424,6 +425,15 @@ "provenance": "documented", "source": "docs/assistant.md \"limited to what the person can read\"" }, + { + "entry_point": "Lens extension side panel", + "parameter": "answers shown there (model text that can repeat third-party text), the sign-in coming back", + "attacker_controllable": "yes", + "component": "assistant", + "caller_must_enforce": "none; answers render as Markdown cleaned by DOMPurify (no scripts, images or forms; only http and https links), and the sign-in checks its state and PKCE", + "provenance": "documented", + "source": "extensions/browser/src/core/assistant.js answerHtml docstring \"cleaned of anything that could run or fetch on its own\"" + }, { "entry_point": "extensions (prompt, graph and http tools, hooks), custom nodes including template nodes", "parameter": "tool spec, graph JSON, template text", diff --git a/threat-model.md b/threat-model.md index 5a1bc3ece..b689bf0f6 100644 --- a/threat-model.md +++ b/threat-model.md @@ -279,6 +279,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | admin settings: model, embeddings, speech, decisions, Fedora, bridge, telemetry, IIIF import, rclone remotes, IMAP, iCal, MQTT bridge, OIDC issuer | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* | | replies from remote servers | web page HTML and scripts, IIIF manifests, model replies, OIDC discovery, calendars and their redirects | yes, the remote | data, topology | treat as untrusted | *(inferred, Q17)* | | chat, `/graph/ask`, MCP tools | prompt text; model-chosen tool calls | yes; also indirectly through content | data, x-prompt-injection | none | *(documented, docs/assistant.md "limited to what the person can read")* | +| Lens extension side panel | answers shown there (model text that can repeat third-party text), the sign-in coming back | yes, indirectly through content | data, x-prompt-injection | none; answers render as Markdown cleaned by DOMPurify (no scripts, images or forms; only http and https links), and the sign-in checks its `state` and PKCE | *(documented, extensions/browser/src/core/assistant.js answerHtml docstring "cleaned of anything that could run or fetch on its own")* | | extensions (prompt, graph and `http` tools, hooks), custom nodes including `template` nodes (Q23) | tool spec, graph JSON, template text | yes, any writer for their own; sharing is gated | callback/code (declarative) | person enables only extensions they trust | *(documented, docs/assistant.md "other people's run only when an admin shared them")* | | Python code tools, shipped and admin-created templates, workflows, pipelines, routines | Python, Jinja source, regexes | admin only | callback/code | admin is trusted | *(maintainer, 2026-10)* | | MQTT 1883, syslog 5514, `POST /sensors/push/{token}/{stream}` | payloads, hub logins, token in the path | yes, once enabled | data, size/rate, x-credential | admin enables; operator firewalls | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | diff --git a/threat-model.yaml b/threat-model.yaml index a80ae5597..7e9fda8d5 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:42d9bc92f48ce93ea2d58412a61b086dde8ab4522af95f628304d1b33919832f" +prose_version: "threat-model.md@sha256:08a738a66f55bcc1f5830f4dba5c5278143767cccc7405f0d4fd856d975d6b87" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -359,6 +359,15 @@ entry_points: control_kinds: [data, x-prompt-injection] caller_must_enforce: "none — P-RETRIEVAL" provenance: {kind: documented, source: 'docs/assistant.md "limited to what the person can read"'} + - id: "Lens extension side panel" + component: assistant + parameters: + - name: "answers shown there (model text that can repeat third-party text), the sign-in coming back" + attacker_controllable: true + x-condition: "indirectly through content" + control_kinds: [data, x-prompt-injection] + caller_must_enforce: "none; answers render as Markdown cleaned by DOMPurify (no scripts, images or forms; only http and https links), and the sign-in checks its state and PKCE" + provenance: {kind: documented, source: 'extensions/browser/src/core/assistant.js answerHtml docstring "cleaned of anything that could run or fetch on its own"'} - id: "extensions (prompt, graph and http tools, hooks), custom nodes including template nodes" component: assistant parameters: From 813842d3c0b88fb730dc8dbfcbe87aaf0b718091 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 10 Oct 2026 21:52:22 +0000 Subject: [PATCH 2/2] feat(clipper): open the assistant in Firefox's sidebar Firefox has no side_panel; its target maps the panel to sidebar_action (not opened at install) and drops the sidePanel permission. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ --- docs/browser-assistant.md | 5 +++-- docs/web-clipper.md | 3 ++- extensions/browser/targets/firefox.json | 8 ++++++++ 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/browser-assistant.md b/docs/browser-assistant.md index feb4f1715..b1c83b10a 100644 --- a/docs/browser-assistant.md +++ b/docs/browser-assistant.md @@ -31,8 +31,9 @@ are listed above the answer, and anything it proposes (an import, say) waits in **Decline**. Enter sends, Shift+Enter starts a new line and Esc stops the answer. -Safari has no side panel and doesn't let extensions show a sign-in page, so there **Ask Lens** opens in a tab, and you -sign in with a write API token you make in Lens (your account's API tokens). Firefox's sidebar comes with its build. +Firefox shows the panel in its sidebar (also under **View → Sidebar → Ask Lens**) and signs in the same way. Safari +has no side panel and doesn't let extensions show a sign-in page, so there **Ask Lens** opens in a tab, and you sign in +with a write API token you make in Lens (your account's API tokens). ## How it works diff --git a/docs/web-clipper.md b/docs/web-clipper.md index f5ef3a0d6..7e75d2aa6 100644 --- a/docs/web-clipper.md +++ b/docs/web-clipper.md @@ -114,7 +114,8 @@ that. The code reaches the extension API as `globalThis.browser ?? globalThis.ch ### Firefox Firefox 128 or later (desktop). `targets/firefox.json` runs the background as a script instead of a service worker -and gives the add-on its id, `lens-clipper@adeelahmad.github.io`. To try a build, open `about:debugging`, choose +and gives the add-on its id, `lens-clipper@adeelahmad.github.io`; the assistant opens in Firefox's sidebar +(`sidebar_action`) instead of a side panel. To try a build, open `about:debugging`, choose **This Firefox**, then **Load Temporary Add-on…** and pick `dist/firefox/manifest.json`; it stays until Firefox restarts. diff --git a/extensions/browser/targets/firefox.json b/extensions/browser/targets/firefox.json index f2260dab5..b1ccd3658 100644 --- a/extensions/browser/targets/firefox.json +++ b/extensions/browser/targets/firefox.json @@ -1,5 +1,13 @@ { "background": { "service_worker": null, "scripts": ["background.js"] }, + "side_panel": null, + "sidebar_action": { + "default_panel": "sidepanel.html", + "default_title": "Ask Lens", + "default_icon": { "16": "icons/icon-16.png", "32": "icons/icon-32.png" }, + "open_at_install": false + }, + "permissions": ["activeTab", "scripting", "contextMenus", "storage", "identity"], "browser_specific_settings": { "gecko": { "id": "lens-clipper@adeelahmad.github.io",