From c6e2310ce772ee7ee53b80d91b118f1351469df3 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 10 Oct 2026 21:39:10 +0000 Subject: [PATCH 1/2] feat(clipper): let the assistant use the browser from the side panel, on sites you allow While the side panel is open, it connects this browser to Lens and carries out the assistant's browser actions in its window: reading a page with its links, buttons and fields numbered (names and roles from dom-accessibility-api), clicking, typing, keys and scrolling, opening addresses, and downloading files (or the page as Markdown) into the conversation as held uploads. A site is used only once the person allows it in the panel; a consequential click, submit or Enter waits for their yes; password, card and file fields are never typed into or read; each action shows as it happens. Allowed sites are listed and removable in the options. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ --- CHANGELOG.md | 4 +- DEVLOG.md | 7 +- docs/browser-assistant.md | 39 +- extensions/browser/README.md | 4 +- extensions/browser/edge/smoke.mjs | 90 ++++- extensions/browser/package-lock.json | 7 + extensions/browser/package.json | 1 + extensions/browser/scripts/build.mjs | 1 + extensions/browser/src/agent.js | 54 +++ extensions/browser/src/core/act.js | 152 ++++++++ extensions/browser/src/core/browser.js | 438 +++++++++++++++++++++++ extensions/browser/src/core/snapshot.js | 186 ++++++++++ extensions/browser/src/ui/options.html | 8 + extensions/browser/src/ui/options.js | 35 +- extensions/browser/src/ui/sidepanel.html | 10 + extensions/browser/src/ui/sidepanel.js | 146 +++++++- extensions/browser/test/act.test.js | 83 +++++ extensions/browser/test/browser.test.js | 84 +++++ extensions/browser/test/snapshot.test.js | 84 +++++ threat-model.json | 8 + threat-model.md | 1 + threat-model.yaml | 10 +- 22 files changed, 1438 insertions(+), 14 deletions(-) create mode 100644 extensions/browser/src/agent.js create mode 100644 extensions/browser/src/core/act.js create mode 100644 extensions/browser/src/core/browser.js create mode 100644 extensions/browser/src/core/snapshot.js create mode 100644 extensions/browser/test/act.test.js create mode 100644 extensions/browser/test/browser.test.js create mode 100644 extensions/browser/test/snapshot.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 84f6d93f4..421944252 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,9 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version - **Ask Lens from your browser.** The web clipper extension has the Lens assistant in a side panel: **Ask Lens** in its popup or on the right-click menu. It signs in with one click through Lens's own sign-in page (OAuth, nothing to copy), and the conversations are the same ones Lens's chat shows, with cited answers, the steps taken and approvals. - Letting the assistant use the browser itself is in progress ([The assistant in your browser](docs/browser-assistant.md)). + While the panel is open, the assistant can use that browser: open, read, click, type and download files into the + conversation, only on sites you allow, asking before anything that sends, deletes, buys or posts, never typing + passwords or card numbers, and showing each action as it happens ([The assistant in your browser](docs/browser-assistant.md)). ## 0.7.0 October 10, 2026 {id="0.7.0"} diff --git a/DEVLOG.md b/DEVLOG.md index fb9b74fb2..6ecf37ca6 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -19,13 +19,14 @@ PRs: - [x] Server: sessions, actions, the browser tools, `ai.browser` and `ai.browser_steps`, the API for agents - [x] Extension: side panel chat, OAuth sign-in (the Edge smoke test signs in through a stand-in consent page) -- [ ] Extension: browser control (read with refs, click, type, keys, download into the conversation), site - permissions, asking first, the action log +- [x] Extension: browser control (read with refs, click, type, keys, download into the conversation), site + permissions, asking first, the action log (the Edge smoke test drives each against a stand-in Lens) Refine later: the browser from WhatsApp/Slack while the side panel is open (a setting, off); keep-alives in the chat stream while an action waits; browser tools over MCP; real input events (Chrome's debugger) for sites that ignore synthetic ones; asking about the page or selection from the right-click menu; the model and namespaces picker in the -side panel. +side panel; webmail attachments behind a download button, and files redirected to another site; frames; Safari's +default tab. ## 2026-10-10 · Similar content diff --git a/docs/browser-assistant.md b/docs/browser-assistant.md index b1c83b10a..5cadb9f02 100644 --- a/docs/browser-assistant.md +++ b/docs/browser-assistant.md @@ -1,8 +1,9 @@ # The assistant in your browser -**Status: in progress.** Built: the server side (sessions, the assistant's browser tools and the API) and the Lens -extension's side panel, where you sign in and chat with the assistant. In progress: the side panel carrying out the -assistant's browser actions, with the sites you allow, your yes for anything consequential and the action log. +**Status: built** in the Lens extension for Chrome and Edge: the side panel, where you sign in and chat with the +assistant, and the assistant using that browser on the sites you allow. Safari runs it in a tab; Firefox's sidebar +comes with its build. Webmail attachments behind a download button rather than a link are not handled yet (Refine +later). Ask the assistant from your browser, and let it do things there for you: "check my email and import the important attachments", "find last month's invoice on the supplier's site and put it in Accounts", "save this page". It opens @@ -35,6 +36,32 @@ Firefox shows the panel in its sidebar (also under **View → Sidebar → Ask Le has no side panel and doesn't let extensions show a sign-in page, so there **Ask Lens** opens in a tab, and you sign in with a write API token you make in Lens (your account's API tokens). +## Letting it use your browser + +While the side panel is open, **Let Lens use this browser** (under the question box, on unless you turn it off) lets +the assistant work in that window when you ask it to: + +- **A site is used only once you allow it.** The first time the assistant wants to read or act on a site, or open an + address there, the panel asks **Let Lens read and act on mail.example.com?** Allow, and the browser may ask too, for + the extension's access to that site. Allowed sites are listed in the extension's options, under The assistant, where + you remove them again. +- **Anything consequential waits for your yes.** A click or a submit that sends, deletes, buys, books, posts, shares + or changes an account shows what it is about to do (**Click “Send” on mail.example.com?**) with **Yes, do it** and + **No**. It asks when the assistant marks the action consequential, or when the button's words say so (send, delete, + pay, order, post, submit, confirm, unsubscribe...), or when the form posts somewhere. No answer within 80 seconds + counts as No. +- **Never passwords or card numbers.** It doesn't type into password fields, card number or security code fields, + or file pickers, and doesn't see what's in them: you type those. +- **You see each action as it happens**, in the answer it's for: opening, reading, clicking, typing, downloading, + with ✓ when done and ✗ when not (and why). An app using your browser through the API shows the same way. +- **Pages are third-party text.** What a page or an email says reaches the assistant marked as material to read, never + as instructions; opening an address on a site you haven't allowed asks first, so what it read can't be carried + off in a link. + +Files the assistant downloads (an attachment, a PDF behind a sign-in, or the page itself as Markdown) are attached to +the conversation like a file you dropped in, and it puts them in a namespace with `import_files`, which waits for +your **Approve**. + ## How it works The Lens extension's side panel signs in to your Lens and connects a **session**. While the side panel is open, a @@ -82,6 +109,12 @@ A question sent with `browser` (the session's id) on `POST /api/v1/chats/{id}/me ## Refine later +- Webmail attachments behind a download button (no link to read), and files a site hands over from another site + after a redirect, which need that site allowed too. +- Pages inside frames (other than the page's own shadow trees), and real input events through the browser's + debugger for pages that ignore synthetic ones. +- Safari, where the panel is a tab of its own: the assistant picks the page's tab from `browser_tabs` today; it + should default to the tab you were on before. - Ask about the page you're on, or the text you selected, straight from the right-click menu. - Pick the model, or the namespaces to look in, in the side panel (the web app's chat has both). - The assistant's browser for conversations elsewhere (WhatsApp, Slack, Telegram) while the side panel is open, as diff --git a/extensions/browser/README.md b/extensions/browser/README.md index 3ad1c2c7f..f8ebe2986 100644 --- a/extensions/browser/README.md +++ b/extensions/browser/README.md @@ -9,6 +9,8 @@ npm ci && npm test && npm run build # then Load unpacked: dist/chrome - `src/core/`: the shared code every browser uses: `clip.js` (page, selection, element), `metadata.js`, `markdown.js` (Turndown), `template.js`, `picker.js`, `lens.js` (pushing to Lens), `settings.js`; for the assistant's - side panel, `account.js` (signing in with OAuth) and `assistant.js` (the chat API, answers as safe HTML) + side panel, `account.js` (signing in with OAuth) and `assistant.js` (the chat API, answers as safe HTML); for the + assistant using the browser, `browser.js` (the session and its actions, sites and asks), `snapshot.js` (a page read + with numbered elements) and `act.js` (clicking, typing, keys), the last two injected as `src/agent.js` - `src/background.js`, `src/content.js`, `src/ui/`: the extension around it - `targets/.json`: what each browser's manifest changes from `src/manifest.json` diff --git a/extensions/browser/edge/smoke.mjs b/extensions/browser/edge/smoke.mjs index 16b40ca42..70bf85383 100644 --- a/extensions/browser/edge/smoke.mjs +++ b/extensions/browser/edge/smoke.mjs @@ -1,6 +1,8 @@ // Edge smoke test: loads dist/edge (npm run build in ..) into Microsoft Edge, opens the popup and options, clips an // article page and checks a stand-in Lens import webhook got its Markdown. Then it signs the side panel in to the -// stand-in Lens through its OAuth consent page, asks the assistant and approves what it proposes. Run under a display +// stand-in Lens through its OAuth consent page, asks the assistant and approves what it proposes, and has the stand-in +// Lens send the side panel browser actions: reading a page once its site is allowed, typing, a click that waits for +// the person's yes, refusing a password field, and downloading a file into a held upload. Run under a display // (xvfb-run in CI). // EDGE_CHANNEL=chromium runs the same test in Playwright's Chromium, for machines without Edge. import { createHash } from "node:crypto"; @@ -61,6 +63,23 @@ const server = createServer((req, res) => { res.writeHead(404, cors).end(); }); +// the browser, as Lens uses it: a session, its stream of actions, and their answers +const browserSide = { label: null, stream: null, answers: new Map(), waiting: new Map(), ended: false, uploaded: null, next: 1 }; +function act(tool, args = {}, chat = null) { + const id = `a${browserSide.next++}`; + const done = new Promise((resolve, reject) => { + browserSide.waiting.set(id, resolve); + setTimeout(() => reject(new Error(`no answer to ${tool} ${id}`)), 30000); + }); + browserSide.stream.write(`event: action\ndata: ${JSON.stringify({ id, tool, args, chat })}\n\n`); + return done; +} +const COMPOSE = `Compose

New message

+ + +

Draft

Attached: report.pdf

`; +const PDF = "%PDF-1.4\n% a stand-in report\n%%EOF\n"; + function lens(req, res) { const url = new URL(req.url, origin); const reply = (status, body, headers = {}) => res.writeHead(status, { "Content-Type": "application/json", ...headers }).end(JSON.stringify(body)); @@ -108,6 +127,42 @@ function lens(req, res) { decided.push(JSON.parse(body)); reply(200, { status: "done" }); }); + } else if (url.pathname === "/compose") { + res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" }).end(COMPOSE); + } else if (url.pathname === "/files/report.pdf") { + res.writeHead(200, { "Content-Type": "application/pdf", "Content-Disposition": 'attachment; filename="report.pdf"' }).end(PDF); + } else if (url.pathname === "/api/v1/browser/sessions" && req.method === "POST") { + read((body) => { + browserSide.label = JSON.parse(body || "{}").label; + reply(201, { id: "s1", label: browserSide.label, live: true }); + }); + } else if (url.pathname === "/api/v1/browser/sessions/s1/actions") { + res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-store" }); + res.write(": open\n\n"); + browserSide.stream = res; + req.on("close", () => browserSide.stream === res && (browserSide.stream = null)); + } else if (url.pathname.startsWith("/api/v1/browser/actions/") && req.method === "POST") { + read((body) => { + const id = url.pathname.split("/").pop(); + browserSide.answers.set(id, JSON.parse(body)); + browserSide.waiting.get(id)?.(JSON.parse(body)); + reply(200, { ok: true }); + }); + } else if (url.pathname === "/api/v1/browser/sessions/s1" && req.method === "DELETE") { + browserSide.ended = true; + reply(200, { ok: true }); + } else if (url.pathname === "/api/v1/uploads/limits") { + reply(200, { max_mb: 10, chunk_mb: 8, extensions: [".pdf"] }); + } else if (url.pathname === "/api/v1/uploads" && req.method === "POST") { + read((body) => { + const b = JSON.parse(body); + reply(201, { id: "up1", filename: b.filename, size: b.size, offset: 0, state: "receiving", hold: b.hold }); + }); + } else if (url.pathname === "/api/v1/uploads/up1" && req.method === "PUT") { + read((body) => { + browserSide.uploaded = body; + reply(200, { id: "up1", filename: "report.pdf", size: body.length, offset: body.length, state: "held" }); + }); } else { return false; } @@ -209,6 +264,39 @@ try { await panel.click(".approval button.primary"); await panel.waitForFunction(() => document.querySelector(".approval")?.textContent.includes("Done"), null, { timeout: 5000 }).catch(() => null); check(decided.at(-1)?.decision === "approve", "approving sends the person's pick to Lens"); + + // the assistant using this browser: Lens sends actions, the side panel carries them out on sites the person allows + await panel.waitForFunction(() => document.getElementById("browserState").textContent.includes("can use"), null, { timeout: 10000 }).catch(() => null); + check(Boolean(browserSide.stream) && /\S/.test(browserSide.label ?? ""), `the side panel connects this browser to Lens (${browserSide.label})`); + const compose = await context.newPage(); + await compose.goto(`${origin}/compose`); + const tab_id = await panel.evaluate(async (url) => (await chrome.tabs.query({ url })).at(0)?.id, `${origin}/compose`); + const reading = act("read", { tab_id }); + await panel.locator(".ask.site button.primary").click({ timeout: 10000 }); + let a = await reading; + check(a.ok && a.result.text.includes("New message"), `Lens reads a page once the person allows its site${a.ok ? "" : `: ${a.error}`}`); + const refOf = (name) => new RegExp(`\\[(e\\d+)\\] [a-z ]+"${name}"`).exec(a.result?.text ?? "")?.[1]; + const [to, pw, send, file] = ["To", "Password", "Send", "report.pdf"].map(refOf); + a = await act("type", { tab_id, ref: to, text: "ada@example.com" }); + check(a.ok && (await compose.inputValue("#to")) === "ada@example.com", `it types into a field${a.ok ? "" : `: ${a.error}`}`); + a = await act("type", { tab_id, ref: pw, text: "hunter2" }); + check(!a.ok && /password/.test(a.error) && (await compose.inputValue("#pw")) === "", "and never into a password field"); + const declined = act("click", { tab_id, ref: send }); + await panel.locator(".ask.action").last().getByRole("button", { name: "No" }).click({ timeout: 10000 }); + a = await declined; + check(!a.ok && /said no/.test(a.error) && (await compose.textContent("#state")) === "Draft", "a consequential click waits for the person, and No means it doesn't happen"); + const sending = act("click", { tab_id, ref: send }); + await panel.locator(".ask.action").last().getByRole("button", { name: "Yes, do it" }).click({ timeout: 10000 }); + a = await sending; + check(a.ok && (await compose.textContent("#state")) === "Sent to ada@example.com", `and Yes does it${a.ok ? "" : `: ${a.error}`}`); + check(a.result?.text?.includes("Sent to ada@example.com"), "the answer has the page as it is after the click"); + a = await act("import", { tab_id, ref: file }); + check(a.ok && a.result.upload_id === "up1" && browserSide.uploaded === PDF, `it downloads a file into a held upload${a.ok ? "" : `: ${a.error}`}`); + const logged = await panel.locator(".steps li").allTextContents(); + check(logged.some((t) => t.startsWith("Clicked “Send”")) && logged.some((t) => t.startsWith("Downloaded report.pdf")), "each action shows in the side panel"); + await panel.close({ runBeforeUnload: true }); + for (let i = 0; i < 20 && !browserSide.ended; i++) await new Promise((ok) => setTimeout(ok, 100)); + check(browserSide.ended, "closing the side panel ends the session"); check(errors.length === 0, `no page errors${errors.length ? `: ${errors.join("; ")}` : ""}`); } finally { await context.close(); diff --git a/extensions/browser/package-lock.json b/extensions/browser/package-lock.json index 0cce984a4..0f2a70da7 100644 --- a/extensions/browser/package-lock.json +++ b/extensions/browser/package-lock.json @@ -10,6 +10,7 @@ "license": "MIT", "dependencies": { "@mozilla/readability": "0.6.0", + "dom-accessibility-api": "0.7.1", "dompurify": "3.4.16", "eventsource-parser": "4.1.1", "marked": "18.0.14", @@ -677,6 +678,12 @@ "dev": true, "license": "MIT" }, + "node_modules/dom-accessibility-api": { + "version": "0.7.1", + "resolved": "https://registry.npmjs.org/dom-accessibility-api/-/dom-accessibility-api-0.7.1.tgz", + "integrity": "sha512-vdnCeZD+3wZ+8h8xXL/ZtBlvvoobOFyPzSiIfO6sGOZDqjFx4aLMAjZhl4rawj5xYz3UwP6Tgvyh0iH4IOCVnQ==", + "license": "MIT" + }, "node_modules/dompurify": { "version": "3.4.16", "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.16.tgz", diff --git a/extensions/browser/package.json b/extensions/browser/package.json index 243ea7aff..5e0d10316 100644 --- a/extensions/browser/package.json +++ b/extensions/browser/package.json @@ -12,6 +12,7 @@ }, "dependencies": { "@mozilla/readability": "0.6.0", + "dom-accessibility-api": "0.7.1", "dompurify": "3.4.16", "eventsource-parser": "4.1.1", "marked": "18.0.14", diff --git a/extensions/browser/scripts/build.mjs b/extensions/browser/scripts/build.mjs index eb6f5576f..bab704148 100644 --- a/extensions/browser/scripts/build.mjs +++ b/extensions/browser/scripts/build.mjs @@ -50,6 +50,7 @@ for (const target of targets) { entryPoints: { background: join(root, "src/background.js"), content: join(root, "src/content.js"), + agent: join(root, "src/agent.js"), popup: join(root, "src/ui/popup.js"), options: join(root, "src/ui/options.js"), sidepanel: join(root, "src/ui/sidepanel.js"), diff --git a/extensions/browser/src/agent.js b/extensions/browser/src/agent.js new file mode 100644 index 000000000..9249f647f --- /dev/null +++ b/extensions/browser/src/agent.js @@ -0,0 +1,54 @@ +// Injected into a tab the assistant reads or acts on (only sites the person allowed). The side panel calls it through +// scripting.executeScript; it keeps the numbers of what it has read in the extension's own world, for the life of +// the page, and nothing else. +import * as act from "./core/act.js"; +import { describe, element, newRefs, read, secret } from "./core/snapshot.js"; + +if (!globalThis.__lensAgent) { + const refs = newRefs(); + const find = (ref) => { + const el = element(refs, ref); + if (!el) throw new Error(`there's no ${ref} on this page now: read it again for current refs`); + return el; + }; + const answer = (fn) => { + try { + return { ok: true, result: fn() }; + } catch (e) { + return { ok: false, error: e.message }; + } + }; + globalThis.__lensAgent = { + read: (opts) => answer(() => read(document, refs, opts)), + /** What a ref is, and whether acting on it looks consequential, for the side panel to decide what to ask. */ + inspect: (ref, what = "click") => + answer(() => { + const el = find(ref); + return { + ...describe(el), + site: location.origin, + secret: secret(el), + refused: what === "click" ? null : act.refuseTyping(el), + consequential: act.looksConsequential(el, what), + }; + }), + click: (ref) => answer(() => act.click(find(ref))), + type: (ref, text, submit) => + answer(() => { + const el = find(ref); + const why = act.refuseTyping(el); + if (why) throw new Error(why); + const now = act.type(el, text); + if (submit) act.submit(el); + return now; + }), + /** What has the focus, and whether Enter there looks consequential. */ + focused: () => + answer(() => { + const el = document.activeElement; + if (!el || el === document.body) return null; + return { ...describe(el), consequential: act.looksConsequential(el, "submit") }; + }), + press: (key) => answer(() => act.press(document, key)), + }; +} diff --git a/extensions/browser/src/core/act.js b/extensions/browser/src/core/act.js new file mode 100644 index 000000000..21420c3e5 --- /dev/null +++ b/extensions/browser/src/core/act.js @@ -0,0 +1,152 @@ +// Acting on a page for the assistant (browser_click, browser_type, browser_press), in the page, as a person would: +// the pointer and key events a click or a key sends, and a field's value set the way frameworks notice. It never +// fills a password or card field, and says which actions look consequential, for the side panel to ask about first. +import { describe, secret } from "./snapshot.js"; + +// what sends, deletes, buys, books, posts, shares or changes an account, by the words on the control +const CONSEQUENTIAL = new RegExp( + "\\b(send|delete|remove|trash|discard|archive|erase|pay|buy|purchase|place (?:the |my |your )?order|order now|checkout|" + + "check out|book|reserve|subscribe|unsubscribe|post|publish|share|tweet|submit|confirm|transfer|donate|sign up|" + + "register|accept|approve|decline|deactivate|uninstall|cancel (?:my |the |your )?(?:subscription|order|account|booking|plan)|" + + "close (?:my |the |your )?account)\\b", + "i", +); +const SCROLL_KEYS = new Set(["PageDown", "PageUp", "Home", "End"]); +const KEYS = { + Enter: 13, Escape: 27, Tab: 9, Backspace: 8, Delete: 46, Space: 32, + ArrowDown: 40, ArrowUp: 38, ArrowLeft: 37, ArrowRight: 39, +}; // prettier-ignore + +/** Whether doing `what` to `el` looks consequential from its words (or its form's), whatever the model said. */ +export function looksConsequential(el, what = "click") { + const d = describe(el); + const words = [d.name, el.getAttribute?.("title"), el.value && /^(submit|button)$/.test(el.type) ? el.value : ""].join(" "); + if (CONSEQUENTIAL.test(words)) return true; + if (what === "submit" || (what === "click" && el.type === "submit")) { + const form = el.form ?? el.closest?.("form"); + if (!form || form.getAttribute("role") === "search") return false; + const buttons = [...form.querySelectorAll("button, input[type=submit]")].map((b) => describe(b).name).join(" "); + return CONSEQUENTIAL.test(buttons) || (form.method ?? "").toLowerCase() === "post"; + } + return false; +} + +/** Why the assistant may not type into `el`, or null. */ +export function refuseTyping(el) { + const why = secret(el); + if (why) return `that's ${why}: the person types those themselves`; + if (el.disabled || el.readOnly) return "that field can't be changed"; + if (el.tagName === "INPUT" && /^(checkbox|radio)$/.test(el.type)) return "that's a checkbox: click it instead"; + if (el.tagName === "INPUT" && el.type === "file") return "that picks a file from the computer, which the person does"; + const typable = el.tagName === "TEXTAREA" || el.tagName === "SELECT" || el.tagName === "INPUT" || el.isContentEditable; + if (!typable) return "that isn't a field: read the page again and use a field's ref"; + return null; +} + +function point(el) { + el.scrollIntoView?.({ block: "center", inline: "center" }); + const r = el.getBoundingClientRect?.() ?? { left: 0, top: 0, width: 0, height: 0 }; + return { clientX: r.left + r.width / 2, clientY: r.top + r.height / 2 }; +} + +/** Click as a person does: pointer and mouse down and up, focus, then the click. */ +export function click(el) { + const win = el.ownerDocument.defaultView; + const at = { bubbles: true, cancelable: true, composed: true, button: 0, view: win, ...point(el) }; + const Pointer = win.PointerEvent ?? win.MouseEvent; + el.dispatchEvent(new Pointer("pointerdown", { ...at, pointerType: "mouse", isPrimary: true })); + el.dispatchEvent(new win.MouseEvent("mousedown", at)); + el.focus?.({ preventScroll: true }); + el.dispatchEvent(new Pointer("pointerup", { ...at, pointerType: "mouse", isPrimary: true })); + el.dispatchEvent(new win.MouseEvent("mouseup", at)); + el.click(); +} + +function keyEvent(win, type, key) { + const code = key === "Space" ? "Space" : key.length === 1 ? `Key${key.toUpperCase()}` : key; + const keyCode = KEYS[key] ?? 0; + return new win.KeyboardEvent(type, { key: key === "Space" ? " " : key, code, keyCode, which: keyCode, bubbles: true, cancelable: true, composed: true }); +} + +/** Press a key where the focus is (or on `el`); Enter in a form's field submits it when the page doesn't. */ +function sendKey(doc, key, el = doc.activeElement ?? doc.body) { + const win = doc.defaultView; + const down = keyEvent(win, "keydown", key); + const go = el.dispatchEvent(down); + if (key === "Enter") el.dispatchEvent(keyEvent(win, "keypress", key)); + el.dispatchEvent(keyEvent(win, "keyup", key)); + if (!go) return; + if (key === "Enter" && el.tagName === "INPUT" && el.form) el.form.requestSubmit(); + else if (key === "Tab") { + const all = [...doc.querySelectorAll("a[href], button, input, select, textarea, [tabindex]:not([tabindex='-1'])")].filter((e) => !e.disabled); + all[(all.indexOf(el) + 1) % all.length]?.focus(); + } +} + +/** Set a field's value the way a person typing does, so the page's own code sees it: the text, or the option of a + * list whose label (or value) matches. Returns what it now holds. */ +export function type(el, text) { + const win = el.ownerDocument.defaultView; + point(el); + el.focus?.({ preventScroll: true }); + if (el.tagName === "SELECT") { + const want = String(text).trim().toLowerCase(); + const options = [...el.options]; + const option = + options.find((o) => o.label.trim().toLowerCase() === want || o.value.toLowerCase() === want) ?? + options.find((o) => o.label.toLowerCase().includes(want)); + if (!option) throw new Error(`no option ${JSON.stringify(text)}; it has ${options.map((o) => o.label.trim()).slice(0, 25).join(" | ")}`); + el.value = option.value; + el.dispatchEvent(new win.Event("input", { bubbles: true })); + el.dispatchEvent(new win.Event("change", { bubbles: true })); + return option.label.trim(); + } + if (el.isContentEditable) { + const doc = el.ownerDocument; + doc.getSelection()?.selectAllChildren(el); + // insertText keeps the page's undo and its editors in step; where it isn't there, the text is set + if (!doc.execCommand?.("insertText", false, text)) { + el.textContent = text; + el.dispatchEvent(new win.InputEvent("input", { bubbles: true, inputType: "insertText", data: text })); + } + return el.textContent; + } + // the element's own value setter, not the page's: frameworks that watch the property then see a change + const proto = el.tagName === "TEXTAREA" ? win.HTMLTextAreaElement.prototype : win.HTMLInputElement.prototype; + Object.getOwnPropertyDescriptor(proto, "value").set.call(el, text); + el.dispatchEvent(new win.InputEvent("input", { bubbles: true, inputType: "insertText", data: text })); + el.dispatchEvent(new win.Event("change", { bubbles: true })); + return el.value; +} + +/** Press Enter in `el`, after typing: submits its form unless the page handles the key itself. */ +export function submit(el) { + sendKey(el.ownerDocument, "Enter", el); +} + +function scroller(doc) { + // the focused thing's nearest scrolling box (a mail list, say), else the page + for (let n = doc.activeElement; n && n !== doc.body; n = n.parentElement) { + const s = doc.defaultView.getComputedStyle(n); + if (/(auto|scroll)/.test(s.overflowY) && n.scrollHeight > n.clientHeight) return n; + } + return doc.scrollingElement ?? doc.documentElement; +} + +/** Press a key (Enter, Escape, Tab, arrows...) or scroll (PageDown, PageUp, Home, End). */ +export function press(doc, key) { + key = String(key ?? "").trim(); + if (SCROLL_KEYS.has(key)) { + const box = scroller(doc); + const page = Math.max(200, (box.clientHeight || doc.defaultView.innerHeight) * 0.85); + const top = { PageDown: box.scrollTop + page, PageUp: box.scrollTop - page, Home: 0, End: box.scrollHeight }[key]; + if (typeof box.scrollTo === "function") box.scrollTo({ top }); + else box.scrollTop = top; + return `Scrolled ${{ PageDown: "down", PageUp: "up", Home: "to the top", End: "to the end" }[key]}`; + } + if (!(key in KEYS) && !/^[a-z0-9]$/i.test(key)) { + throw new Error(`no key ${JSON.stringify(key)}: press Enter, Escape, Tab, Backspace, Delete, Space, an arrow, or scroll with PageDown, PageUp, Home or End`); + } + sendKey(doc, key); + return `Pressed ${key}`; +} diff --git a/extensions/browser/src/core/browser.js b/extensions/browser/src/core/browser.js new file mode 100644 index 000000000..78ca1f2a2 --- /dev/null +++ b/extensions/browser/src/core/browser.js @@ -0,0 +1,438 @@ +// The side panel's half of the assistant using this browser (docs/browser-assistant.md): a session with Lens, the +// actions Lens sends it, carried out in this window's tabs, and what happened, sent back. What it may do is the +// person's to say: only sites they allowed, their yes before anything consequential, never a password or card field. +// Each action shows in the side panel as it happens. +import { createParser } from "eventsource-parser"; +import { api, apiJson, SignedOut } from "./account.js"; +import * as settingsStore from "./settings.js"; + +const ext = globalThis.browser ?? globalThis.chrome; +export const SITES = "agentSites"; // origins the person let the assistant read and act on +export const ENABLED = "agentEnabled"; // false: the side panel doesn't offer this browser to the assistant +const LOAD_MS = 20000; // the longest a page is waited for +const SETTLE_MS = 700; // after a click or a key, for the page to react +const AFTER_CHARS = 6000; // the page as it is after an action +const EXT_TYPES = { + "application/pdf": "pdf", "text/plain": "txt", "text/markdown": "md", "text/html": "html", "message/rfc822": "eml", + "application/json": "json", "application/zip": "zip", "image/png": "png", "image/jpeg": "jpg", "image/gif": "gif", + "image/webp": "webp", "image/tiff": "tiff", "audio/mpeg": "mp3", "audio/mp4": "m4a", "audio/wav": "wav", + "video/mp4": "mp4", "video/webm": "webm", "application/msword": "doc", "application/rtf": "rtf", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document": "docx", + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": "xlsx", + "application/vnd.openxmlformats-officedocument.presentationml.presentation": "pptx", + "application/vnd.ms-excel": "xls", "application/vnd.ms-powerpoint": "ppt", + "application/vnd.oasis.opendocument.text": "odt", +}; // prettier-ignore + +const sleep = (ms, signal) => + new Promise((ok) => { + const t = setTimeout(ok, ms); + signal?.addEventListener("abort", () => (clearTimeout(t), ok()), { once: true }); + }); +const host = (url) => { + try { + return new URL(url).host; + } catch { + return ""; + } +}; +const shorten = (s, n = 60) => (s.length > n ? `${s.slice(0, n - 1)}…` : s); + +// ---------- what the person allowed ---------- +export async function enabled() { + return (await ext.storage.local.get(ENABLED))[ENABLED] !== false; +} + +export async function setEnabled(on) { + await ext.storage.local.set({ [ENABLED]: Boolean(on) }); +} + +export async function sites() { + return (await ext.storage.local.get(SITES))[SITES] ?? []; +} + +export async function allowSite(origin) { + const all = await sites(); + if (!all.includes(origin)) await ext.storage.local.set({ [SITES]: [...all, origin].sort() }); +} + +export async function forgetSite(origin) { + await ext.storage.local.set({ [SITES]: (await sites()).filter((s) => s !== origin) }); +} + +// ---------- the session ---------- +/** What to call this browser in Lens, e.g. "Google Chrome on macOS". */ +export function label(nav = globalThis.navigator) { + const brands = (nav?.userAgentData?.brands ?? []).map((b) => b.brand).filter((b) => !/not.?a.?brand/i.test(b)); + const ua = nav?.userAgent ?? ""; + const name = + brands.find((b) => b !== "Chromium") ?? + brands[0] ?? + [ + [/Edg\//, "Microsoft Edge"], + [/Firefox\//, "Firefox"], + [/Chrome\//, "Chrome"], + [/Safari\//, "Safari"], + ].find(([re]) => re.test(ua))?.[1] ?? + "Browser"; + const os = nav?.userAgentData?.platform || ""; + return os ? `${name} on ${os}` : name; +} + +export const connect = () => + apiJson("/browser/sessions", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ label: label() }) }); + +/** The side panel is closing: Lens stops offering this browser. Sent with the token in hand, so it goes out even as + * the page does. */ +export function end(sid, account, fetchImpl = (...a) => fetch(...a)) { + return fetchImpl(`${account.server}/api/v1/browser/sessions/${sid}`, { + method: "DELETE", + headers: { Authorization: `Bearer ${account.accessToken}` }, + credentials: "omit", + keepalive: true, + }).catch(() => {}); +} + +export const answer = (aid, out) => + apiJson(`/browser/actions/${aid}`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(out) }); + +/** + * Listen for the actions Lens sends this session, one `onAction(action)` at a time, until `signal` aborts. A dropped + * stream is opened again; `onGone(status)` when Lens no longer has the session (404) or turned this off (403). + */ +export async function listen(sid, onAction, { signal, onGone = () => {}, onLive = () => {} } = {}) { + let wait = 1000; + let chain = Promise.resolve(); + while (!signal?.aborted) { + try { + const res = await api(`/browser/sessions/${sid}/actions`, { headers: { Accept: "text/event-stream" }, signal }); + if (res.status === 404 || res.status === 403) return onGone(res.status); + if (!res.ok) throw new Error(`Lens answered ${res.status}`); + onLive(true); + wait = 1000; + const parser = createParser({ + onEvent(e) { + if (e.event !== "action") return; + const action = JSON.parse(e.data); + chain = chain.then(() => onAction(action)).catch(() => {}); + }, + }); + const reader = res.body.getReader(); + const text = new TextDecoder(); + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + parser.feed(text.decode(value, { stream: true })); + } + } catch (e) { + if (signal?.aborted) return; + if (e instanceof SignedOut) throw e; + } + onLive(false); + await sleep(wait, signal); + wait = Math.min(wait * 2, 15000); + } +} + +// ---------- carrying actions out ---------- +/** + * Carries out Lens's actions in one browser window. `ui.ask({kind, text, detail, origins}, action)` asks the person + * and resolves to their yes or no (for a site, it asks the browser for access to `origins` in the person's own click); + * `ui.log(action, text)` shows an action starting and returns `done(text, ok)`. + */ +export class Hands { + constructor({ windowId, ui, fetchImpl = (...a) => fetch(...a) }) { + this.windowId = windowId; + this.ui = ui; + this.fetch = fetchImpl; + } + + /** Carry out `action` and answer Lens. */ + async run(action) { + this.action = action; + const step = this.ui.log(action, this.starting(action)); + let out; + try { + const tool = this[`t_${action.tool}`]; + if (typeof tool !== "function") throw new Error(`this browser can't ${action.tool} yet: update the Lens extension`); + const result = await tool.call(this, { ...(action.args ?? {}) }); + out = { ok: true, result }; + step.done(result?.said ?? this.starting(action), true); + } catch (e) { + out = { ok: false, error: e.message }; + step.done(e.message, false); + } + this.action = null; + try { + await answer(action.id, out); + } catch { + // given up on in Lens meanwhile (stopped, or too long): nothing to answer + } + return out; + } + + starting({ tool, args = {} }) { + return ( + { + tabs: "Looking at the tabs", + open: `Opening ${host(args.url) || "a page"}`, + read: args.find ? `Reading the page for “${shorten(args.find, 40)}”` : "Reading the page", + click: `Clicking ${args.ref}`, + type: `Typing into ${args.ref}`, + press: `Pressing ${args.key}`, + import: args.page ? "Saving the page" : "Downloading a file", + }[tool] ?? tool + ); + } + + // ---- tabs and sites ---- + async tab(tabId) { + const tab = tabId != null ? await ext.tabs.get(Number(tabId)).catch(() => null) : (await ext.tabs.query({ active: true, windowId: this.windowId }))[0]; + if (!tab) throw new Error(tabId != null ? `there's no tab ${tabId}: see browser_tabs` : "there's no tab open"); + if (tab.url?.startsWith(ext.runtime.getURL(""))) throw new Error("the person is looking at Lens's own panel: say which tab (browser_tabs)"); + return tab; + } + + origin(tab) { + if (!tab.url) { + throw new Error( + "Lens can't see which site that tab is on: open the address with browser_open, or the person opens the side panel from that tab (Ask Lens)", + ); + } + const u = new URL(tab.url); + if (!/^https?:$/.test(u.protocol)) throw new Error("the browser doesn't let extensions use that page"); + return u.origin; + } + + async allowed(origin) { + const granted = ext.permissions?.contains ? await ext.permissions.contains({ origins: [`${origin}/*`] }) : true; + return granted && (await sites()).includes(origin); + } + + /** Make sure the person lets the assistant use `origin`: asked once per site, then remembered. */ + async site(origin) { + if (await this.allowed(origin)) return; + const yes = await this.ui.ask( + { + kind: "site", + text: `Let Lens read and act on ${new URL(origin).host}?`, + detail: "Only when you ask it to, while this panel is open. Anything that sends, deletes, buys or posts still asks you first.", + origins: [`${origin}/*`], + }, + this.action, + ); + if (!yes) throw new Error(`the person didn't let Lens use ${new URL(origin).host}`); + await allowSite(origin); + } + + async call(tabId, name, ...args) { + await ext.scripting.executeScript({ target: { tabId }, files: ["agent.js"] }); + const [first] = await ext.scripting.executeScript({ + target: { tabId }, + func: (n, a) => globalThis.__lensAgent[n](...a), + args: [name, args], + }); + const out = first?.result; + if (!out?.ok) throw new Error(out?.error ?? "the page didn't answer"); + return out.result; + } + + async loaded(tabId) { + const until = Date.now() + LOAD_MS; + for (;;) { + const tab = await ext.tabs.get(tabId); + if (tab.status === "complete" || Date.now() > until) return tab; + await sleep(200); + } + } + + /** The page as it is now, after an action: read when its site is allowed. */ + async after(tabId, said, chars = AFTER_CHARS) { + await sleep(SETTLE_MS); + const tab = await this.loaded(tabId); + const origin = tab.url ? new URL(tab.url).origin : null; + if (!origin || !(await this.allowed(origin))) { + return { said, tab_id: tabId, url: tab.url ?? null, note: "the tab is now on a site Lens hasn't been allowed: browser_read asks the person" }; + } + return { said, tab_id: tabId, ...(await this.call(tabId, "read", { max_chars: chars })) }; + } + + // ---- the tools (browser_) ---- + async t_tabs() { + const tabs = await ext.tabs.query({ windowId: this.windowId }); + const allowed = new Set(await sites()); + const out = tabs + .filter((t) => !t.url?.startsWith(ext.runtime.getURL(""))) + .map((t) => { + const seen = t.url && /^https?:/.test(t.url) ? new URL(t.url).origin : null; + if (seen && allowed.has(seen)) return { id: t.id, title: t.title, url: t.url, active: t.active }; + return { id: t.id, site: seen ? host(seen) : null, active: t.active, note: "a site Lens hasn't been allowed to read" }; + }); + return { said: `Looked at ${out.length} tab${out.length === 1 ? "" : "s"}`, tabs: out }; + } + + async t_open({ url, tab_id: tabId, new_tab: newTab }) { + let u; + try { + u = new URL(String(url ?? "")); + } catch { + throw new Error("that isn't a web address"); + } + if (!/^https?:$/.test(u.protocol)) throw new Error("only http and https addresses open"); + // asked before going there: an address can carry what was read elsewhere + await this.site(u.origin); + const tab = newTab + ? await ext.tabs.create({ url: u.href, windowId: this.windowId, active: true }) + : await ext.tabs.update((await this.tab(tabId)).id, { url: u.href, active: true }); + return this.after(tab.id, `Opened ${u.host}`); + } + + async t_read({ tab_id: tabId, find, max_chars: most }) { + const tab = await this.tab(tabId); + await this.site(this.origin(tab)); + const page = await this.call(tab.id, "read", { find, max_chars: most }); + return { said: `Read ${shorten(page.title || host(page.url), 70)}${find ? ` for “${shorten(find, 40)}”` : ""}`, tab_id: tab.id, ...page }; + } + + async confirm(text, detail) { + if (!(await this.ui.ask({ kind: "action", text, detail }, this.action))) throw new Error("the person said no"); + } + + async t_click({ ref, tab_id: tabId, consequential }) { + const tab = await this.tab(tabId); + const origin = this.origin(tab); + await this.site(origin); + const what = await this.call(tab.id, "inspect", ref, "click"); + if (what.disabled) throw new Error(`${ref} is disabled`); + const name = what.name ? `“${shorten(what.name)}”` : `the ${what.role}`; + if (consequential || what.consequential) await this.confirm(`Click ${name} on ${new URL(origin).host}?`); + await this.call(tab.id, "click", ref); + return this.after(tab.id, `Clicked ${name}`); + } + + async t_type({ ref, text, submit, tab_id: tabId, consequential }) { + const tab = await this.tab(tabId); + const origin = this.origin(tab); + await this.site(origin); + const what = await this.call(tab.id, "inspect", ref, submit ? "submit" : "type"); + if (what.refused) throw new Error(what.refused); + const name = what.name ? `“${shorten(what.name)}”` : `the ${what.role}`; + if (submit && (consequential || what.consequential)) { + await this.confirm(`Type into ${name} and submit it on ${new URL(origin).host}?`, String(text)); + } + await this.call(tab.id, "type", ref, String(text), Boolean(submit)); + return this.after(tab.id, `Typed “${shorten(String(text), 40)}” into ${name}${submit ? " and pressed Enter" : ""}`); + } + + async t_press({ key, tab_id: tabId }) { + const tab = await this.tab(tabId); + const origin = this.origin(tab); + await this.site(origin); + if (key === "Enter") { + const focus = await this.call(tab.id, "focused"); + if (focus?.consequential) await this.confirm(`Press Enter in “${shorten(focus.name || focus.role)}” on ${new URL(origin).host}?`); + } + const said = await this.call(tab.id, "press", key); + return this.after(tab.id, said); + } + + async t_import({ ref, url, tab_id: tabId, page }) { + let file; + if (page) { + const tab = await this.tab(tabId); + await this.site(this.origin(tab)); + file = await this.savePage(tab); + } else { + let address = url; + if (ref) { + const tab = await this.tab(tabId); + await this.site(this.origin(tab)); + address = (await this.call(tab.id, "inspect", ref, "click")).href; + if (!address) throw new Error(`${ref} isn't a link to a file: give the file's address, or click it and read the page it opens`); + } + let u; + try { + u = new URL(String(address ?? "")); + } catch { + throw new Error("give a link's ref, a file's url, or page: true"); + } + if (!/^https?:$/.test(u.protocol)) throw new Error("only http and https files download"); + await this.site(u.origin); + file = await this.download(u.href); + } + const up = await upload(file); + return { + said: `${page ? "Saved the page as" : "Downloaded"} ${up.filename} (${size(up.size)})`, + upload_id: up.id, + filename: up.filename, + size: up.size, + }; + } + + async savePage(tab) { + await ext.scripting.executeScript({ target: { tabId: tab.id }, files: ["content.js"] }); + const settings = await settingsStore.load(); + const [first] = await ext.scripting.executeScript({ + target: { tabId: tab.id }, + func: (args) => globalThis.__lensClipper.run(args), + args: [{ mode: "page", settings, tags: settings.tags }], + }); + const out = first?.result; + if (!out?.ok) throw new Error(out?.error ?? "couldn't read the page"); + const name = `${(out.clip.title || host(tab.url) || "page").replace(/[\\/:*?"<>|\s]+/g, " ").trim().slice(0, 100) || "page"}.md`; + return new File([out.clip.markdown], name, { type: "text/markdown" }); + } + + /** A file, downloaded as the person (their cookies on that site). */ + async download(href) { + let res; + try { + res = await this.fetch(href, { credentials: "include" }); + } catch (e) { + throw new Error(`couldn't download it (${e.message}): the file may come from another site, which needs allowing too`); + } + if (!res.ok) throw new Error(`the site answered ${res.status} for that file`); + const blob = await res.blob(); + return new File([blob], fileName(res.headers.get("content-disposition"), res.url || href, blob.type), { type: blob.type }); + } +} + +const size = (n) => (n >= 1048576 ? `${(n / 1048576).toFixed(1)} MB` : `${Math.max(1, Math.round(n / 1024))} KB`); + +/** A downloaded file's name: from Content-Disposition, else the address, with an extension that says its type. */ +export function fileName(disposition, url, type) { + let name = ""; + const star = /filename\*\s*=\s*(?:UTF-8|utf-8)''([^;]+)/.exec(disposition ?? ""); + const plain = /filename\s*=\s*"?([^";]+)"?/i.exec(disposition ?? ""); + try { + if (star) name = decodeURIComponent(star[1].trim()); + else if (plain) name = plain[1].trim(); + else name = decodeURIComponent(new URL(url).pathname.split("/").pop() ?? ""); + } catch { + name = ""; + } + name = name.replace(/[\\/:*?"<>|\u0000-\u001f]+/g, "-").trim() || "download"; + const want = EXT_TYPES[(type ?? "").split(";")[0].trim().toLowerCase()]; + if (want && !/\.[a-z0-9]{1,5}$/i.test(name)) name = `${name}.${want}`; + return name.slice(-150); +} + +/** Attach a file to the person's conversation: a held upload (POST /uploads with hold), sent in chunks. */ +export async function upload(file) { + const limits = await apiJson("/uploads/limits"); + if (file.size > limits.max_mb * 1048576) throw new Error(`that's ${size(file.size)}; this Lens takes files up to ${limits.max_mb} MB`); + let up = await apiJson("/uploads", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ filename: file.name, size: file.size, hold: true }), + }); + const chunk = Math.max(1, Math.floor((limits.chunk_mb || 8) * 1048576)); + while (up.state === "receiving") { + up = await apiJson(`/uploads/${up.id}?offset=${up.offset}`, { + method: "PUT", + headers: { "Content-Type": "application/octet-stream" }, + body: file.slice(up.offset, up.offset + chunk), + }); + } + return up; +} diff --git a/extensions/browser/src/core/snapshot.js b/extensions/browser/src/core/snapshot.js new file mode 100644 index 000000000..1df920fd0 --- /dev/null +++ b/extensions/browser/src/core/snapshot.js @@ -0,0 +1,186 @@ +// What the assistant sees of a page (browser_read): its text in reading order, with each link, button and field +// numbered (e12) so it can act on them. Runs in the page, in the extension's own world: the page can't see the numbers +// or change what they point at. Names and roles come from the page's accessibility tree, the way a screen reader +// reads it (dom-accessibility-api). Password and card fields show no value. +import { computeAccessibleName, getRole } from "dom-accessibility-api"; + +export const READ_CHARS = 10000; // a read's text, unless asked for more or less +export const MOST_CHARS = 40000; + +const ROLES = [ + "button", "link", "checkbox", "radio", "tab", "menuitem", "menuitemcheckbox", "menuitemradio", "option", "switch", + "combobox", "textbox", "searchbox", "treeitem", "slider", "spinbutton", +]; // prettier-ignore +const INTERACTIVE = [ + "a[href]", "button", "input:not([type=hidden])", "select", "textarea", "summary", "[contenteditable='']", + "[contenteditable='true']", ...ROLES.map((r) => `[role=${r}]`), +].join(","); // prettier-ignore +const SKIP = new Set(["SCRIPT", "STYLE", "NOSCRIPT", "TEMPLATE", "HEAD", "META", "LINK", "IFRAME", "OBJECT", "EMBED", "CANVAS"]); +const BLOCK = new Set([ + "P", "DIV", "SECTION", "ARTICLE", "MAIN", "HEADER", "FOOTER", "NAV", "ASIDE", "LI", "UL", "OL", "TR", "TABLE", + "BLOCKQUOTE", "PRE", "FORM", "FIELDSET", "DL", "DT", "DD", "FIGURE", "FIGCAPTION", "BR", "HR", "H1", "H2", "H3", + "H4", "H5", "H6", "DETAILS", "DIALOG", "ADDRESS", "LEGEND", "CAPTION", "TBODY", "THEAD", "TFOOT", +]); // prettier-ignore +const CARD = /\bcc-|credit.?card|card.?num|\bcvc\b|\bcvv\b|\bcsc\b|security.?code/i; + +const clean = (s) => String(s ?? "").replace(/\s+/g, " ").trim(); +const cut = (s, n) => (s.length > n ? `${s.slice(0, n - 1)}…` : s); +const quote = (s) => JSON.stringify(s); + +/** The numbers a document's elements have had, kept for the life of the page. */ +export function newRefs() { + return { next: 1, byRef: new Map(), byEl: new WeakMap() }; +} + +function refOf(refs, el) { + let ref = refs.byEl.get(el); + if (!ref) { + ref = `e${refs.next++}`; + refs.byEl.set(el, ref); + refs.byRef.set(ref, new WeakRef(el)); + } + return ref; +} + +/** The element a ref names, while it's still on the page. */ +export function element(refs, ref) { + const el = refs.byRef.get(String(ref ?? "").trim())?.deref(); + return el?.isConnected ? el : null; +} + +function hidden(el) { + if (el.hidden || el.getAttribute("aria-hidden") === "true" || el.inert) return true; + if (typeof el.checkVisibility === "function") return !el.checkVisibility({ visibilityProperty: true }); + const style = el.ownerDocument.defaultView?.getComputedStyle?.(el); + return style ? style.display === "none" || style.visibility === "hidden" : false; +} + +/** A password or card field, which the assistant never fills (and never sees the value of): why, or null. */ +export function secret(el) { + if (el.tagName === "INPUT" && el.type === "password") return "a password field"; + const auto = el.getAttribute?.("autocomplete") ?? ""; + if (/current-password|new-password|one-time-code/.test(auto)) return "a password field"; + const hints = [auto, el.getAttribute?.("name"), el.id, el.getAttribute?.("aria-label"), el.getAttribute?.("placeholder")].join(" "); + if (CARD.test(hints)) return "a card field"; + return null; +} + +/** An element as the assistant reads it: role, name and state, e.g. textbox "To" = "ada@example.com". */ +export function describe(el) { + const tag = el.tagName; + let role = getRole(el) || tag.toLowerCase(); + if (tag === "INPUT" && el.type === "password") role = "password field"; + const name = cut(clean(computeAccessibleName(el)), 150); + const out = { role, name }; + if (tag === "A" && el.href && /^https?:/i.test(el.href)) out.href = el.href; + if (tag === "SELECT") { + out.value = clean(el.selectedOptions?.[0]?.label ?? ""); + out.options = [...el.options].slice(0, 25).map((o) => clean(o.label)); + } else if (tag === "INPUT" && /^(checkbox|radio)$/.test(el.type)) { + out.checked = el.checked; + } else if (/^(checkbox|radio|switch|menuitemcheckbox|menuitemradio)$/.test(role)) { + out.checked = el.getAttribute("aria-checked") === "true"; + } else if ((tag === "INPUT" && !/^(button|submit|reset|image|file|range|color)$/.test(el.type)) || tag === "TEXTAREA") { + out.value = secret(el) ? null : cut(clean(el.value), 300); + } else if (el.isContentEditable) { + out.value = cut(clean(el.innerText ?? el.textContent), 300); + } + if (el.disabled || el.getAttribute("aria-disabled") === "true") out.disabled = true; + if (el.getAttribute("aria-expanded")) out.expanded = el.getAttribute("aria-expanded") === "true"; + if (el.getAttribute("aria-selected") === "true") out.selected = true; + return out; +} + +function line(ref, d) { + let s = `[${ref}] ${d.role}${d.name ? ` ${quote(d.name)}` : ""}`; + if (d.href) s += ` → ${cut(d.href, 200)}`; + if (d.value === null) s += " (the person types this)"; + else if (d.value !== undefined && d.value !== "") s += ` = ${quote(d.value)}`; + if (d.options) s += ` (options: ${d.options.join(" | ")})`; + if (d.checked !== undefined) s += d.checked ? " (checked)" : " (not checked)"; + if (d.expanded !== undefined) s += d.expanded ? " (open)" : " (closed)"; + if (d.selected) s += " (selected)"; + if (d.disabled) s += " (disabled)"; + return `${s} `; +} + +/** The page as lines of text, interactive elements numbered in place. */ +function lines(doc, refs) { + const out = []; + let current = ""; + const flush = () => { + const t = clean(current); + if (t) out.push(t); + current = ""; + }; + // what's drawn inside a node: an element's shadow tree in place of its children, a slot's assigned nodes + const inside = (node) => { + if (node.shadowRoot) return node.shadowRoot.childNodes; + if (node.tagName === "SLOT") { + const assigned = node.assignedNodes?.({ flatten: true }) ?? []; + if (assigned.length) return assigned; + } + return node.childNodes; + }; + const walk = (node) => { + for (const n of inside(node)) { + if (n.nodeType === 3) { + current += n.nodeValue; + continue; + } + if (n.nodeType !== 1 || SKIP.has(n.tagName) || n.tagName === "svg" || hidden(n)) continue; + const block = BLOCK.has(n.tagName); + if (block) flush(); + if (/^H[1-6]$/.test(n.tagName)) current += `${"#".repeat(Number(n.tagName[1]))} `; + else if (n.tagName === "LI") current += "- "; + if (n.matches(INTERACTIVE)) { + current += line(refOf(refs, n), describe(n)); + if (n.tagName === "SELECT" || n.tagName === "TEXTAREA") continue; + // a link or button speaks for what's inside it, unless it holds more of the page + if (!n.querySelector(INTERACTIVE) && (n.textContent ?? "").length < 300) continue; + } + if (n.tagName === "TD" || n.tagName === "TH") current += " | "; + walk(n); + if (n.tagName === "IMG" && n.alt) current += ` [image: ${n.alt}] `; + if (block) flush(); + } + }; + walk(doc.body ?? doc.documentElement); + flush(); + return out; +} + +/** Only the lines that mention one of these words, with the line around each. */ +function found(all, words) { + const terms = clean(words).toLowerCase().split(" ").filter(Boolean); + const keep = new Set(); + all.forEach((l, i) => { + const low = l.toLowerCase(); + if (terms.some((t) => low.includes(t))) for (const j of [i - 1, i, i + 1]) if (j >= 0 && j < all.length) keep.add(j); + }); + const out = []; + let last = -2; + for (const i of [...keep].sort((a, b) => a - b)) { + if (i !== last + 1 && out.length) out.push("…"); + out.push(all[i]); + last = i; + } + return out; +} + +/** Read the page: {url, title, text, cut_short?}. `find` keeps only what mentions its words. */ +export function read(doc, refs, { find = "", max_chars: most = READ_CHARS } = {}) { + const limit = Math.max(500, Math.min(Number(most) || READ_CHARS, MOST_CHARS)); + let all = lines(doc, refs); + if (clean(find)) { + all = found(all, find); + if (!all.length) all = [`Nothing on this page mentions ${quote(clean(find))}.`]; + } + let text = all.join("\n"); + const page = { url: doc.URL, title: clean(doc.title) }; + if (text.length > limit) { + text = `${text.slice(0, limit)}\n…`; + page.cut_short = "the page goes on: read with find for the part you need, or a larger max_chars"; + } + return { ...page, text }; +} diff --git a/extensions/browser/src/ui/options.html b/extensions/browser/src/ui/options.html index fd9bc1db2..c3ad81eaa 100644 --- a/extensions/browser/src/ui/options.html +++ b/extensions/browser/src/ui/options.html @@ -88,6 +88,14 @@

Clipping

Shortcuts: Alt+Shift+L opens the clipper, Alt+Shift+P clips the page, Alt+Shift+S the selection, Alt+Shift+K picks. Change them in your browser's extension shortcuts.

+

The assistant

+

Ask Lens from its side panel: Ask Lens in the popup or on the right-click menu. While the panel is open, the + assistant can open, read and act on pages in that window, only on sites you allowed, and it asks you before anything that sends, + deletes, buys or posts.

+ + +
+

diff --git a/extensions/browser/src/ui/options.js b/extensions/browser/src/ui/options.js index ce975a3cd..03c9ec106 100644 --- a/extensions/browser/src/ui/options.js +++ b/extensions/browser/src/ui/options.js @@ -1,4 +1,6 @@ -// The options page: where clips go, the template, Markdown style and clipping choices. Changes save as they're made. +// The options page: where clips go, the template, Markdown style and clipping choices, and what the assistant may do +// in this browser. Changes save as they're made. +import * as browserCtl from "../core/browser.js"; import * as settingsStore from "../core/settings.js"; import { check } from "../core/lens.js"; import { FILTERS, VARIABLES } from "../core/template.js"; @@ -170,8 +172,39 @@ $("vars").replaceChildren( ); $("filters").textContent = `Filters: ${Object.keys(FILTERS).join(", ")}.`; +// ---------- the assistant ---------- +async function renderSites() { + const all = await browserCtl.sites(); + $("agentSites").replaceChildren( + ...all.map((origin) => { + const row = document.createElement("p"); + row.className = "row"; + const name = document.createElement("span"); + name.className = "grow"; + name.textContent = new URL(origin).host; + const remove = document.createElement("button"); + remove.textContent = "Remove"; + remove.addEventListener("click", async () => { + await browserCtl.forgetSite(origin); + saved("Removed: the assistant asks again before using it"); + renderSites(); + }); + row.append(name, remove); + return row; + }), + ); + if (!all.length) $("agentSites").replaceChildren(Object.assign(document.createElement("p"), { className: "hint", textContent: "None yet." })); +} + +$("agentEnabled").addEventListener("change", async () => { + await browserCtl.setEnabled($("agentEnabled").checked); + saved(); +}); + (async () => { settings = await settingsStore.load(); fill(); renderDestinations(); + $("agentEnabled").checked = await browserCtl.enabled(); + renderSites(); })(); diff --git a/extensions/browser/src/ui/sidepanel.html b/extensions/browser/src/ui/sidepanel.html index 5a84213b0..2f9e9403f 100644 --- a/extensions/browser/src/ui/sidepanel.html +++ b/extensions/browser/src/ui/sidepanel.html @@ -28,12 +28,19 @@ a.cite { font-size: 11px; text-decoration: none; vertical-align: super; } .steps { list-style: none; margin: 0 0 6px; padding: 0; font-size: 12px; color: var(--muted); } .steps li::before { content: "✓ "; color: var(--ok); } + .steps li.running { color: var(--fg); } + .steps li.running::before { content: "◌ "; color: var(--accent); } + .steps li.bad::before { content: "✗ "; color: var(--bad); } .note { font-size: 12px; color: var(--muted); margin: 4px 0 0; } .note.bad { color: var(--bad); } .thinking { font-size: 12px; color: var(--muted); } .approval { border: 1px solid var(--accent); border-radius: 8px; padding: 8px 10px; margin-top: 6px; } .approval p { margin: 0 0 6px; } .approval .row { flex-wrap: wrap; } + .approval .detail { white-space: pre-wrap; background: var(--field); border-radius: 6px; padding: 4px 8px; max-height: 120px; overflow-y: auto; } + .browser { display: flex; align-items: center; gap: 6px; margin: 0 10px 8px; font-size: 12px; } + .browser input { margin: 0; } + .browser .hint.bad { color: var(--bad); } #ask { display: flex; gap: 6px; align-items: flex-end; padding: 8px 10px 4px; border-top: 1px solid var(--line); } #q { flex: 1; resize: none; max-height: 160px; font-family: inherit; font-size: 13px; } .keys { margin: 0 10px 8px; } @@ -68,6 +75,9 @@

Ask Lens

Enter sends, Shift+Enter starts a new line, Esc stops the answer.

+ diff --git a/extensions/browser/src/ui/sidepanel.js b/extensions/browser/src/ui/sidepanel.js index 4f14ffd5b..b48be858f 100644 --- a/extensions/browser/src/ui/sidepanel.js +++ b/extensions/browser/src/ui/sidepanel.js @@ -1,7 +1,10 @@ // The side panel: the Lens assistant next to any page (docs/browser-assistant.md). Sign in once, through Lens's own -// sign-in page, then ask. The conversation is the same one the web app shows, so it can be picked up there. +// sign-in page, then ask. The conversation is the same one the web app shows, so it can be picked up there. While the +// panel is open, the assistant can also use this browser: each action shows here, a site is used only once the person +// allows it, and anything consequential waits for their yes. import * as account from "../core/account.js"; import * as assistant from "../core/assistant.js"; +import * as browserCtl from "../core/browser.js"; import * as settingsStore from "../core/settings.js"; const ext = globalThis.browser ?? globalThis.chrome; @@ -10,7 +13,10 @@ const CHAT_KEY = "assistantChat"; // {server, id}: the conversation the side pan let me = null; // the signed-in account let chatId = null; -let answering = null; // {controller} while an answer streams +let answering = null; // {controller, view} while an answer streams +let session = null; // this browser's session with Lens, while the assistant may use it +let listening = null; // aborts the session's action stream +const ASK_MS = 80_000; // a question to the person lapses before Lens stops waiting for the action (90 seconds) function el(tag, props = {}, ...children) { const node = document.createElement(tag); @@ -76,6 +82,7 @@ $("signinForm").addEventListener("submit", (e) => { $("signout").addEventListener("click", async () => { if (answering) await stopAnswer(); + stopBrowser(); await ext.storage.local.remove(CHAT_KEY); await account.signOut(); $("log").replaceChildren(); @@ -121,6 +128,12 @@ function addAnswer() { steps.append(el("li", { textContent: text })); scrollDown(); }, + /** An action in the browser, as it happens: running, then done or not. */ + live(text) { + steps.hidden = false; + thinking.textContent = "Using the browser…"; + return liveStep(steps, text); + }, answer(text, passages) { thinking.remove(); body.innerHTML = assistant.answerHtml(text, passages, me.server, window); @@ -140,6 +153,129 @@ function addAnswer() { }; } +function liveStep(list, text) { + const item = el("li", { className: "running", textContent: `${text}…` }); + list.append(item); + scrollDown(); + return { + done(said, ok) { + item.className = ok ? "" : "bad"; + item.textContent = said; + scrollDown(); + }, + }; +} + +// ---------- the browser, for the assistant ---------- +/** Where an action shows: in the answer it's for, or on its own (an agent asking through the API). */ +function placeFor(action) { + if (answering?.view && action.chat === chatId) return answering.view; + return null; +} + +const ui = { + log(action, text) { + const view = placeFor(action); + if (view) return view.live(text); + $("log").querySelector(".empty")?.remove(); + const list = el("ul", { className: "steps" }); + $("log").append(el("div", { className: "msg assistant" }, el("p", { className: "note", textContent: "An app is using this browser through Lens:" }), list)); + return liveStep(list, text); + }, + /** Ask the person; resolves to their yes. For a site, the browser's own access is asked for in their click. */ + ask({ kind, text, detail, origins }, action = {}) { + return new Promise((resolve) => { + const status = el("span", { className: "status" }); + const yes = el("button", { className: "primary", type: "button", textContent: kind === "site" ? "Allow" : "Yes, do it" }); + const no = el("button", { type: "button", textContent: kind === "site" ? "Not this site" : "No" }); + const buttons = el("p", { className: "row" }, yes, no, status); + const card = el( + "div", + { className: `approval ask ${kind}`, role: "alertdialog", ariaLabel: text }, + el("p", { textContent: text }), + detail ? el("p", { className: kind === "action" ? "detail" : "hint", textContent: detail }) : null, + buttons, + ); + const finish = (ok, said) => { + clearTimeout(timer); + buttons.replaceChildren(el("span", { className: `status ${ok ? "ok" : ""}`, textContent: said })); + resolve(ok); + }; + const timer = setTimeout(() => finish(false, "No answer, so Lens didn't"), ASK_MS); + yes.addEventListener("click", () => { + // asked straight from the click: browsers grant a site only from the person's own gesture + const granted = origins && ext.permissions?.request ? ext.permissions.request({ origins }) : Promise.resolve(true); + granted.then( + (ok) => finish(ok, ok ? (kind === "site" ? "Allowed" : "Done") : "The browser didn't allow it"), + (e) => finish(false, e.message), + ); + }); + no.addEventListener("click", () => finish(false, kind === "site" ? "Not allowed" : "Not done")); + const view = placeFor(action); + (view?.root ?? $("log")).append(card); + scrollDown(true); + }); + }, +}; + +function browserState(text, kind = "") { + $("browserState").className = `hint ${kind}`; + $("browserState").textContent = text; +} + +/** Offer this browser to the assistant: a session with Lens, and its actions carried out as they come. */ +async function startBrowser() { + stopBrowser(); + $("useBrowser").checked = await browserCtl.enabled(); + if (!$("useBrowser").checked || !me) return browserState(""); + try { + session = await browserCtl.connect(); + } catch (err) { + if (err instanceof account.SignedOut) return showSignIn(err.message); + return browserState(err.status === 403 ? "Lens's admin turned this off" : `Not connected: ${err.message}`, err.status === 403 ? "" : "bad"); + } + const windowId = (await ext.windows.getCurrent()).id; + const hands = new browserCtl.Hands({ windowId, ui }); + listening = new AbortController(); + const mine = session; + browserCtl + .listen(mine.id, (action) => hands.run(action), { + signal: listening.signal, + onLive: (on) => browserState(on ? "Lens can use this browser on sites you allow" : "Reconnecting…"), + onGone: (status) => { + if (session !== mine) return; + session = null; + if (status === 403) browserState("Lens's admin turned this off"); + else startBrowser(); // Lens forgot the session (a restart): connect again + }, + }) + .catch((err) => (err instanceof account.SignedOut ? showSignIn(err.message) : browserState(err.message, "bad"))); +} + +function stopBrowser() { + listening?.abort(); + listening = null; + if (session && me) browserCtl.end(session.id, me); + session = null; +} + +$("useBrowser").addEventListener("change", async () => { + await browserCtl.setEnabled($("useBrowser").checked); + if ($("useBrowser").checked) await startBrowser().catch((err) => browserState(err.message, "bad")); + else { + stopBrowser(); + browserState(""); + } +}); + +// the panel closing ends the session at once (it would lapse by itself after a few seconds) +addEventListener("pagehide", () => stopBrowser()); + +// a renewed token, or signing out in another panel +ext.storage.onChanged.addListener((changes, area) => { + if (area === "local" && changes.account && me) me = changes.account.newValue ?? me; +}); + const DECIDED = { done: "Done", declined: "Declined" }; /** Work the assistant proposed, waiting for the person's pick (it never does it on its own). */ @@ -205,6 +341,7 @@ async function showChat() { emptyHint(); scrollDown(true); $("q").focus(); + startBrowser().catch((err) => browserState(err.message, "bad")); } $("new").addEventListener("click", async () => { @@ -236,14 +373,17 @@ async function send(text) { if (!chatId) await setChat(await assistant.newChat()); addUser(text); view.current = addAnswer(); - answering = { controller: new AbortController() }; + answering = { controller: new AbortController(), view: view.current }; setAnswering(true); let answer = ""; let passages = []; + const browser = session?.id; await assistant.ask(chatId, text, { + browser, signal: answering.controller.signal, onEvent(name, data) { const v = view.current; + if (name === "step" && browser && String(data.tool).startsWith("browser_")) return; // shown as it happened if (name === "step") v.step(data.summary || data.tool); else if (name === "approval") v.approval(data); else if (name === "notice") v.note(data.message); diff --git a/extensions/browser/test/act.test.js b/extensions/browser/test/act.test.js new file mode 100644 index 000000000..d824179b1 --- /dev/null +++ b/extensions/browser/test/act.test.js @@ -0,0 +1,83 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { JSDOM } from "jsdom"; +import { click, looksConsequential, press, refuseTyping, submit, type } from "../src/core/act.js"; + +const page = (body) => new JSDOM(`${body}`, { url: "https://shop.example.com/" }).window.document; + +test("typing sets the value the way a person's typing does, so the page's code sees it", () => { + const doc = page(''); + const seen = []; + doc.getElementById("q").addEventListener("input", (e) => seen.push(["input", e.target.value])); + doc.getElementById("q").addEventListener("change", (e) => seen.push(["change", e.target.value])); + assert.equal(type(doc.getElementById("q"), "lens"), "lens"); + assert.deepEqual(seen, [ + ["input", "lens"], + ["change", "lens"], + ]); + assert.equal(type(doc.getElementById("s"), "japan"), "Japan"); + assert.equal(doc.getElementById("s").value, "jp"); + assert.throws(() => type(doc.getElementById("s"), "Peru"), /no option "Peru"; it has France \| Japan/); +}); + +test("passwords, card numbers, checkboxes and file pickers aren't typed into", () => { + const doc = page(` + +
text
`); + const why = (id) => refuseTyping(doc.getElementById(id)); + assert.match(why("pw"), /password field: the person types those themselves/); + assert.match(why("cc"), /card field/); + assert.match(why("cvc"), /card field/); + assert.match(why("box"), /click it instead/); + assert.match(why("file"), /picks a file/); + assert.match(why("div"), /isn't a field/); + assert.equal(why("ok"), null); +}); + +test("sending, deleting, buying and posting look consequential; searching doesn't", () => { + const doc = page(` + + Place order +
+
`); + const looks = (id, what) => looksConsequential(doc.getElementById(id), what); + assert.equal(looks("send"), true); + assert.equal(looks("del"), true); + assert.equal(looks("buy"), true); + assert.equal(looks("next"), false); + assert.equal(looks("comment", "submit"), true, "submitting a form that posts"); + assert.equal(looks("comment", "type"), false, "typing alone sends nothing"); + assert.equal(looks("search", "submit"), false); +}); + +test("a click goes as a person's: pointer and mouse down and up, then the click", () => { + const doc = page(''); + const seen = []; + for (const t of ["pointerdown", "mousedown", "pointerup", "mouseup", "click"]) doc.getElementById("b").addEventListener(t, () => seen.push(t)); + click(doc.getElementById("b")); + assert.deepEqual(seen, ["pointerdown", "mousedown", "pointerup", "mouseup", "click"]); + assert.equal(doc.activeElement.id, "b"); +}); + +test("Enter in a form's field submits it unless the page takes the key; other keys go where the focus is", () => { + const doc = page('
'); + let submitted = 0; + doc.getElementById("f").addEventListener("submit", (e) => { + e.preventDefault(); + submitted += 1; + }); + type(doc.getElementById("q"), "lens"); + submit(doc.getElementById("q")); + assert.equal(submitted, 1); + doc.getElementById("q").addEventListener("keydown", (e) => e.key === "Enter" && e.preventDefault()); + submit(doc.getElementById("q")); + assert.equal(submitted, 1, "the page handled Enter itself"); + + const keys = []; + doc.getElementById("mine").addEventListener("keydown", (e) => keys.push(e.key)); + doc.getElementById("mine").focus(); + assert.equal(press(doc, "Escape"), "Pressed Escape"); + assert.deepEqual(keys, ["Escape"]); + assert.match(press(doc, "PageDown"), /Scrolled down/); + assert.throws(() => press(doc, "F12"), /no key "F12"/); +}); diff --git a/extensions/browser/test/browser.test.js b/extensions/browser/test/browser.test.js new file mode 100644 index 000000000..21a287e4e --- /dev/null +++ b/extensions/browser/test/browser.test.js @@ -0,0 +1,84 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; + +const area = new Map([["account", { server: "https://lens.test", accessToken: "la_token", apiToken: true }]]); +globalThis.chrome = { + storage: { + local: { + async get(key) { + return area.has(key) ? { [key]: structuredClone(area.get(key)) } : {}; + }, + async set(items) { + for (const [k, v] of Object.entries(items)) area.set(k, structuredClone(v)); + }, + async remove(key) { + area.delete(key); + }, + }, + }, +}; +const browser = await import("../src/core/browser.js"); + +test("a downloaded file keeps its name, and gets an extension that says what it is", () => { + assert.equal(browser.fileName('attachment; filename="Invoice March.pdf"', "https://x.test/a", "application/pdf"), "Invoice March.pdf"); + assert.equal(browser.fileName("attachment; filename*=UTF-8''R%C3%A9sum%C3%A9.docx", "https://x.test/a", ""), "Résumé.docx"); + assert.equal(browser.fileName(null, "https://x.test/files/report%202026.xlsx?dl=1", ""), "report 2026.xlsx"); + assert.equal(browser.fileName(null, "https://mail.x.test/attachment?id=7", "application/pdf"), "attachment.pdf"); + assert.equal(browser.fileName('attachment; filename="../../etc/passwd"', "https://x.test/", "text/plain"), "..-..-etc-passwd.txt"); + assert.equal(browser.fileName(null, "https://x.test/", ""), "download"); +}); + +test("the browser is named for Lens by its brand and system", () => { + const chrome = { userAgentData: { brands: [{ brand: "Not)A;Brand" }, { brand: "Chromium" }, { brand: "Google Chrome" }], platform: "macOS" } }; + assert.equal(browser.label(chrome), "Google Chrome on macOS"); + assert.equal(browser.label({ userAgentData: { brands: [{ brand: "Chromium" }, { brand: "Not.A/Brand" }], platform: "Linux" } }), "Chromium on Linux"); + assert.equal(browser.label({ userAgent: "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0" }), "Firefox"); + assert.equal(browser.label({ userAgent: "Mozilla/5.0 (Macintosh) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Safari/605.1.15" }), "Safari"); +}); + +test("sites the person allowed are remembered, and can be forgotten", async () => { + assert.equal(await browser.enabled(), true, "on until the person turns it off"); + await browser.allowSite("https://mail.example.com"); + await browser.allowSite("https://mail.example.com"); + await browser.allowSite("https://bank.example.com"); + assert.deepEqual(await browser.sites(), ["https://bank.example.com", "https://mail.example.com"]); + await browser.forgetSite("https://bank.example.com"); + assert.deepEqual(await browser.sites(), ["https://mail.example.com"]); + await browser.setEnabled(false); + assert.equal(await browser.enabled(), false); +}); + +test("a file is attached to the conversation as a held upload, sent in chunks", async () => { + const sent = []; + const realFetch = globalThis.fetch; + globalThis.fetch = async (url, init = {}) => { + const path = url.replace("https://lens.test/api/v1", ""); + if (path === "/uploads/limits") return Response.json({ max_mb: 1, chunk_mb: 100 / 1048576 }); + if (path === "/uploads" && init.method === "POST") { + const body = JSON.parse(init.body); + assert.equal(body.hold, true); + return Response.json({ id: "u1", filename: body.filename, size: body.size, offset: 0, state: "receiving" }, { status: 201 }); + } + const m = /^\/uploads\/u1\?offset=(\d+)$/.exec(path); + if (m && init.method === "PUT") { + const bytes = new Uint8Array(await init.body.arrayBuffer()); + sent.push([Number(m[1]), bytes.length]); + const offset = Number(m[1]) + bytes.length; + return Response.json({ id: "u1", filename: "notes.txt", size: 250, offset, state: offset >= 250 ? "held" : "receiving" }); + } + return Response.json({ detail: "not found" }, { status: 404 }); + }; + try { + const up = await browser.upload(new File(["x".repeat(250)], "notes.txt", { type: "text/plain" })); + assert.equal(up.state, "held"); + assert.equal(up.id, "u1"); + assert.deepEqual(sent, [ + [0, 100], + [100, 100], + [200, 50], + ]); + await assert.rejects(browser.upload(new File(["x".repeat(2 * 1048576)], "big.txt")), /up to 1 MB/); + } finally { + globalThis.fetch = realFetch; + } +}); diff --git a/extensions/browser/test/snapshot.test.js b/extensions/browser/test/snapshot.test.js new file mode 100644 index 000000000..4b46ddb4e --- /dev/null +++ b/extensions/browser/test/snapshot.test.js @@ -0,0 +1,84 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { JSDOM } from "jsdom"; +import { element, newRefs, read } from "../src/core/snapshot.js"; + +const page = (body) => new JSDOM(`Inbox (3)${body}`, { url: "https://mail.example.com/inbox" }).window.document; + +const MAIL = ` +

Inbox

+ +
AdaInvoice for Marchinvoice.pdf
+

Hello there, this is the page.

+
secret hidden text
+ +
+ + + + + + + +
`; + +test("a page reads as text in order, with its links, buttons and fields numbered", () => { + const doc = page(MAIL); + const refs = newRefs(); + const out = read(doc, refs); + assert.equal(out.title, "Inbox (3)"); + assert.equal(out.url, "https://mail.example.com/inbox"); + const text = out.text; + assert.match(text, /^# Inbox$/m); + assert.match(text, /\[e\d+\] link "Invoice for March" → https:\/\/mail\.example\.com\/m\/1/); + assert.match(text, /\[e\d+\] link "invoice\.pdf" → https:\/\/mail\.example\.com\/att\/1\/invoice\.pdf/); + assert.match(text, /\[e\d+\] link "Settings"(?! →)/, "a javascript: link has no address"); + assert.match(text, /Hello there, this is the page\./); + assert.match(text, /\[e\d+\] textbox "To" = "ada@example\.com"/); + assert.match(text, /\[e\d+\] password field "Password" \(the person types this\)/); + assert.match(text, /\[e\d+\] textbox "Card number" \(the person types this\)/); + assert.match(text, /combobox "Country" = "Japan" \(options: France \| Japan\)/); + assert.match(text, /checkbox "Remember me" \(checked\)/); + assert.match(text, /textbox "Message" = "See you"/); + assert.match(text, /\[e\d+\] button "Send"/); + assert.doesNotMatch(text, /hunter2|4111|secret hidden|Hidden button|also hidden/); + assert.ok(text.indexOf("Inbox") < text.indexOf("Invoice") && text.indexOf("Invoice") < text.indexOf("Send"), "reading order"); +}); + +test("refs stay with their elements across reads, and go when the element does", () => { + const doc = page(MAIL); + const refs = newRefs(); + const first = read(doc, refs).text; + const ref = /\[(e\d+)\] button "Send"/.exec(first)[1]; + assert.equal(element(refs, ref).textContent, "Send"); + doc.body.insertAdjacentHTML("afterbegin", ''); + const again = read(doc, refs).text; + assert.match(again, new RegExp(`\\[${ref}\\] button "Send"`), "the same element keeps its ref"); + element(refs, ref).remove(); + assert.equal(element(refs, ref), null); + assert.equal(element(refs, "e999"), null); +}); + +test("find keeps what mentions its words, with the line around it", () => { + const refs = newRefs(); + const out = read(page(MAIL), refs, { find: "invoice" }); + assert.match(out.text, /Invoice for March/); + assert.doesNotMatch(out.text, /Remember me/); + assert.match(read(page(MAIL), refs, { find: "zebra" }).text, /Nothing on this page mentions "zebra"/); +}); + +test("a long page is cut short, saying how to get the rest", () => { + const doc = page(`

${"word ".repeat(5000)}

`); + const out = read(doc, newRefs(), { max_chars: 1000 }); + assert.ok(out.text.length <= 1002); + assert.match(out.cut_short, /find/); +}); + +test("what's inside shadow roots and slots is read too", () => { + const doc = page('
Slotted title
'); + const root = doc.getElementById("host").attachShadow({ mode: "open" }); + root.innerHTML = '

'; + const text = read(doc, newRefs()).text; + assert.match(text, /## Slotted title/); + assert.match(text, /button "In the shadow"/); +}); diff --git a/threat-model.json b/threat-model.json index e257b48a1..9fcfe88ad 100644 --- a/threat-model.json +++ b/threat-model.json @@ -1056,6 +1056,14 @@ "provenance": "documented", "source": "docs/browser-assistant.md \"Only you\"" }, + { + "property": "P-BROWSER-GUARD", + "violation_symptom": "integrity-bypass", + "severity_tier": "security", + "conditions": "the side panel reads and acts only on sites the person allowed, opens addresses only there, asks before a consequential click, submit or Enter (model flag, the control's words, a posting form), never types into or reports password, card or file fields, and shows each action", + "provenance": "documented", + "source": "docs/browser-assistant.md \"Letting it use your browser\"" + }, { "property": "MX-AUTHZ-LIST-CAPS", "violation_symptom": "x-cap-exceeded", diff --git a/threat-model.md b/threat-model.md index 3cc3495b8..c6f63c6ac 100644 --- a/threat-model.md +++ b/threat-model.md @@ -507,6 +507,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-HIERARCHY | Collections nest at most 8 deep, without cycles, 5000 per namespace. | bad-data-accepted | correctness-only | *(documented, docs/api.md "at most 8 deep")* | Voided by: Checks only in `_check_parent` — `domain/hierarchy.py:132`, `:142-143`. `grep -rln '"parent"\]' fastapi_backend/app/domain` — only `domain/hierarchy.py` writes it (Q34 covers unread movers). | | P-FLOW | Flow and tool graphs never loop back and run within a step budget. | hang | correctness-only | *(documented, domain/flow.py docstring "Graphs never loop back")* | Voided by: none identified (not required for this symptom; no exhaustive search) | | P-BROWSER | The assistant uses a person's browser only from that person's own conversations and API calls, while their own Lens side panel is connected (seen in the last 20 seconds, not closed). Only that person sees the session's actions and answers them, once each. `ai.browser` off refuses new sessions and offers no browser tools. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Only you")* | Voided by: The side panel carries out what it's asked: which sites it acts on, and what it asks the person first, is the extension's (docs/browser-assistant.md); an agent holding the person's write token can ask too — `api/v1/routes/browser.py`. | +| P-BROWSER-GUARD | The Lens extension's side panel reads and acts only on sites the person allowed in it (and the browser granted), and opens an address only on such a site. It asks the person before a click, submit or Enter that the model marks consequential or whose control's words or posting form look so. It never types into password, card or file fields, never reports their values, and shows each action as it happens. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Letting it use your browser")* | Voided by: The consequential check is the model's flag plus a word list (`extensions/browser/src/core/act.js`): a control with no telling words that a misled model doesn't mark runs without asking; synthetic events reach only the page's own handlers. Removing a site from the options stops the side panel acting there, but leaves the browser's access to it for the clipper. | ### Worked routing examples diff --git a/threat-model.yaml b/threat-model.yaml index 842ce2a09..88781ae2a 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:96398fee54d8a0ff2efcfe809becf5307438e15538c79b4baffc56a3f78ecee8" +prose_version: "threat-model.md@sha256:6dedfe71cd1ec807c7a04b3338c03cb01f70b61882d620d46a236bbe6e257399" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -1451,6 +1451,14 @@ properties_claimed: violation_symptoms: [integrity-bypass] x-voided-by: "the side panel carries out what it's asked (which sites, what it asks first: the extension's); an agent with the person's write token can ask too (api/v1/routes/browser.py)" provenance: {kind: documented, source: 'docs/browser-assistant.md "Only you"'} + - id: P-BROWSER-GUARD + kind: integrity + components: [assistant] + tier: security-critical + conditions: "the side panel reads and acts only on sites the person allowed, opens addresses only there, asks before a consequential click, submit or Enter (model flag, the control's words, a posting form), never types into or reports password, card or file fields, and shows each action" + violation_symptoms: [integrity-bypass] + x-voided-by: "the consequential check is the model's flag plus a word list (extensions/browser/src/core/act.js): an unlabelled control a misled model doesn't mark runs without asking; removing a site keeps the browser's access for the clipper" + provenance: {kind: documented, source: 'docs/browser-assistant.md "Letting it use your browser"'} - id: MX-AUTHZ-LIST-CAPS kind: resource-bound components: [authz] From 72cd8bfef897f314e40f425deb46b928e1db7c57 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sat, 10 Oct 2026 22:25:26 +0000 Subject: [PATCH 2/2] docs(browser-assistant): say which browsers the end-to-end test covers Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ --- docs/browser-assistant.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/browser-assistant.md b/docs/browser-assistant.md index 5cadb9f02..8c469074e 100644 --- a/docs/browser-assistant.md +++ b/docs/browser-assistant.md @@ -1,9 +1,9 @@ # The assistant in your browser **Status: built** in the Lens extension for Chrome and Edge: the side panel, where you sign in and chat with the -assistant, and the assistant using that browser on the sites you allow. Safari runs it in a tab; Firefox's sidebar -comes with its build. Webmail attachments behind a download button rather than a link are not handled yet (Refine -later). +assistant, and the assistant using that browser on the sites you allow. Firefox (in its sidebar) and Safari (in a tab) +run the same code, but only Chromium is covered by the end-to-end test so far. Webmail attachments behind a download +button rather than a link are not handled yet (Refine later). Ask the assistant from your browser, and let it do things there for you: "check my email and import the important attachments", "find last month's invoice on the supplier's site and put it in Accounts", "save this page". It opens