Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
- **Ask Lens from your browser.** The web clipper extension has the Lens assistant in a side panel: **Ask Lens** in
its popup or on the right-click menu. It signs in with one click through Lens's own sign-in page (OAuth, nothing to
copy), and the conversations are the same ones Lens's chat shows, with cited answers, the steps taken and approvals.
Letting the assistant use the browser itself is in progress ([The assistant in your browser](docs/browser-assistant.md)).
While the panel is open, the assistant can use that browser: open, read, click, type and download files into the
conversation, only on sites you allow, asking before anything that sends, deletes, buys or posts, never typing
passwords or card numbers, and showing each action as it happens ([The assistant in your browser](docs/browser-assistant.md)).

## 0.7.0 <small>October 10, 2026</small> {id="0.7.0"}

Expand Down
7 changes: 4 additions & 3 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,14 @@ PRs:

- [x] Server: sessions, actions, the browser tools, `ai.browser` and `ai.browser_steps`, the API for agents
- [x] Extension: side panel chat, OAuth sign-in (the Edge smoke test signs in through a stand-in consent page)
- [ ] Extension: browser control (read with refs, click, type, keys, download into the conversation), site
permissions, asking first, the action log
- [x] Extension: browser control (read with refs, click, type, keys, download into the conversation), site
permissions, asking first, the action log (the Edge smoke test drives each against a stand-in Lens)

Refine later: the browser from WhatsApp/Slack while the side panel is open (a setting, off); keep-alives in the chat
stream while an action waits; browser tools over MCP; real input events (Chrome's debugger) for sites that ignore
synthetic ones; asking about the page or selection from the right-click menu; the model and namespaces picker in the
side panel.
side panel; webmail attachments behind a download button, and files redirected to another site; frames; Safari's
default tab.

## 2026-10-10 · Similar content

Expand Down
39 changes: 36 additions & 3 deletions docs/browser-assistant.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# The assistant in your browser

**Status: in progress.** Built: the server side (sessions, the assistant's browser tools and the API) and the Lens
extension's side panel, where you sign in and chat with the assistant. In progress: the side panel carrying out the
assistant's browser actions, with the sites you allow, your yes for anything consequential and the action log.
**Status: built** in the Lens extension for Chrome and Edge: the side panel, where you sign in and chat with the
assistant, and the assistant using that browser on the sites you allow. Firefox (in its sidebar) and Safari (in a tab)
run the same code, but only Chromium is covered by the end-to-end test so far. Webmail attachments behind a download
button rather than a link are not handled yet (Refine later).

Ask the assistant from your browser, and let it do things there for you: "check my email and import the important
attachments", "find last month's invoice on the supplier's site and put it in Accounts", "save this page". It opens
Expand Down Expand Up @@ -35,6 +36,32 @@ Firefox shows the panel in its sidebar (also under **View → Sidebar → Ask Le
has no side panel and doesn't let extensions show a sign-in page, so there **Ask Lens** opens in a tab, and you sign in
with a write API token you make in Lens (your account's API tokens).

## Letting it use your browser

While the side panel is open, **Let Lens use this browser** (under the question box, on unless you turn it off) lets
the assistant work in that window when you ask it to:

- **A site is used only once you allow it.** The first time the assistant wants to read or act on a site, or open an
address there, the panel asks **Let Lens read and act on mail.example.com?** Allow, and the browser may ask too, for
the extension's access to that site. Allowed sites are listed in the extension's options, under The assistant, where
you remove them again.
- **Anything consequential waits for your yes.** A click or a submit that sends, deletes, buys, books, posts, shares
or changes an account shows what it is about to do (**Click “Send” on mail.example.com?**) with **Yes, do it** and
**No**. It asks when the assistant marks the action consequential, or when the button's words say so (send, delete,
pay, order, post, submit, confirm, unsubscribe...), or when the form posts somewhere. No answer within 80 seconds
counts as No.
- **Never passwords or card numbers.** It doesn't type into password fields, card number or security code fields,
or file pickers, and doesn't see what's in them: you type those.
- **You see each action as it happens**, in the answer it's for: opening, reading, clicking, typing, downloading,
with ✓ when done and ✗ when not (and why). An app using your browser through the API shows the same way.
- **Pages are third-party text.** What a page or an email says reaches the assistant marked as material to read, never
as instructions; opening an address on a site you haven't allowed asks first, so what it read can't be carried
off in a link.

Files the assistant downloads (an attachment, a PDF behind a sign-in, or the page itself as Markdown) are attached to
the conversation like a file you dropped in, and it puts them in a namespace with `import_files`, which waits for
your **Approve**.

## How it works

The Lens extension's side panel signs in to your Lens and connects a **session**. While the side panel is open, a
Expand Down Expand Up @@ -82,6 +109,12 @@ A question sent with `browser` (the session's id) on `POST /api/v1/chats/{id}/me

## Refine later

- Webmail attachments behind a download button (no link to read), and files a site hands over from another site
after a redirect, which need that site allowed too.
- Pages inside frames (other than the page's own shadow trees), and real input events through the browser's
debugger for pages that ignore synthetic ones.
- Safari, where the panel is a tab of its own: the assistant picks the page's tab from `browser_tabs` today; it
should default to the tab you were on before.
- Ask about the page you're on, or the text you selected, straight from the right-click menu.
- Pick the model, or the namespaces to look in, in the side panel (the web app's chat has both).
- The assistant's browser for conversations elsewhere (WhatsApp, Slack, Telegram) while the side panel is open, as
Expand Down
4 changes: 3 additions & 1 deletion extensions/browser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ npm ci && npm test && npm run build # then Load unpacked: dist/chrome

- `src/core/`: the shared code every browser uses: `clip.js` (page, selection, element), `metadata.js`,
`markdown.js` (Turndown), `template.js`, `picker.js`, `lens.js` (pushing to Lens), `settings.js`; for the assistant's
side panel, `account.js` (signing in with OAuth) and `assistant.js` (the chat API, answers as safe HTML)
side panel, `account.js` (signing in with OAuth) and `assistant.js` (the chat API, answers as safe HTML); for the
assistant using the browser, `browser.js` (the session and its actions, sites and asks), `snapshot.js` (a page read
with numbered elements) and `act.js` (clicking, typing, keys), the last two injected as `src/agent.js`
- `src/background.js`, `src/content.js`, `src/ui/`: the extension around it
- `targets/<browser>.json`: what each browser's manifest changes from `src/manifest.json`
90 changes: 89 additions & 1 deletion extensions/browser/edge/smoke.mjs
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Edge smoke test: loads dist/edge (npm run build in ..) into Microsoft Edge, opens the popup and options, clips an
// article page and checks a stand-in Lens import webhook got its Markdown. Then it signs the side panel in to the
// stand-in Lens through its OAuth consent page, asks the assistant and approves what it proposes. Run under a display
// stand-in Lens through its OAuth consent page, asks the assistant and approves what it proposes, and has the stand-in
// Lens send the side panel browser actions: reading a page once its site is allowed, typing, a click that waits for
// the person's yes, refusing a password field, and downloading a file into a held upload. Run under a display
// (xvfb-run in CI).
// EDGE_CHANNEL=chromium runs the same test in Playwright's Chromium, for machines without Edge.
import { createHash } from "node:crypto";
Expand Down Expand Up @@ -61,6 +63,23 @@ const server = createServer((req, res) => {
res.writeHead(404, cors).end();
});

// the browser, as Lens uses it: a session, its stream of actions, and their answers
const browserSide = { label: null, stream: null, answers: new Map(), waiting: new Map(), ended: false, uploaded: null, next: 1 };
function act(tool, args = {}, chat = null) {
const id = `a${browserSide.next++}`;
const done = new Promise((resolve, reject) => {
browserSide.waiting.set(id, resolve);
setTimeout(() => reject(new Error(`no answer to ${tool} ${id}`)), 30000);
});
browserSide.stream.write(`event: action\ndata: ${JSON.stringify({ id, tool, args, chat })}\n\n`);
return done;
}
const COMPOSE = `<!doctype html><html lang="en"><head><title>Compose</title></head><body><h1>New message</h1>
<label>To <input id="to"></label> <label>Password <input id="pw" type="password"></label>
<button id="send" onclick="document.getElementById('state').textContent = 'Sent to ' + document.getElementById('to').value">Send</button>
<p id="state">Draft</p><p>Attached: <a href="/files/report.pdf">report.pdf</a></p></body></html>`;
const PDF = "%PDF-1.4\n% a stand-in report\n%%EOF\n";

function lens(req, res) {
const url = new URL(req.url, origin);
const reply = (status, body, headers = {}) => res.writeHead(status, { "Content-Type": "application/json", ...headers }).end(JSON.stringify(body));
Expand Down Expand Up @@ -108,6 +127,42 @@ function lens(req, res) {
decided.push(JSON.parse(body));
reply(200, { status: "done" });
});
} else if (url.pathname === "/compose") {
res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" }).end(COMPOSE);
} else if (url.pathname === "/files/report.pdf") {
res.writeHead(200, { "Content-Type": "application/pdf", "Content-Disposition": 'attachment; filename="report.pdf"' }).end(PDF);
} else if (url.pathname === "/api/v1/browser/sessions" && req.method === "POST") {
read((body) => {
browserSide.label = JSON.parse(body || "{}").label;
reply(201, { id: "s1", label: browserSide.label, live: true });
});
} else if (url.pathname === "/api/v1/browser/sessions/s1/actions") {
res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-store" });
res.write(": open\n\n");
browserSide.stream = res;
req.on("close", () => browserSide.stream === res && (browserSide.stream = null));
} else if (url.pathname.startsWith("/api/v1/browser/actions/") && req.method === "POST") {
read((body) => {
const id = url.pathname.split("/").pop();
browserSide.answers.set(id, JSON.parse(body));
browserSide.waiting.get(id)?.(JSON.parse(body));
reply(200, { ok: true });
});
} else if (url.pathname === "/api/v1/browser/sessions/s1" && req.method === "DELETE") {
browserSide.ended = true;
reply(200, { ok: true });
} else if (url.pathname === "/api/v1/uploads/limits") {
reply(200, { max_mb: 10, chunk_mb: 8, extensions: [".pdf"] });
} else if (url.pathname === "/api/v1/uploads" && req.method === "POST") {
read((body) => {
const b = JSON.parse(body);
reply(201, { id: "up1", filename: b.filename, size: b.size, offset: 0, state: "receiving", hold: b.hold });
});
} else if (url.pathname === "/api/v1/uploads/up1" && req.method === "PUT") {
read((body) => {
browserSide.uploaded = body;
reply(200, { id: "up1", filename: "report.pdf", size: body.length, offset: body.length, state: "held" });
});
} else {
return false;
}
Expand Down Expand Up @@ -209,6 +264,39 @@ try {
await panel.click(".approval button.primary");
await panel.waitForFunction(() => document.querySelector(".approval")?.textContent.includes("Done"), null, { timeout: 5000 }).catch(() => null);
check(decided.at(-1)?.decision === "approve", "approving sends the person's pick to Lens");

// the assistant using this browser: Lens sends actions, the side panel carries them out on sites the person allows
await panel.waitForFunction(() => document.getElementById("browserState").textContent.includes("can use"), null, { timeout: 10000 }).catch(() => null);
check(Boolean(browserSide.stream) && /\S/.test(browserSide.label ?? ""), `the side panel connects this browser to Lens (${browserSide.label})`);
const compose = await context.newPage();
await compose.goto(`${origin}/compose`);
const tab_id = await panel.evaluate(async (url) => (await chrome.tabs.query({ url })).at(0)?.id, `${origin}/compose`);
const reading = act("read", { tab_id });
await panel.locator(".ask.site button.primary").click({ timeout: 10000 });
let a = await reading;
check(a.ok && a.result.text.includes("New message"), `Lens reads a page once the person allows its site${a.ok ? "" : `: ${a.error}`}`);
const refOf = (name) => new RegExp(`\\[(e\\d+)\\] [a-z ]+"${name}"`).exec(a.result?.text ?? "")?.[1];
const [to, pw, send, file] = ["To", "Password", "Send", "report.pdf"].map(refOf);
a = await act("type", { tab_id, ref: to, text: "ada@example.com" });
check(a.ok && (await compose.inputValue("#to")) === "ada@example.com", `it types into a field${a.ok ? "" : `: ${a.error}`}`);
a = await act("type", { tab_id, ref: pw, text: "hunter2" });
check(!a.ok && /password/.test(a.error) && (await compose.inputValue("#pw")) === "", "and never into a password field");
const declined = act("click", { tab_id, ref: send });
await panel.locator(".ask.action").last().getByRole("button", { name: "No" }).click({ timeout: 10000 });
a = await declined;
check(!a.ok && /said no/.test(a.error) && (await compose.textContent("#state")) === "Draft", "a consequential click waits for the person, and No means it doesn't happen");
const sending = act("click", { tab_id, ref: send });
await panel.locator(".ask.action").last().getByRole("button", { name: "Yes, do it" }).click({ timeout: 10000 });
a = await sending;
check(a.ok && (await compose.textContent("#state")) === "Sent to ada@example.com", `and Yes does it${a.ok ? "" : `: ${a.error}`}`);
check(a.result?.text?.includes("Sent to ada@example.com"), "the answer has the page as it is after the click");
a = await act("import", { tab_id, ref: file });
check(a.ok && a.result.upload_id === "up1" && browserSide.uploaded === PDF, `it downloads a file into a held upload${a.ok ? "" : `: ${a.error}`}`);
const logged = await panel.locator(".steps li").allTextContents();
check(logged.some((t) => t.startsWith("Clicked “Send”")) && logged.some((t) => t.startsWith("Downloaded report.pdf")), "each action shows in the side panel");
await panel.close({ runBeforeUnload: true });
for (let i = 0; i < 20 && !browserSide.ended; i++) await new Promise((ok) => setTimeout(ok, 100));
check(browserSide.ended, "closing the side panel ends the session");
check(errors.length === 0, `no page errors${errors.length ? `: ${errors.join("; ")}` : ""}`);
} finally {
await context.close();
Expand Down
7 changes: 7 additions & 0 deletions extensions/browser/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions extensions/browser/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
},
"dependencies": {
"@mozilla/readability": "0.6.0",
"dom-accessibility-api": "0.7.1",
"dompurify": "3.4.16",
"eventsource-parser": "4.1.1",
"marked": "18.0.14",
Expand Down
1 change: 1 addition & 0 deletions extensions/browser/scripts/build.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ for (const target of targets) {
entryPoints: {
background: join(root, "src/background.js"),
content: join(root, "src/content.js"),
agent: join(root, "src/agent.js"),
popup: join(root, "src/ui/popup.js"),
options: join(root, "src/ui/options.js"),
sidepanel: join(root, "src/ui/sidepanel.js"),
Expand Down
54 changes: 54 additions & 0 deletions extensions/browser/src/agent.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
// Injected into a tab the assistant reads or acts on (only sites the person allowed). The side panel calls it through
// scripting.executeScript; it keeps the numbers of what it has read in the extension's own world, for the life of
// the page, and nothing else.
import * as act from "./core/act.js";
import { describe, element, newRefs, read, secret } from "./core/snapshot.js";

if (!globalThis.__lensAgent) {
const refs = newRefs();
const find = (ref) => {
const el = element(refs, ref);
if (!el) throw new Error(`there's no ${ref} on this page now: read it again for current refs`);
return el;
};
const answer = (fn) => {
try {
return { ok: true, result: fn() };
} catch (e) {
return { ok: false, error: e.message };
}
};
globalThis.__lensAgent = {
read: (opts) => answer(() => read(document, refs, opts)),
/** What a ref is, and whether acting on it looks consequential, for the side panel to decide what to ask. */
inspect: (ref, what = "click") =>
answer(() => {
const el = find(ref);
return {
...describe(el),
site: location.origin,
secret: secret(el),
refused: what === "click" ? null : act.refuseTyping(el),
consequential: act.looksConsequential(el, what),
};
}),
click: (ref) => answer(() => act.click(find(ref))),
type: (ref, text, submit) =>
answer(() => {
const el = find(ref);
const why = act.refuseTyping(el);
if (why) throw new Error(why);
const now = act.type(el, text);
if (submit) act.submit(el);
return now;
}),
/** What has the focus, and whether Enter there looks consequential. */
focused: () =>
answer(() => {
const el = document.activeElement;
if (!el || el === document.body) return null;
return { ...describe(el), consequential: act.looksConsequential(el, "submit") };
}),
press: (key) => answer(() => act.press(document, key)),
};
}
Loading
Loading