Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
month. **Costs** in the left menu shows spend by day, model and resource, and sets the namespace's budget
([Budgets](docs/budgets.md#the-meter-and-the-costs-page)).

- **Ask Lens from your browser.** The web clipper extension has the Lens assistant in a side panel: **Ask Lens** in
its popup or on the right-click menu. It signs in with one click through Lens's own sign-in page (OAuth, nothing to
copy), and the conversations are the same ones Lens's chat shows, with cited answers, the steps taken and approvals.
Letting the assistant use the browser itself is in progress ([The assistant in your browser](docs/browser-assistant.md)).

## 0.7.0 <small>October 10, 2026</small> {id="0.7.0"}

- **Prometheus metrics.** Lens can serve `/metrics` for Prometheus and Grafana: jobs by status and pipeline, imports,
Expand Down
5 changes: 3 additions & 2 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,14 @@ action log. Page text reaches the model as tool results (third-party text, untru
PRs:

- [x] Server: sessions, actions, the browser tools, `ai.browser` and `ai.browser_steps`, the API for agents
- [ ] Extension: side panel chat, OAuth sign-in
- [x] Extension: side panel chat, OAuth sign-in (the Edge smoke test signs in through a stand-in consent page)
- [ ] Extension: browser control (read with refs, click, type, keys, download into the conversation), site
permissions, asking first, the action log

Refine later: the browser from WhatsApp/Slack while the side panel is open (a setting, off); keep-alives in the chat
stream while an action waits; browser tools over MCP; real input events (Chrome's debugger) for sites that ignore
synthetic ones.
synthetic ones; asking about the page or selection from the right-click menu; the model and namespaces picker in the
side panel.

## 2026-10-10 · Similar content

Expand Down
29 changes: 27 additions & 2 deletions docs/browser-assistant.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# The assistant in your browser

**Status: in progress.** Built: the server side, below: sessions, the assistant's browser tools and the API. In
progress: the Lens extension's side panel, where you chat with the assistant and it uses your browser.
**Status: in progress.** Built: the server side (sessions, the assistant's browser tools and the API) and the Lens
extension's side panel, where you sign in and chat with the assistant. In progress: the side panel carrying out the
assistant's browser actions, with the sites you allow, your yes for anything consequential and the action log.

Ask the assistant from your browser, and let it do things there for you: "check my email and import the important
attachments", "find last month's invoice on the supplier's site and put it in Accounts", "save this page". It opens
Expand All @@ -12,6 +13,28 @@ Lens looks in what it already has first. Mail from an account connected as a sou
pages you captured need no browsing; the browser is for what Lens can't reach itself, such as webmail without IMAP,
or a site with no API.

## The side panel

Open it from the extension's popup (**Ask Lens**), by right-clicking any page (**Ask Lens…**), or with a shortcut you
give it in your browser's extension shortcuts (browsers suggest at most four per extension, and the clipper has them).

**Signing in** takes one click. The address is filled in from the clipper's Lens; press **Sign in**, let the extension
reach that address when the browser asks, and allow the "Lens browser extension" on Lens's own sign-in page, where you
are signed in with your passkey. Nothing to copy or paste: the extension signs in like any app (OAuth, see
[Authentication](authentication.md#oauth)), so it needs **Let apps and AI assistants sign in** on, in Settings → API
keys (on by default). Its tokens stay on this device and renew themselves; **Sign out** hands them back to Lens, and
you can also take its access away under API tokens → Apps with access.

**Asking** works as in Lens's chat, and it's the same conversation: **Open in Lens** picks it up there, and **New**
starts another. Answers cite their sources (a citation opens the moment it came from), the steps the assistant took
are listed above the answer, and anything it proposes (an import, say) waits in the panel for **Approve** or
**Decline**. <kbd>Enter</kbd> sends, <kbd>Shift</kbd>+<kbd>Enter</kbd> starts a new line and <kbd>Esc</kbd> stops the
answer.

Firefox shows the panel in its sidebar (also under **View → Sidebar → Ask Lens**) and signs in the same way. Safari
has no side panel and doesn't let extensions show a sign-in page, so there **Ask Lens** opens in a tab, and you sign in
with a write API token you make in Lens (your account's API tokens).

## How it works

The Lens extension's side panel signs in to your Lens and connects a **session**. While the side panel is open, a
Expand Down Expand Up @@ -59,6 +82,8 @@ A question sent with `browser` (the session's id) on `POST /api/v1/chats/{id}/me

## Refine later

- Ask about the page you're on, or the text you selected, straight from the right-click menu.
- Pick the model, or the namespaces to look in, in the side panel (the web app's chat has both).
- The assistant's browser for conversations elsewhere (WhatsApp, Slack, Telegram) while the side panel is open, as
a setting that's off by default.
- Keep-alive events in the chat stream while an action waits for you, for proxies that cut quiet streams short.
Expand Down
6 changes: 5 additions & 1 deletion docs/web-clipper.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ network.

With **Right-click and shortcut clips open the popup** on, those open the popup to look over first instead.

The same extension has the Lens assistant in a side panel: **Ask Lens** in the popup, or **Ask Lens…** on the
right-click menu. See [The assistant in your browser](browser-assistant.md).

## Templates

What's sent is the template with the clip's variables filled in. The default puts the metadata in YAML front matter
Expand Down Expand Up @@ -111,7 +114,8 @@ that. The code reaches the extension API as `globalThis.browser ?? globalThis.ch
### Firefox

Firefox 128 or later (desktop). `targets/firefox.json` runs the background as a script instead of a service worker
and gives the add-on its id, `lens-clipper@adeelahmad.github.io`. To try a build, open `about:debugging`, choose
and gives the add-on its id, `lens-clipper@adeelahmad.github.io`; the assistant opens in Firefox's sidebar
(`sidebar_action`) instead of a side panel. To try a build, open `about:debugging`, choose
**This Firefox**, then **Load Temporary Add-on…** and pick `dist/firefox/manifest.json`; it stays until Firefox
restarts.

Expand Down
3 changes: 2 additions & 1 deletion extensions/browser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ npm ci && npm test && npm run build # then Load unpacked: dist/chrome
```

- `src/core/`: the shared code every browser uses: `clip.js` (page, selection, element), `metadata.js`,
`markdown.js` (Turndown), `template.js`, `picker.js`, `lens.js` (pushing to Lens), `settings.js`
`markdown.js` (Turndown), `template.js`, `picker.js`, `lens.js` (pushing to Lens), `settings.js`; for the assistant's
side panel, `account.js` (signing in with OAuth) and `assistant.js` (the chat API, answers as safe HTML)
- `src/background.js`, `src/content.js`, `src/ui/`: the extension around it
- `targets/<browser>.json`: what each browser's manifest changes from `src/manifest.json`
100 changes: 97 additions & 3 deletions extensions/browser/edge/smoke.mjs
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
// Edge smoke test: loads dist/edge (npm run build in ..) into Microsoft Edge, opens the popup and options, clips an
// article page and checks a stand-in Lens import webhook got its Markdown. Run under a display (xvfb-run in CI).
// article page and checks a stand-in Lens import webhook got its Markdown. Then it signs the side panel in to the
// stand-in Lens through its OAuth consent page, asks the assistant and approves what it proposes. Run under a display
// (xvfb-run in CI).
// EDGE_CHANNEL=chromium runs the same test in Playwright's Chromium, for machines without Edge.
import { createHash } from "node:crypto";
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
Expand All @@ -22,15 +25,28 @@ ${"<p>The Lens clipper turns what you read into Markdown with its metadata and i
<ul><li>first point</li><li>second point</li></ul><pre><code class="language-js">const x = 1;</code></pre>
</article><footer>footer links</footer></body></html>`;

// the article page and a stand-in for Lens's import webhook, on one local port
// the article page and a stand-in for Lens (its import webhook, app sign-in and assistant), on one local port
const pushes = [];
const asked = [];
const decided = [];
const ACCESS = "lens-smoke-access-token";
let challenge = "";
const sse = (events) => events.map(([name, data]) => `event: ${name}\ndata: ${JSON.stringify(data)}\n\n`).join("");
const ANSWER = sse([
["step", { tool: "search", args: {}, summary: "Searched for “renewal”" }],
["approval", { id: 41, tool: "import_files", summary: "Import invoice.pdf into Personal" }],
["passages", [{ n: 1, recording_id: 7, title: "Renewal call", time: "12:34", t0: 754000, text: "renewal" }]],
["token", { text: "The renewal is **due in March** [1]." }],
["done", { message: 12 }],
]);
const server = createServer((req, res) => {
const cors = {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Headers": "Authorization, Content-Type, Accept",
"Access-Control-Allow-Methods": "POST, OPTIONS",
};
if (req.method === "OPTIONS") return res.writeHead(204, cors).end();
if (lens(req, res)) return;
if (req.url === "/article") return res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" }).end(ARTICLE);
if (req.url === "/api/v1/hooks/import" && req.method === "POST") {
let body = "";
Expand All @@ -44,6 +60,60 @@ const server = createServer((req, res) => {
}
res.writeHead(404, cors).end();
});

function lens(req, res) {
const url = new URL(req.url, origin);
const reply = (status, body, headers = {}) => res.writeHead(status, { "Content-Type": "application/json", ...headers }).end(JSON.stringify(body));
const read = (then) => {
let body = "";
req.on("data", (c) => (body += c));
req.on("end", () => then(body));
};
if (url.pathname === "/.well-known/oauth-authorization-server") {
reply(200, {
issuer: origin,
authorization_endpoint: `${origin}/oauth/authorize`,
token_endpoint: `${origin}/api/v1/oauth/token`,
registration_endpoint: `${origin}/api/v1/oauth/register`,
revocation_endpoint: `${origin}/api/v1/oauth/revoke`,
});
} else if (url.pathname === "/api/v1/oauth/register") {
read(() => reply(201, { client_id: "smoke-client" }));
} else if (url.pathname === "/oauth/authorize") {
// the consent page, with the person saying yes at once: back to the extension with a code
challenge = url.searchParams.get("code_challenge");
const back = new URL(url.searchParams.get("redirect_uri"));
back.searchParams.set("code", "smoke-code");
back.searchParams.set("state", url.searchParams.get("state"));
res.writeHead(302, { Location: back.toString() }).end();
} else if (url.pathname === "/api/v1/oauth/token") {
read((body) => {
const form = new URLSearchParams(body);
const ok = form.get("code") === "smoke-code" && createHash("sha256").update(form.get("code_verifier")).digest("base64url") === challenge;
reply(ok ? 200 : 400, ok ? { access_token: ACCESS, refresh_token: "smoke-refresh", expires_in: 3600 } : { error: "invalid_grant" });
});
} else if (url.pathname.startsWith("/api/v1/") && req.headers.authorization !== `Bearer ${ACCESS}` && !url.pathname.startsWith("/api/v1/hooks/")) {
reply(401, { detail: "sign in" });
} else if (url.pathname === "/api/v1/auth/me") {
reply(200, { user: { email: "ada@example.com", name: "Ada" } });
} else if (url.pathname === "/api/v1/chats" && req.method === "POST") {
read(() => reply(200, { id: 5 }));
} else if (url.pathname === "/api/v1/chats/5/messages" && req.method === "POST") {
read((body) => {
asked.push(JSON.parse(body));
res.writeHead(200, { "Content-Type": "text/event-stream" }).end(ANSWER);
});
} else if (url.pathname === "/api/v1/approvals/41" && req.method === "POST") {
read((body) => {
decided.push(JSON.parse(body));
reply(200, { status: "done" });
});
} else {
return false;
}
return true;
}

await new Promise((ok) => server.listen(0, "127.0.0.1", ok));
const origin = `http://127.0.0.1:${server.address().port}`;

Expand Down Expand Up @@ -78,7 +148,7 @@ try {

// a first install opens the options page; it and the popup render without errors
const errors = [];
for (const page of ["options.html", "popup.html"]) {
for (const page of ["options.html", "popup.html", "sidepanel.html"]) {
const p = await context.newPage();
p.on("pageerror", (e) => errors.push(`${page}: ${e.message}`));
await p.goto(`chrome-extension://${id}/${page}`);
Expand Down Expand Up @@ -116,6 +186,30 @@ try {
const push = pushes.at(-1);
check(push?.auth === `Bearer ${TOKEN}`, "Lens got it with the hook's token");
check(push && JSON.parse(push.body).text === md, "and the Markdown as text");

// the side panel: sign in through the consent page, ask, see the cited answer, approve what it proposes
check(manifest.side_panel?.default_path === "sidepanel.html" && manifest.permissions.includes("sidePanel"), "the manifest has the side panel");
const panel = await context.newPage();
panel.on("pageerror", (e) => errors.push(`side panel: ${e.message}`));
await panel.goto(`chrome-extension://${id}/sidepanel.html`);
check((await panel.inputValue("#server")) === origin, "the side panel offers the clipper's Lens to sign in to");
await panel.click("#signinButton");
await panel.waitForSelector("#q:visible", { timeout: 20000 }).catch(() => null);
check(await panel.isVisible("#q"), `the side panel signs in through Lens's consent page${(await panel.isVisible("#q")) ? "" : `: ${await panel.textContent("#signinStatus")}`}`);
const kept = await panel.evaluate(async () => (await chrome.storage.local.get("account")).account);
check(kept?.accessToken === ACCESS && kept.email === "ada@example.com", "and keeps the token and who signed in");
await panel.fill("#q", "When is the renewal?");
await panel.press("#q", "Enter");
await panel.waitForSelector(".msg.assistant a.cite", { timeout: 10000 }).catch(() => null);
check(asked.at(-1)?.content === "When is the renewal?", "Enter asks the question");
check((await panel.textContent(".msg.assistant .steps")).includes("Searched for"), "the steps show");
const cite = await panel.getAttribute(".msg.assistant a.cite", "href").catch(() => null);
check(cite === `${origin}/resources/7?t=754`, "the answer cites the moment it came from");
check((await panel.textContent(".msg.assistant strong").catch(() => "")) === "due in March", "the answer is Markdown");
await panel.click(".approval button.primary");
await panel.waitForFunction(() => document.querySelector(".approval")?.textContent.includes("Done"), null, { timeout: 5000 }).catch(() => null);
check(decided.at(-1)?.decision === "approve", "approving sends the person's pick to Lens");
check(errors.length === 0, `no page errors${errors.length ? `: ${errors.join("; ")}` : ""}`);
} finally {
await context.close();
server.close();
Expand Down
40 changes: 40 additions & 0 deletions extensions/browser/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions extensions/browser/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
},
"dependencies": {
"@mozilla/readability": "0.6.0",
"dompurify": "3.4.16",
"eventsource-parser": "4.1.1",
"marked": "18.0.14",
"turndown": "7.2.4",
"turndown-plugin-gfm": "1.0.2"
},
Expand Down
6 changes: 4 additions & 2 deletions extensions/browser/scripts/build.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ function merge(base, over) {

function referenced(m) {
const files = new Set([...Object.values(m.icons ?? {}), ...Object.values(m.action?.default_icon ?? {})]);
for (const f of [m.action?.default_popup, m.options_ui?.page, m.background?.service_worker, ...(m.background?.scripts ?? [])]) if (f) files.add(f);
const pages = [m.action?.default_popup, m.options_ui?.page, m.side_panel?.default_path, m.sidebar_action?.default_panel];
for (const f of [...pages, m.background?.service_worker, ...(m.background?.scripts ?? [])]) if (f) files.add(f);
return [...files];
}

Expand All @@ -51,6 +52,7 @@ for (const target of targets) {
content: join(root, "src/content.js"),
popup: join(root, "src/ui/popup.js"),
options: join(root, "src/ui/options.js"),
sidepanel: join(root, "src/ui/sidepanel.js"),
},
outdir: out,
bundle: true,
Expand All @@ -61,7 +63,7 @@ for (const target of targets) {
define: { "process.env.NODE_ENV": '"production"' },
logLevel: "warning",
});
for (const f of ["popup.html", "options.html", "style.css"]) cpSync(join(root, "src/ui", f), join(out, f));
for (const f of ["popup.html", "options.html", "sidepanel.html", "style.css"]) cpSync(join(root, "src/ui", f), join(out, f));
cpSync(join(root, "src/icons"), join(out, "icons"), { recursive: true });
const manifest = merge({ ...base, version }, JSON.parse(readFileSync(join(root, "targets", `${target}.json`), "utf8")));
writeFileSync(join(out, "manifest.json"), `${JSON.stringify(manifest, null, 2)}\n`);
Expand Down
Loading
Loading