Skip to content

feat(clipper): let the assistant use the browser from the side panel, on sites you allow - #330

Merged
adeelahmad merged 2 commits into
mainfrom
claude/project-thread-mr9hx9
Oct 10, 2026
Merged

adeelahmad merged 2 commits into
mainfrom
claude/project-thread-mr9hx9

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: the side panel (#322) could chat with the assistant, but the assistant couldn't do anything in the browser.

After: while the side panel is open, the assistant can use this browser for you, the way you asked: "check my email and import the important attachments". It reads the page you're on (links, buttons and fields numbered, so it can point at them), clicks, types, presses keys, scrolls, opens addresses, and imports a file (or the page as Markdown) into the conversation as a held upload. Each action appears in the panel as it happens.

It acts only on sites you allow. The first time it wants a site, the panel asks Allow or Not this site, and Allow also grants the browser's host permission in the same click. A click, submit or Enter that sends, deletes, buys, posts and so on waits for Yes, do it. It never types into or reads password, card or file fields. Allowed sites are listed in the options page, each with Remove, and the assistant's browser use can be turned off there. Closing the panel ends the session, and the browser tools go away with it.

This is the last of three PRs: #311 added the server side and #322 the side panel.

How:

  • src/core/snapshot.js reads a page as lines like [e12] link "Invoice.pdf" → https://…, with names and roles from dom-accessibility-api (MIT). It skips hidden and inert elements, walks shadow roots, and keeps refs in a WeakMap inside the content script's isolated world.
  • src/core/act.js clicks with real pointer and mouse events, types with the native value setter (so React sees it) or insertText for rich editors, submits with requestSubmit, and decides what is consequential: the model's own flag, a word list on the target's name, or a form that posts.
  • src/core/browser.js connects the session (POST /browser/sessions), listens to the actions stream (SSE, one action at a time, reconnecting with backoff), runs each in the panel's window through Hands, answers POST /browser/actions/{id}, and uploads downloads in chunks to /uploads with hold: true.
  • src/agent.js is the small script injected into a page to read and act there.
  • The Edge smoke test drives all of it in Chromium: connect, Allow then read, type, the password refusal, No then Yes on a consequential click, the page after the click, a download into a held upload, the action log, and closing the panel.
  • The threat model gains P-BROWSER-GUARD.

Motivation and Context

Adeel asked to chat with the assistant from the browser and have it do tasks there "just like the Claude browser extension", for example checking email and importing important attachments. When the mailbox is connected through IMAP, the assistant still prefers that source (#311 tells it so); the browser covers webmail and sites without an API.

Steps to reproduce (if appropriate):

Load the extension, open Ask Lens, sign in, turn on Use this browser, and ask "what's on this page?". Allow the site, then ask it to open an attachment and import it.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project.
  • I added or updated tests: test/snapshot.test.js, test/act.test.js and test/browser.test.js (48 extension tests pass), plus the Edge smoke test (passes locally in Chromium).
  • My change requires documentation updates, and I have updated docs/ accordingly (docs/browser-assistant.md); mkdocs build --strict is clean.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: the assistant chats and acts in your browser, and imports attachments. It merges on green. Download buttons that aren't links are written down as Refine later.
  • Agents: the browser tools are assistant tools (feat(assistant): let the assistant use a browser connected through the Lens side panel #311), so any agent talking to Lens through the chat API uses them while a panel is connected.
  • Zero setup: nothing to install on the server. Allowing a site is one click in the panel.
  • Fewest clicks: one click allows a site and grants the browser's permission together. Consequential steps take one click.
  • Configurable: allowed sites and turning browser use off are in the extension's options. The server switch ai.browser (from feat(assistant): let the assistant use a browser connected through the Lens side panel #311) is in Settings.
  • Light: no server change. One small MIT library in the extension.
  • Private and secure: pages are read in your own browser and sent only to your Lens. Password, card and file fields are never read or typed. Sites need your allow, and the threat model gains P-BROWSER-GUARD.
  • Cost: page reads are capped (10,000 characters by default, 40,000 at most), and model calls go through the existing chat ledger.
  • User decides: a new site, and anything that sends, deletes, buys or posts, waits for the person. Browser use is off unless the panel is open.
  • Upgrades: n/a, no stored server data changes. The allow list is new extension storage, and it starts empty.
  • Honest: the docs say it's built for Chrome and Edge, and that only Chromium is covered end to end.
  • Tests and docs updated. No generated files are touched.
  • Open standards: the ARIA accessible name and role computation (W3C accname) and SSE. dom-accessibility-api is MIT.
  • Refine later items are in docs/browser-assistant.md and DEVLOG.md: download buttons, redirects to other sites, frames, and Safari's default tab.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ


Generated by Claude Code

… on sites you allow

While the side panel is open, it connects this browser to Lens and carries out the assistant's browser actions in its
window: reading a page with its links, buttons and fields numbered (names and roles from dom-accessibility-api),
clicking, typing, keys and scrolling, opening addresses, and downloading files (or the page as Markdown) into the
conversation as held uploads. A site is used only once the person allows it in the panel; a consequential click,
submit or Enter waits for their yes; password, card and file fields are never typed into or read; each action shows
as it happens. Allowed sites are listed and removable in the options.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ
@adeelahmad
adeelahmad merged commit f21e628 into main Oct 10, 2026
13 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-mr9hx9 branch October 10, 2026 22:36
adeelahmad added a commit that referenced this pull request Oct 10, 2026
…s as built (#332)

#322, #330, #297 and #309 merged, so the README lists them as built and keeps
only open PRs under "In progress".


Claude-Session: https://claude.ai/code/session_01HDx7PPUauyjMn1LZs8JAsS

Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants