Repository navigation
feat(clipper): let the assistant use the browser from the side panel, on sites you allow - #330
Merged
Merged
Conversation
… on sites you allow While the side panel is open, it connects this browser to Lens and carries out the assistant's browser actions in its window: reading a page with its links, buttons and fields numbered (names and roles from dom-accessibility-api), clicking, typing, keys and scrolling, opening addresses, and downloading files (or the page as Markdown) into the conversation as held uploads. A site is used only once the person allows it in the panel; a consequential click, submit or Enter waits for their yes; password, card and file fields are never typed into or read; each action shows as it happens. Allowed sites are listed and removable in the options. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: the side panel (#322) could chat with the assistant, but the assistant couldn't do anything in the browser.
After: while the side panel is open, the assistant can use this browser for you, the way you asked: "check my email and import the important attachments". It reads the page you're on (links, buttons and fields numbered, so it can point at them), clicks, types, presses keys, scrolls, opens addresses, and imports a file (or the page as Markdown) into the conversation as a held upload. Each action appears in the panel as it happens.
It acts only on sites you allow. The first time it wants a site, the panel asks Allow or Not this site, and Allow also grants the browser's host permission in the same click. A click, submit or Enter that sends, deletes, buys, posts and so on waits for Yes, do it. It never types into or reads password, card or file fields. Allowed sites are listed in the options page, each with Remove, and the assistant's browser use can be turned off there. Closing the panel ends the session, and the browser tools go away with it.
This is the last of three PRs: #311 added the server side and #322 the side panel.
How:
src/core/snapshot.jsreads a page as lines like[e12] link "Invoice.pdf" → https://…, with names and roles fromdom-accessibility-api(MIT). It skips hidden and inert elements, walks shadow roots, and keeps refs in a WeakMap inside the content script's isolated world.src/core/act.jsclicks with real pointer and mouse events, types with the native value setter (so React sees it) orinsertTextfor rich editors, submits withrequestSubmit, and decides what is consequential: the model's own flag, a word list on the target's name, or a form that posts.src/core/browser.jsconnects the session (POST /browser/sessions), listens to the actions stream (SSE, one action at a time, reconnecting with backoff), runs each in the panel's window throughHands, answersPOST /browser/actions/{id}, and uploads downloads in chunks to/uploadswithhold: true.src/agent.jsis the small script injected into a page to read and act there.Motivation and Context
Adeel asked to chat with the assistant from the browser and have it do tasks there "just like the Claude browser extension", for example checking email and importing important attachments. When the mailbox is connected through IMAP, the assistant still prefers that source (#311 tells it so); the browser covers webmail and sites without an API.
Steps to reproduce (if appropriate):
Load the extension, open Ask Lens, sign in, turn on Use this browser, and ask "what's on this page?". Allow the site, then ask it to open an attachment and import it.
Types of changes
!in the title)Checklist:
test/snapshot.test.js,test/act.test.jsandtest/browser.test.js(48 extension tests pass), plus the Edge smoke test (passes locally in Chromium).docs/accordingly (docs/browser-assistant.md);mkdocs build --strictis clean.## UnreleasedinCHANGELOG.md.Done questions (AGENTS.md)
ai.browser(from feat(assistant): let the assistant use a browser connected through the Lens side panel #311) is in Settings.docs/browser-assistant.mdandDEVLOG.md: download buttons, redirects to other sites, frames, and Safari's default tab.🤖 Generated with Claude Code
https://claude.ai/code/session_01PhTrpdr3XSgcD91oFxAiMQ
Generated by Claude Code