Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,30 @@

Plans and progress for work in flight. Newest first.

## 2026-10-10 · The assistant in your browser

Goal (Adeel): chat with the assistant from the browser extension, and let it do things by using the browser, like the
Claude browser extension: "check my email and import the important attachments".

Model (docs/browser-assistant.md): the clipper extension (extensions/browser) gets a side panel. It signs in with
OAuth (PKCE, `identity.launchWebAuthFlow`, nothing to copy), chats through `/chats` like the web app, and connects a
browser session (fastapi_backend/app/domain/browser.py). While it's connected, a question from it gets the
`browser_` tools; each action is a `browser_action` row the side panel takes from its event stream, carries out and
answers, and the tool waits for it in the database (any process, like stopping an answer). The extension guards what
the browser does: sites the person allowed, a yes before anything consequential, no passwords or card numbers, an
action log. Page text reaches the model as tool results (third-party text, untrusted.py once #302 is in).

PRs:

- [x] Server: sessions, actions, the browser tools, `ai.browser` and `ai.browser_steps`, the API for agents
- [ ] Extension: side panel chat, OAuth sign-in
- [ ] Extension: browser control (read with refs, click, type, keys, download into the conversation), site
permissions, asking first, the action log

Refine later: the browser from WhatsApp/Slack while the side panel is open (a setting, off); keep-alives in the chat
stream while an action waits; browser tools over MCP; real input events (Chrome's debugger) for sites that ignore
synthetic ones.

## 2026-10-10 · Similar content

Goal (Adeel): a Similar tab on every resource, and selecting text anywhere (a resource's text or title, notes, topics,
Expand Down
15 changes: 15 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1180,6 +1180,21 @@ answers, so a filter collection's new recordings count, and a deleted collection
`error`, and its latest source `check` (`{claims, supported, verdicts, uncited}`), as they were when it was written
or checked.

### browser

```
POST /api/v1/browser/sessions
GET /api/v1/browser/sessions
DELETE /api/v1/browser/sessions/{sid}
GET /api/v1/browser/sessions/{sid}/actions
POST /api/v1/browser/actions/{aid}
POST /api/v1/browser/sessions/{sid}/run
```

A browser's Lens side panel connects a session and carries out the actions the assistant asks of it; a question sent
with `browser` (`POST /chats/{cid}/messages {content, browser}`) can use that browser while it's connected. See
[The assistant in your browser](browser-assistant.md).

## collections

Saved collections: lists of recordings from anywhere, to chat with or run batches on. A namespace's own collections,
Expand Down
3 changes: 3 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ When the configured model supports function calling, chat becomes an agent.
bookmark export) and `import_urls` captures each page as a document, as `POST /api/v1/import/web/links` does. Like
`import_files`, it picks the namespace when you don't name one, and waits for your approval.
- **Keep typing:** what you send while an answer is being written waits its turn and goes next.
- **Your browser** (in progress): asked from the Lens extension's side panel, the assistant can open, read, click and
type in that browser and download files into the conversation, on sites you allow, asking before anything that
sends, deletes or buys. See [The assistant in your browser](browser-assistant.md).

## Third-party content and prompt injection

Expand Down
65 changes: 65 additions & 0 deletions docs/browser-assistant.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# The assistant in your browser

**Status: in progress.** Built: the server side, below: sessions, the assistant's browser tools and the API. In
progress: the Lens extension's side panel, where you chat with the assistant and it uses your browser.

Ask the assistant from your browser, and let it do things there for you: "check my email and import the important
attachments", "find last month's invoice on the supplier's site and put it in Accounts", "save this page". It opens
pages, reads them, clicks and types in your own browser, signed in wherever you are, and shows you each step as it
goes. Anything that sends, deletes, buys or posts waits for your yes.

Lens looks in what it already has first. Mail from an account connected as a source (IMAP), files you imported and
pages you captured need no browsing; the browser is for what Lens can't reach itself, such as webmail without IMAP,
or a site with no API.

## How it works

The Lens extension's side panel signs in to your Lens and connects a **session**. While the side panel is open, a
question you ask in it can use that browser: the assistant gets the browser tools, and each action it takes goes to
the side panel, which carries it out and sends back what happened. Close the side panel and the assistant can't use
the browser any more.

| Tool | What it does |
| --- | --- |
| `browser_tabs` | the tabs open in that browser window, and which one you're looking at |
| `browser_open` | open an address: in the tab you're looking at, another tab, or a new one |
| `browser_read` | the page's text, with its links, buttons and fields, each numbered (`e12`) to act on |
| `browser_click` | click a link, button, tab or checkbox |
| `browser_type` | type into a field, or pick an option of a list; optionally press Enter after |
| `browser_press` | press a key (Enter, Escape, Tab, arrows) or scroll (PageDown, PageUp, Home, End) |
| `browser_import` | download a file with your browser (an email's attachment, a file behind a sign-in), or save the page as Markdown, and attach it to the conversation; the assistant then imports it into a namespace with `import_files`, which waits for your approval like any import |

An answer that uses the browser may take up to **Tool steps when using the browser** steps (`ai.browser_steps`, 25
by default), since it reads and clicks its way through.

## Only you

- A session is yours: only you see its actions and answer them, and only your own conversations and API calls use it.
- The browser does what the side panel allows. It acts only on sites you allowed, asks you before anything
consequential, never types passwords or card numbers, and shows each action as it happens.
- What a page, an email or a file says is third-party text. The assistant reads it as material, never as
instructions, and is told never to put what it read into an address or a field unless you asked for exactly that.
- An admin can turn it off for everyone: Settings → AI assistant → **Let the assistant use a connected browser**
(`ai.browser`). Off, side panels can't connect and the assistant has no browser tools.

## For agents and scripts

Everything the side panel does goes through the API, with an API token or an app's token that can write:

| | |
| --- | --- |
| `POST /api/v1/browser/sessions` | connect a browser (`{label}`): its session |
| `GET /api/v1/browser/sessions` | your browsers connected now |
| `GET /api/v1/browser/sessions/{id}/actions` | server-sent `action` events `{id, tool, args, chat}` for that browser to carry out; `once=true` sends what's waiting and closes |
| `POST /api/v1/browser/actions/{id}` | what the browser did: `{ok, result}` or `{ok: false, error}` |
| `POST /api/v1/browser/sessions/{id}/run` | do one thing in your connected browser and wait for it: `{tool, args}`, the browser tools without `browser_` |
| `DELETE /api/v1/browser/sessions/{id}` | the side panel closed |

A question sent with `browser` (the session's id) on `POST /api/v1/chats/{id}/messages` can use that browser.

## Refine later

- The assistant's browser for conversations elsewhere (WhatsApp, Slack, Telegram) while the side panel is open, as
a setting that's off by default.
- Keep-alive events in the chat stream while an action waits for you, for proxies that cut quiet streams short.
- Browser tools over MCP (the MCP server is read-only today).
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
auth,
backups,
batches,
browser,
budgets,
chats,
collections,
Expand Down Expand Up @@ -113,6 +114,7 @@
extensions,
marketplaces,
chats,
browser,
collections,
batches,
public,
Expand Down
128 changes: 128 additions & 0 deletions fastapi_backend/app/api/v1/routes/browser.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
"""The person's browser, for the assistant (docs/browser-assistant.md): the Lens extension's side panel connects a
session and listens for actions, carries each one out in the browser and sends back what happened.

``POST /browser/sessions`` connects one; ``GET /browser/sessions/{id}/actions`` is its stream of ``action`` events
(server-sent, read with fetch: it needs the Authorization header); ``POST /browser/actions/{id}`` answers one. While the
stream is open, a question sent from the side panel (``browser`` on POST /chats/{cid}/messages) can use the browser, and
an agent can too (``POST /browser/sessions/{id}/run``). Only the session's own person reaches any of it.
"""

from __future__ import annotations

import asyncio
import json
import time
from collections.abc import AsyncIterator

from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import StreamingResponse
from starlette.concurrency import run_in_threadpool

from app.api.deps import Cfg, CurrentUser, Db, Principal, Writer
from app.domain import auth, browser
from app.domain.store import DB
from app.schemas.browser import BrowserAnswer, BrowserConnect, BrowserResult, BrowserRun, BrowserSession
from app.schemas.common import Ok

router = APIRouter(prefix="/browser", tags=["browser"])
OFF = "an admin turned off using a browser for the assistant (Settings → AI assistant)"


def _on(cfg: dict) -> None:
if not browser.enabled(cfg):
raise HTTPException(403, OFF)


def _session(db: DB, sid: str, user: Principal) -> dict:
try:
return browser.get(db, sid, user.id)
except KeyError:
raise HTTPException(404, "not found") from None


@router.post("/sessions", status_code=201)
def connect(user: Writer, db: Db, cfg: Cfg, body: BrowserConnect | None = None) -> BrowserSession:
"""Connect a browser's Lens side panel: then open its action stream. Audited as `browser.connect`."""
_on(cfg)
s = browser.connect(db, user.id, (body or BrowserConnect()).label)
auth.audit(db, user.as_audit(), "browser.connect", f"browser_session:{s['id']}", {"label": s["label"]})
return BrowserSession(**s)


@router.get("/sessions")
def list_sessions(user: CurrentUser, db: Db) -> list[BrowserSession]:
"""Your browsers whose side panel is connected now, the latest first."""
return [BrowserSession(**s) for s in browser.mine(db, user.id)]


@router.delete("/sessions/{sid}")
def disconnect(sid: str, user: Writer, db: Db) -> Ok:
"""The side panel closed: the assistant can't use that browser any more, and actions waiting on it fail."""
_session(db, sid, user)
browser.end(db, sid)
return Ok()


@router.get(
"/sessions/{sid}/actions",
response_class=StreamingResponse,
responses={200: {"content": {"text/event-stream": {}}, "description": "`event: action` with {id, tool, args, chat}"}},
)
async def actions(sid: str, request: Request, user: Writer, cfg: Cfg, once: bool = False) -> StreamingResponse:
"""The actions the assistant (or an agent) asks of this browser, as server-sent `action` events {id, tool, args,
chat}: carry each one out and answer it with POST /browser/actions/{id}. The session stays connected while this
stream is open; a comment line every 15 seconds keeps it so through proxies. `once=true` sends the actions waiting
now and closes, for a client that asks again and again instead (each time keeps it connected for a while)."""
_on(cfg)
db = request.app.state.db
await run_in_threadpool(_session, db, sid, user)
await run_in_threadpool(browser.seen, db, sid)

async def gen() -> AsyncIterator[str]:
yield ": open\n\n"
beat = quiet = time.monotonic()
while True:
for a in await run_in_threadpool(browser.take, db, sid):
quiet = time.monotonic()
yield f"event: action\ndata: {json.dumps(a, default=str)}\n\n"
if once:
break
now = time.monotonic()
if now - beat >= browser.BEAT_SECONDS:
beat = now
await run_in_threadpool(browser.seen, db, sid)
if now - quiet >= 15:
quiet = now
yield ": keep-alive\n\n"
if await request.is_disconnected():
break
await asyncio.sleep(browser.POLL_SECONDS)

return StreamingResponse(gen(), media_type="text/event-stream", headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"})


@router.post("/actions/{aid}")
def answer_action(aid: str, body: BrowserAnswer, user: Writer, db: Db) -> Ok:
"""What the browser did with an action: its result, or why it didn't (the person said no, the site isn't allowed...).
404 for an action that isn't one of your browser's, or that's been answered or given up on."""
try:
sid = browser.session_of(db, aid)
_session(db, sid, user)
browser.answer(db, sid, aid, body.ok, body.result, body.error)
except KeyError:
raise HTTPException(404, "not found") from None
return Ok()


@router.post("/sessions/{sid}/run")
def run(sid: str, body: BrowserRun, user: Writer, db: Db, cfg: Cfg) -> BrowserResult:
"""Do one thing in your connected browser, as the assistant's browser_ tools do, and wait for it: for agents and
scripts. The side panel shows it and asks you first, as it does for the assistant. Audited as `browser.run`."""
_on(cfg)
_session(db, sid, user)
auth.audit(db, user.as_audit(), "browser.run", f"browser_session:{sid}", {"tool": body.tool})
try:
out = browser.run(db, sid, user.id, body.tool, body.args, wait=body.wait)
except browser.BrowserError as e:
return BrowserResult(ok=False, error=str(e))
return BrowserResult(ok=True, result=out)
12 changes: 9 additions & 3 deletions fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer
from app.api.media import sign_urls
from app.api.v1.routes.collections import own_collection
from app.domain import ai_tools, auth, auto_scope, chat, llm, store, uploads
from app.domain import ai_tools, auth, auto_scope, browser, chat, llm, store, uploads
from app.domain.store import DB
from app.schemas.chats import (
AnswerCheck,
Expand Down Expand Up @@ -183,7 +183,11 @@ async def send_message(cid: int, body: MessageCreate, user: Writer, acl: Acl, db
from (where, and the highlighted text; not the page's text, which isn't kept).

`/clear` makes the model start afresh (the messages stay to read) and `/compact [what to keep]` folds the
conversation so far into its summary now; each streams cleared or compacted, then its reply as an answer."""
conversation so far into its summary now; each streams cleared or compacted, then its reply as an answer.

With `browser` (the session of a browser's Lens side panel, POST /browser/sessions), the assistant can use that
browser while the side panel is connected: open pages, read them, click, type and download files into the
conversation (docs/browser-assistant.md)."""
c = await run_in_threadpool(_own_chat, db, cid, user)
q = body.content.strip()[:4000]
files = await run_in_threadpool(_attachments, db, body.attachments, user)
Expand Down Expand Up @@ -284,10 +288,12 @@ def answer(on: chat.Answering) -> Iterator[str]:
user.admin,
act=setup,
said=q,
browser=body.browser,
)
try:
answer = ""
steps_max = max(cfg["ai"].get("max_steps") or 6, 12 if setup else 0)
# setting the server up, or working in the person's browser, takes more steps than looking things up
steps_max = max(cfg["ai"].get("max_steps") or 6, 12 if setup else 0, browser.steps(cfg) if box.browser else 0)
for kind, data in chat.tool_answer(cfg, box, asked, past, steps_max, model, setup=setup, summary=summary):
if kind == "step":
steps.append(data)
Expand Down
Loading
Loading