Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
only the version showing. In the web app, Regenerate (and Regenerate with… another model) replaces Try another
model, and `‹ 2 / 3 ›` under a question or an answer moves between its versions. Existing conversations are upgraded
on the next start (`chat-tree`) ([Versions](docs/assistant.md)).
- **The assistant searches the web.** With web search on (off by default; Settings → AI assistant → Web search), the
assistant gets `web_search` and `web_read`: it searches with DuckDuckGo (no key), your own SearXNG, Brave Search or
Tavily, and reads pages and online PDFs, public addresses only, through the same guard as captured pages. Pages are
cited [n] like excerpts, with their site, and open in a new tab; each search and read is in the activity ledger, and
once an answer has read outside text, one it chose itself asks first
([Web search and reading pages](docs/assistant.md#web-search-and-reading-pages)).
- **Add an MCP server with a form, or paste its config.** Extensions → Add MCP server (admins) takes an address and
its headers, or a command with its arguments and environment, or reads a config pasted from Claude (`.mcp.json`) or
VS Code, one server or many; Test lists its tools before it's saved. Tokens and keys in headers and env are now
Expand Down
4 changes: 2 additions & 2 deletions docs/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ its namespace, its pipeline and the routine. A request counts for the resources
/api/v1/routines/3` is `routine:3`), for what a `POST` made (from the `id` in its reply), and for who sent it.

Besides the resources, a row has its kind (`in`, `out` or `run`), what it was (`POST /api/v1/routines/{rid}`,
`model.chat`, `embeddings`, `decision`, `notify.webhook`, `tool.http`, `google.token`, `gmail.list`, `job.succeeded`,
`routine.done`), the model,
`model.chat`, `embeddings`, `decision`, `notify.webhook`, `tool.http`, `web.search`, `web.read`, `google.token`,
`gmail.list`, `job.succeeded`, `routine.done`), the model,
tokens in and out, the estimated cost in USD, how long it took, and how it ended (an error's type or HTTP status,
never its message). It never holds prompt, reply, asset text or file content.

Expand Down
50 changes: 49 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,54 @@ When the configured model supports function calling, chat becomes an agent.
- **Your browser** (Chrome and Edge): asked from the Lens extension's side panel, the assistant can open, read, click
and type in that browser and download files into the conversation, on sites you allow, asking before anything that
sends, deletes or buys. See [The assistant in your browser](browser-assistant.md).
- **The web:** when an admin turns on web search, `web_search` and `web_read` look up what the archive doesn't hold,
cited like excerpts ([below](#web-search-and-reading-pages)).

### Web search and reading pages

Built, off by default. An admin turns it on in Settings → AI assistant → Web search (`web.search`), and the assistant
gets two more tools, for everyone who may use the assistant:

- **`web_search`** searches the web: each result's title, address, site and snippet, numbered like the archive's
excerpts, so an answer cites [n] across both. `time_range` keeps only results from the last day, week, month or
year. The step reads 'Searched the web for "solar output": 5 result(s)'.
- **`web_read`** reads a page or an online PDF by its address: its title and text, numbered to cite. `find` keeps only
the paragraphs that mention some words, with the ones around them. It doesn't save the page; `import_urls` captures
it into the archive.
- **Citations:** a cited page's chip shows its site ("example.org · Harbour news") and opens the page in a new tab;
the sources list shows it with its site, and a conversation keeps its web citations when it is opened again (they
aren't the archive's, so nobody's reads hide them). In [chat rooms](chat-rooms.md) the sources line gives the
page's address.
- **Providers** (`web.search`): `duckduckgo` needs no key or account. DuckDuckGo has no API, so Lens reads its results
page with [ddgs](https://pypi.org/project/ddgs/) (MIT), an unofficial client that can stop working when the page
changes; it is about 30 MB and is fetched the first time an admin picks it (Settings → Components shows it, as
"DuckDuckGo search"). `searxng` asks your own [SearXNG](https://docs.searxng.org/) instance at `web.searxng_url`,
whose JSON output must be on (`search.formats: [html, json]` in its `settings.yml`). `brave` and `tavily` use those
services' APIs with the key in `web.api_key`, kept encrypted like other secrets; searches past their free
allowance are charged by them.
- **What leaves the server:** what it searches for goes to the provider, and the pages it reads come straight from
their sites. Like [captured pages](api.md#web-pages), every request goes through Lens's network guard: public
addresses only (plus the networks allowed for pages, `documents.web_networks`), on the web's ports, every redirect
checked again, so neither a result nor a redirect reaches this machine's own services. Pages and PDFs are read up
to 10 MB, and the model reads at most `web.read_chars` of a page's text (20,000 characters by default; `find` picks
the parts that matter in a longer page).
- **Pages that need scripts:** with `web.render` on (Settings → AI assistant → Web search → Reading pages), a page
that comes back nearly empty is loaded in Chromium, its scripts run, and read again. That needs Chromium on the
server (the full edition); the light install reads the page as it comes.
- **Asks first after outside text:** a search or a page read sends words out of Lens, so once an answer has read text
from outside Lens ([below](#third-party-content-and-prompt-injection)), one the model wrote itself waits for your
approval. Searching for what you typed, and reading an address you wrote or that a search or page gave back, run at
once. Once approved, the assistant asks the same thing again and gets what the approved call found, without
searching twice.
- **Cost:** each search and each page read is a row in the activity ledger (`web.search` with the provider and how
many results came back, `web.read` with the site), never with the query or the page. Brave and Tavily searches are
marked as estimates, since Lens doesn't know your plan's price.
- **Agents:** an agent turns it on like any setting, with `PUT /api/v1/settings/web` (admins) or by asking the
[setup chat](#setting-up-and-running-the-server-by-chat), which waits for your approval before it changes web search.
Each tool can be switched off in Tools it may use (`ai.disabled_tools`).

Refine later: an opt-in SearXNG service in the Docker Compose file (SearXNG is AGPL, so it would stay an extra),
prices per search for the paid providers, web access per namespace, and keeping recent results to search less.

## Third-party content and prompt injection

Expand Down Expand Up @@ -189,7 +237,7 @@ apart from what the person asks:
other than Lens's own, the page it was asked from, the namespace assistant's memories, a message in a chat room
open to anyone, or an earlier answer in the conversation that read any of these), anything that could carry what
it read out of Lens waits for your approval, even where it would otherwise run at once: your own `read` web tools,
other servers' MCP tools and Python tools with the network, the changes a setup conversation makes itself, and, in
a [web search or page read](#web-search-and-reading-pages) you or a tool didn't give it, other servers' MCP tools and Python tools with the network, the changes a setup conversation makes itself, and, in
[your browser](browser-assistant.md), opening or downloading an address or typing text you didn't write. The
approval card says why ("asked first: this answer read text from outside Lens: tool search_transcripts"). Notes and
memories are still written at once: they stay in Lens, and are marked as data when read back.
Expand Down
1 change: 1 addition & 0 deletions docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ never fetch a speech or voice model.
| Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` |
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| DuckDuckGo search | `web.search` is `duckduckgo` | ddgs (MIT, about 30 MB with lxml and primp), the client the assistant's [web search](assistant.md#web-search-and-reading-pages) reads DuckDuckGo's results with |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
| Outlook .msg emails | listed in `components.also` | extract-msg (GPL-3.0, so only when asked for) |
Expand Down
14 changes: 14 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,20 @@ People can choose the model a conversation uses, and ask a question again with a
offered and stays the default. Left empty, they may pick whatever the model server lists (`GET /models`, kept for a
minute); on providers that list many models, or charge by model, list the ones you want offered.

## Web search

Whether the assistant searches the web and reads pages ([Web search and reading pages](assistant.md#web-search-and-reading-pages)).
Settings → AI assistant → Web search; the setup chat always asks before it changes these:

| Setting | Default | |
|---|---|---|
| `web.search` | `off` | the search provider: `off`, `duckduckgo` (no key; fetches the ddgs client, about 30 MB, when picked), `searxng` (your own instance), `brave` or `tavily` (an API key) |
| `web.searxng_url` | null | your SearXNG instance's address, such as `http://searxng:8080`, with its JSON output on; asked directly, so it may be on your network |
| `web.api_key` | none | the Brave Search or Tavily API key, stored encrypted |
| `web.results` | 5 | results per search when the model doesn't say, 1–20 |
| `web.read_chars` | 20000 | the most of a page's text the model reads at once, 1,000–200,000 characters |
| `web.render` | false | load a page that comes back nearly empty in Chromium, scripts run, and read it again (needs Chromium: the full edition) |

## Descriptions

The describe step ([API](api.md#descriptions)) sends each page of a document or an image, and each shot's keyframe of
Expand Down
7 changes: 4 additions & 3 deletions fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ def create_chat(user: Writer, acl: Acl, db: Db, cfg: Cfg, body: ChatCreate | Non
@router.get("/chats/{cid}")
def get_chat(cid: int, user: CurrentUser, acl: Acl, db: Db) -> Chat:
"""A conversation with the version showing (`messages`, each with its `versions`) and every version's messages
(`tree`). Citations follow your current access: ones you can no longer read are left out."""
(`tree`). Citations follow your current access: ones you can no longer read are left out (web pages it cited stay:
they aren't the archive's)."""
return _chat_view(db, acl, _own_chat(db, cid, user))


Expand All @@ -135,7 +136,7 @@ def _chat_view(db: DB, acl: Access, c: dict[str, Any]) -> Chat:
msgs, ok, versions = chat.path(db, c["id"], c.get("head"), nodes), set(acl.roles), chat.versions(nodes)
for m in msgs:
m["versions"] = versions.get(m["id"])
ids = {p["recording_id"] for m in msgs for p in m.get("passages") or []}
ids = {p["recording_id"] for m in msgs for p in m.get("passages") or [] if p.get("recording_id") is not None}
visible = (
{
r["id"]
Expand All @@ -146,7 +147,7 @@ def _chat_view(db: DB, acl: Access, c: dict[str, Any]) -> Chat:
else set()
)
for m in msgs:
m["passages"] = [p for p in m.get("passages") or [] if p["recording_id"] in visible]
m["passages"] = [p for p in m.get("passages") or [] if p.get("source") == "web" or p.get("recording_id") in visible]
return Chat.model_validate(sign_urls({**c, "messages": msgs, "tree": nodes}, full=True))


Expand Down
85 changes: 83 additions & 2 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@
topics,
untrusted,
uploads,
web as webmod,
webcapture,
wiki,
)

Expand Down Expand Up @@ -458,7 +460,12 @@
def builtin_names():
"""The assistant's own tool names, which extensions can't take."""
every = TOOLS + ops_tools.ADMIN_TOOLS + ops_tools.FILE_TOOLS + AUTHOR_TOOLS + MEMORY_TOOLS + AUTOMATION_TOOLS + browsermod.TOOLS
return {t[0] for t in every} | {"use_skill"}
return {t[0] for t in every + webmod.TOOLS} | {"use_skill"}


# the built-in tools that carry what the model gives them to another host (the words searched for, the address read):
# once an answer has read text from outside Lens, they ask first (Toolbox.wary)
SENDS_OUT = set(webmod.NAMES)


# tools whose results the model may take as Lens's own word: the server's state and settings, the skills the person
Expand Down Expand Up @@ -543,11 +550,14 @@ def __init__(
# the browser the person is talking from, while its side panel is connected (and an admin hasn't turned it off)
on = browser and browsermod.enabled(cfg) and browsermod.live(db, browser, user["id"])
self.browser = str(browser) if on else None
self.web = webmod.enabled(cfg) # web search and page reading, once an admin picked a provider (web.py)

def system_note(self):
"""What the model is told besides its usual instructions: the skills it can follow, and context that hooks add
for this question."""
out = self._home_note() + (browsermod.NOTE if self.browser else "")
if self.web and webmod.NAMES & {s["function"]["name"] for s in self.specs()}:
out += webmod.NOTE
if not self.ext:
return out
out += self.ext.system_note()
Expand Down Expand Up @@ -619,6 +629,8 @@ def specs(self):
tools += [t for t in MEMORY_TOOLS if self.home[0] in self.editable or not t[4]]
if self.browser:
tools += browsermod.TOOLS
if self.web:
tools += webmod.TOOLS
if self.ext:
tools += self.ext.tool_specs(can_act or self.admin)
return [
Expand Down Expand Up @@ -816,6 +828,69 @@ def t_browser_import(self, ref=None, url=None, tab_id=None, page=None):
attached = {"upload_id": uid, "filename": up["filename"], "size": up["size"], "next": "import_files puts it in a namespace"}
return attached, f"Attached {up['filename']} from the browser"

# ---- the web (web.py): searching it and reading pages, once an admin picked a search provider ----
def _web(self, tool, args, what, own):
"""A web search or page read, as (what it found, None), or (None, the approval it waits for): the words or the
address leave the machine, so once this answer has read text from outside Lens, which may have put them there,
it asks first, unless the person wrote them (`own`) or already approved the same call in this conversation."""
if own or not self.wary():
return webmod.run(self.cfg, tool, args), None
rows = (
self.db.rows(
"SELECT id, args, result FROM approval WHERE chat = $c AND account = $a AND tool = $t AND status = 'done' ORDER BY id DESC LIMIT 20",
c=self.chat,
a=self.account,
t=tool,
)
if self.chat is not None
else []
)
for a in rows:
got = a.get("result")
if webmod.same_call(tool, a.get("args"), args) and isinstance(got, dict) and "output" in got:
return got["output"], None
return None, self._approval(tool, args, what + self.why())

def web_ref(self, url, title, site, text, fetched_at=None):
"""A web page or search result to cite as [n] (web.py): its address instead of an asset's moment. The same
address keeps its number, with the page's own text once it's read."""
for r in self.refs:
if r.get("source") == "web" and r.get("url") == url:
r.update(
store.clean(
{"title": title or r.get("title"), "text": text or r.get("text"), "fetched_at": fetched_at or r.get("fetched_at")}
)
)
return r["n"]
n = len(self.refs) + 1
self.refs.append(
store.clean({"n": n, "source": "web", "url": url, "title": title, "site": site, "fetched_at": fetched_at, "text": text})
)
return n

def t_web_search(self, query, limit=None, time_range=None):
args = store.clean({"query": " ".join(str(query or "").split()), "limit": limit, "time_range": time_range})
out, asked = self._web("web_search", args, f"Search the web for “{args['query'][:120]}”", self.said_it(args["query"]))
if asked:
return asked
when = store.now()
found = [
{"ref": self.web_ref(r["url"], r.get("title"), r.get("site"), r.get("snippet") or r.get("title"), when), **r}
for r in out["results"]
]
return {"query": out["query"], "results": found}, webmod.summary("web_search", args, out)

def t_web_read(self, url, find=None):
args = store.clean({"url": str(url or "").strip(), "find": find})
own = args["url"] in self.links or self.said_it(args["url"]) # an address a tool gave back carries nothing it read
out, asked = self._web("web_read", args, f"Read the web page {webcapture.placeholder(args['url'])[:160]}", own)
if asked:
return asked
excerpt = " ".join(out.get("text", "").split())[:600]
return {"ref": self.web_ref(out["url"], out.get("title"), out.get("site"), excerpt, out.get("fetched_at")), **out}, webmod.summary(
"web_read", args, out
)

# ---- extensions, from chat ----
def _ext_named(self, name):
for g in extensions.visible(self.db, self.me):
Expand Down Expand Up @@ -899,7 +974,11 @@ def _run(self, name, fn, args):
except Exception as e: # noqa: BLE001 - the model hears what went wrong instead of the answer breaking off
log.exception("assistant tool %s failed", name)
return json.dumps({"error": f"the tool failed: {e}"}), f"{name} failed"
return json.dumps(out, ensure_ascii=False, default=str)[:12000], summary
return json.dumps(out, ensure_ascii=False, default=str)[: self._most(name)], summary

def _most(self, name):
"""The most of a tool's result the model reads: a web page read takes what web.read_chars allows."""
return max(12000, int(webmod.read_chars(self.cfg) * 1.2) + 4000) if name == "web_read" else 12000

# ---- read tools ----
def t_search_transcripts(self, query, namespace=None, limit=8):
Expand Down Expand Up @@ -1725,6 +1804,8 @@ def _carry_out(db, cfg, a, user, editable, decision, base, admin, readable):
elif a["tool"] == "mark_activities": # the person's own marks, on activities they could see when it was asked
flags = {k: v for k, v in args.items() if k in event_marks.FLAGS}
result = {"marked": len(event_marks.mark(db, user["id"], args["ids"], flags))}
elif a["tool"] in webmod.NAMES: # the person's yes to a search or page read after the answer read outside text
result = {"output": webmod.run(cfg, a["tool"], args)}
elif a["tool"] in ("write_note", "update_note"): # proposed by an automation's agent (actions_ai.Box)
box = Toolbox(db, cfg, user, readable if readable is not None else editable, editable, None, a["chat"], base, admin)
result = getattr(box, "t_" + a["tool"])(**args)[0]
Expand Down
Loading
Loading