Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
stream could wait for a database connection its own thread already held, and every request after it waited too.
A query made inside another on the same thread now runs on that connection
([Concurrency](docs/database.md#concurrency)).
- **Mail, calendars and files in chat.** An admin's assistant can search and read the mailboxes, calendars and
storage they connected under Settings → Sources, through the new tools `connections`, `search_mail`, `read_mail`,
`calendar_events`, `list_files`, `find_files`, `read_file` and `import_from_connection`; the tool picker shows them as
"Mail, calendars and files". Gmail is searched with its own query language and IMAP by words in the newest subjects and
senders; Google Calendar gives the events of a window; files are found by name and read when they are text, PDFs or
Word files. Answers cite a message or an event with a link that opens it, and bringing things into Lens waits for the
person's yes. Nothing is sent, written or deleted yet
([Mail, calendars and files](docs/assistant.md#mail-calendars-and-files)).
- **The decision model types the names the rules can't.** When Jev or Laya is set up, analysis asks it what each new
Term of a recording is (a person, an organisation, a product...), with the lines it's said in, in one call per
recording, and keeps the types it is sure of. Settings › Analysis › Names the rules can't type (`analysis.typing`)
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,17 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
- **Ask, and get cited answers.** A chat assistant on every page, and by voice, searches your archive by words (and by
meaning, once turned on), reads it, cites it and acts on it with tools: changes wait for your approval. Once an admin
turns them on, it also searches the web and reads pages, cited like the archive, and runs Python and bash in a sandbox
of each conversation's own. Regenerating an answer or editing a question makes a new version beside the old one, and
of each conversation's own. An admin's assistant searches and reads the mail, calendars and files connected to Lens
(Gmail, Google Calendar, Google Drive, S3 and the rest), cites messages and events, and brings them in once you
approve. Regenerating an answer or editing a question makes a new version beside the old one, and
nothing is lost. Each answer keeps a record of what produced it (the model, its tools, how long it took and what it
cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine
(opt-in), any message in one click, and `/clear` and `/compact` keep long conversations short. Conversations title
themselves after their first answer, unless you name them. Each namespace can have its own assistant with
its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in
Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you
email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages),
[code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records),
[mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records),
[chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md);
[checks](docs/requirements.md#ask-and-get-cited-answers))
- **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python),
Expand Down
48 changes: 48 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,9 @@ When the configured model supports function calling, chat becomes an agent.
conversation's own, to compute, analyse or convert files. See [Code sandbox](#code-sandbox).
- **The web:** when an admin turns on web search, `web_search` and `web_read` look up what the archive doesn't hold,
cited like excerpts ([below](#web-search-and-reading-pages)).
- **Mail, calendars and files** (admins): the assistant searches and reads the mail, calendars and files connected to
Lens in Settings → Sources, Gmail, Google Calendar, Google Drive and S3 among them, and brings what it finds into a
namespace once you approve ([below](#mail-calendars-and-files)).
- **Lens itself:** questions about Lens rather than the archive go to `lens_help` ([below](#questions-about-lens-itself)).

### Questions about Lens itself
Expand Down Expand Up @@ -353,6 +356,51 @@ gets two more tools, for everyone who may use the assistant:
Refine later: an opt-in SearXNG service in the Docker Compose file (SearXNG is AGPL, so it would stay an extra),
prices per search for the paid providers, web access per namespace, and keeping recent results to search less.

### Mail, calendars and files

Built. The accounts and storage connected to Lens in Settings → Sources are an admin's, so an admin's assistant gets
tools for them, only for the kinds connected, with nothing to turn on:

- **`connections`** lists them: each one's id, name, what it holds (mail, calendar, files or feed), its account,
whether it works, and the tools that read it. The other tools take a connection by its id or name, and leave it out
when there's only one of its kind.
- **`search_mail`** searches a mailbox, newest first: who sent each message to whom, its subject, date, labels and a
snippet. A Gmail or Google Workspace connection takes Gmail's own search words (`from:ana subject:invoice
after:2026/10/01 has:attachment is:unread label:work`, and plain words); an IMAP mailbox matches the words in the
subject and sender of the newest messages in its inbox. With no words, it lists the newest in the inbox.
- **`read_mail`** reads one message in full: who sent it to whom and when, its attachments by name, and its text (up
to 8,000 characters, `max_chars` up to 11,000), by its id from `search_mail` or its path from `list_files`.
- **`calendar_events`** lists a Google Calendar connection's events between two times (`from` and `to`, ISO 8601
dates or times; the next 7 days by default), each time a repeating event happens, optionally only those with some
words: title, when, where, organizer, who's invited and their answers, video call, and the start of its
description. It says the time now and the span it read; a date or time without an offset is taken as UTC.
- **`list_files`** shows a folder of any connection: a bucket's or drive's folders and files, a mailbox's folders or
labels and their newest messages, a calendar's events. **`find_files`** finds files whose names contain some words,
any case, in a storage connection's folder and up to 12 folders below it. **`read_file`** reads a storage file's
text: plain text, Markdown, CSV, JSON, code, HTML, email, calendar, PDF or Word, up to 20 MB.
- **`import_from_connection`** brings files, messages or events into a namespace, as Import → From a connection does
(`POST /api/v1/import/source`): they stay where they are and run the namespace's pipeline. It picks the namespace
when you don't name one, and waits for your approval, unless an admin's setup conversation acts at once and hasn't
read outside text.
- **Citations:** each message and event is numbered, so an answer cites [n]; the chip shows the connection and the
subject or title and opens the message in Gmail or the event in Google Calendar. An IMAP message has no page to
open, so it isn't numbered. The answer's header counts them ("2 emails and 1 event").
- **Read only:** each connection reads as it is set up to: Gmail and Google Calendar with read-only access, so nothing
changes in the account and mail stays unread. What they hold was written by other people: the model reads it marked
as data, and once it has, tools that could carry it out of Lens ask first
([below](#third-party-content-and-prompt-injection)).
- **Cost:** no model call of its own. Each search, read and listing is a row in the activity ledger (`gmail.search`,
`gmail.read`, `google.calendar`, `connections.list`, `connections.find`, `connections.read`), with what was called,
never what came back.
- **Agents and switches:** an agent asks the assistant (chat, or [MCP](mcp.md)'s chat tool) as an admin. Each tool can
be switched off in Tools it may use (`ai.disabled_tools`), and a question can leave the group out
([below](#choosing-tools-namespaces-and-a-limit)); the tool picker says why someone has none ("for admins", or
"nothing is connected yet").

Refine later: sending mail, adding events and writing files, each after your yes; searching an IMAP mailbox on its
server; Google Drive's full-text search; events of an iCal feed by time; and these tools for people other than admins,
for connections an admin shares with them.

### Choosing tools, namespaces and a limit

By default (Auto) an answer may use every tool the person has. A conversation, or one question, can instead pick the
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Answers cite their sources, and only what the person may read | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_chat_cites_only_what_you_can_read`; [chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat.test.tsx) `splits [n], [n, m] and [n][m]` |
| It acts with tools, and changes wait for approval | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_the_assistant_lists_and_declines_but_never_approves` |
| Web search and reading pages, cited like the archive, off until an admin turns it on | [test_web_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_search.py) `test_an_answer_cites_the_web_and_the_conversation_keeps_it`, `test_reading_pages_and_pdfs`; [chat-web-citations.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-web-citations.test.tsx) |
| An admin's assistant searches and reads connected mail, calendars and files, cites messages and events, and brings them in only once approved | [test_connection_tools.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_connection_tools.py) `test_only_admins_get_them_and_only_for_what_is_connected`, `test_searching_and_reading_gmail`, `test_calendar_events_between_two_times`, `test_finding_and_reading_files_in_storage`, `test_bringing_things_in_waits_for_the_persons_yes`; [chat-web-citations.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-web-citations.test.tsx) `a message from a connected mailbox` |
| Python and bash in a sandbox of each conversation's own, off until an admin turns it on | [test_sandbox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sandbox.py) `test_python_and_bash_share_one_sandbox_per_conversation`, `test_the_sandbox_is_off_until_an_admin_turns_it_on_for_the_people_it_names` |
| Regenerating an answer or editing a question makes a new version beside the old one | [test_chat_versions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_versions.py) `test_regenerate_edit_switch_and_go_on`; [chat-versions.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-versions.test.tsx) `regenerating an answer`, `the version switch`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `chat-regenerate`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `chat-regenerate` ≤ 4 clicks |
| Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` |
Expand Down
3 changes: 2 additions & 1 deletion fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@
from app.schemas.common import Created, Ok

router = APIRouter(tags=["chats"])
OUTSIDE = {"web", "docs", "server"} # citations that aren't the archive's: web pages, Lens's docs, this server's facts
# citations that aren't the archive's: web pages, Lens's docs, this server's facts, a connection's messages and events
OUTSIDE = {"web", "docs", "server", "mail", "event"}
R = store.R
log = logging.getLogger("lens")
SSE_HEADERS = {"Cache-Control": "no-store", "X-Accel-Buffering": "no"}
Expand Down
Loading
Loading