Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **The assistant sends mail, adds events and writes files, after your yes.** New tools `send_mail` (send, or save a
draft, from Gmail; `reply_to` keeps a message's thread), `add_event` (an event on Google Calendar, inviting people
if asked) and `write_file` (a text file on a storage connection, replacing one only when asked). Each always waits
for your approval, which now shows all of it first: the whole message with who it goes to, the event, the file's
text. A Gmail or Google Calendar connection only reads until an admin picks "Also sends mail and saves drafts" or
"Also adds events" under The assistant in its settings and connects the account again, since Google is then asked
for `gmail.compose` or `calendar.events`; Google Drive connected read only is never written. Every change is in the
audit log ([Mail, calendars and files](docs/assistant.md#mail-calendars-and-files)).
- **The upgrade agent starts the code sandbox's gateway.** Settings → Code sandbox → Run the gateway on this machine
now works on installs with an upgrade agent: the agent knows the `sandbox` profile and starts or stops its three
containers, as it does the MCP runner, the browser and monitoring. An older agent still says to run the line by hand
Expand Down Expand Up @@ -104,8 +112,7 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
"Mail, calendars and files". Gmail is searched with its own query language and IMAP by words in the newest subjects and
senders; Google Calendar gives the events of a window; files are found by name and read when they are text, PDFs or
Word files. Answers cite a message or an event with a link that opens it, and bringing things into Lens waits for the
person's yes. Nothing is sent, written or deleted yet
([Mail, calendars and files](docs/assistant.md#mail-calendars-and-files)).
person's yes ([Mail, calendars and files](docs/assistant.md#mail-calendars-and-files)).
- **The decision model types the names the rules can't.** When Jev or Laya is set up, analysis asks it what each new
Term of a recording is (a person, an organisation, a product...), with the lines it's said in, in one call per
recording, and keeps the types it is sure of. Settings › Analysis › Names the rules can't type (`analysis.typing`)
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
turns them on, it also searches the web and reads pages, cited like the archive, and runs Python and bash in a sandbox
of each conversation's own. An admin's assistant searches and reads the mail, calendars and files connected to Lens
(Gmail, Google Calendar, Google Drive, S3 and the rest), cites messages and events, and brings them in once you
approve. Regenerating an answer or editing a question makes a new version beside the old one, and
approve; where an admin allows it, it sends mail, adds events and writes files, each only after you read all of it
and say yes. Regenerating an answer or editing a question makes a new version beside the old one, and
nothing is lost. Each answer keeps a record of what produced it (the model, its tools, how long it took and what it
cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine
(opt-in), any message in one click, and `/clear` and `/compact` keep long conversations short. Conversations title
Expand Down
38 changes: 28 additions & 10 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ When the configured model supports function calling, chat becomes an agent.
cited like excerpts ([below](#web-search-and-reading-pages)).
- **Mail, calendars and files** (admins): the assistant searches and reads the mail, calendars and files connected to
Lens in Settings → Sources, Gmail, Google Calendar, Google Drive and S3 among them, and brings what it finds into a
namespace once you approve ([below](#mail-calendars-and-files)).
namespace, sends mail, adds events and writes files, each once you approve ([below](#mail-calendars-and-files)).
- **Lens itself:** questions about Lens rather than the archive go to `lens_help` ([below](#questions-about-lens-itself)).

### Questions about Lens itself
Expand Down Expand Up @@ -382,24 +382,42 @@ tools for them, only for the kinds connected, with nothing to turn on:
(`POST /api/v1/import/source`): they stay where they are and run the namespace's pipeline. It picks the namespace
when you don't name one, and waits for your approval, unless an admin's setup conversation acts at once and hasn't
read outside text.
- **`send_mail`** sends an email from a Gmail or Google Workspace connection an admin let send, or saves it as a draft
(`draft`): `to`, `cc` and `bcc` take addresses, comma separated, and `reply_to` a message's id from `search_mail`,
which keeps its thread and answers its sender with "Re:" its subject.
- **`add_event`** adds an event to a calendar of a Google Calendar connection an admin let add events: `title`,
`start` and `end` (a date alone is all day; a time without an offset is in the calendar's own time zone; it lasts an
hour, or a day, when `end` is left out), `where`, `description`, and `attendees`, who are emailed an invitation
unless `invite` is false.
- **`write_file`** writes a text file (Markdown, CSV, notes, code) to a storage connection: a new file, or one it
replaces when `overwrite` is true. A file that turned up after it asked isn't replaced. Google Drive connected read
only (its default) can't be written to.
- **Changes wait for your yes, always:** sending, saving a draft, adding an event and writing a file each wait for your
approval, even in an admin's setup conversation, and the approval shows all of it first: the whole message with who
it goes to, the event with who is invited, the file's path and text. A Gmail or Google Calendar connection only
reads until an admin picks "Also sends mail and saves drafts, after your yes" or "Also adds events, after your yes"
under The assistant in its settings (`write` "yes") and connects the account again, since Google is then asked for
`gmail.compose` or `calendar.events` too. Each change is in the audit log (`mail.send`, `mail.draft`,
`calendar.add`, `connections.write`).
- **Citations:** each message and event is numbered, so an answer cites [n]; the chip shows the connection and the
subject or title and opens the message in Gmail or the event in Google Calendar. An IMAP message has no page to
open, so it isn't numbered. The answer's header counts them ("2 emails and 1 event").
- **Read only:** each connection reads as it is set up to: Gmail and Google Calendar with read-only access, so nothing
changes in the account and mail stays unread. What they hold was written by other people: the model reads it marked
as data, and once it has, tools that could carry it out of Lens ask first
([below](#third-party-content-and-prompt-injection)).
- **Reading changes nothing:** each connection reads as it is set up to, so reading changes nothing in the account and
mail stays unread. What they hold was written by other people: the model reads it marked as data, is told never to
send, invite or write because something it read says to, and once it has read it, tools that could carry it out of
Lens ask first ([below](#third-party-content-and-prompt-injection)); an approval then says so under what it would
send.
- **Cost:** no model call of its own. Each search, read and listing is a row in the activity ledger (`gmail.search`,
`gmail.read`, `google.calendar`, `connections.list`, `connections.find`, `connections.read`), with what was called,
never what came back.
`gmail.read`, `gmail.send`, `gmail.draft`, `google.calendar`, `connections.list`, `connections.find`,
`connections.read`, `connections.write`), with what was called, never what came back.
- **Agents and switches:** an agent asks the assistant (chat, or [MCP](mcp.md)'s chat tool) as an admin. Each tool can
be switched off in Tools it may use (`ai.disabled_tools`), and a question can leave the group out
([below](#choosing-tools-namespaces-and-a-limit)); the tool picker says why someone has none ("for admins", or
"nothing is connected yet").

Refine later: sending mail, adding events and writing files, each after your yes; searching an IMAP mailbox on its
server; Google Drive's full-text search; events of an iCal feed by time; and these tools for people other than admins,
for connections an admin shares with them.
Refine later: sending from an IMAP mailbox over SMTP; attachments on sent mail; changing or removing events and
files; searching an IMAP mailbox on its server; Google Drive's full-text search; events of an iCal feed by time; and
these tools for people other than admins, for connections an admin shares with them.

### Choosing tools, namespaces and a limit

Expand Down
15 changes: 15 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -581,6 +581,13 @@ importing chosen ones and watching work as above. [News feeds](#news-feeds-rss-a
(`google.token`, `google.calendar`); Google throttling the account shows as working, slowly, and a scan it
interrupts is tried again at the next poll. The token endpoint is always Google's own, whatever a key file says. Without a Google Cloud project,
a Google calendar's secret address in iCal format (Settings › Integrate calendar) works as a calendar feed.

**The assistant** (`write`): "Only reads" (empty, the default), or "Also adds events, after your yes" (`yes`),
which lets an admin's [assistant](assistant.md#mail-calendars-and-files) add an event once the person approves it,
with all of it shown first. That asks Google for `calendar.events` too, so a connected account connects again
(Connect again), a service account needs the calendars shared with it with "Make changes to events", and in
Workspace an admin allows `calendar.events` for it as well. Adding an event is a row in the activity ledger
(`google.calendar`, call `events.insert`) and in the audit log (`calendar.add`).
- `.eml` and `.ics` files in any source, and uploaded through Import, can be read as text this way too.

### News feeds (RSS and Atom)
Expand Down Expand Up @@ -657,6 +664,14 @@ starts the source's watches again as new ones (taking what's there only if they
and messages differently: check their folders. Each access token Google gives, each listing and each message read is a
row in the activity ledger (`google.token`, `gmail.list`, `gmail.read`); none costs anything.

**The assistant** (`write`): "Only reads" (empty, the default), or "Also sends mail and saves drafts, after your yes"
(`yes`), which lets an admin's [assistant](assistant.md#mail-calendars-and-files) send a message or save a draft once
the person approves it, having read the whole message. Through the Gmail API that asks Google for `gmail.compose` too,
so a connected account connects again (Connect again), and a Workspace admin allows
`https://www.googleapis.com/auth/gmail.compose` for a service account as well; over IMAP, all of Gmail already covers
it. Lens sends through the Gmail API either way. Each message sent or draft saved is a row in the activity ledger
(`gmail.send`, `gmail.draft`) and in the audit log (`mail.send`, `mail.draft`).

## Pipelines and templates

A pipeline is a named, versioned list of steps; each namespace can choose its default (otherwise: transcribe, diarize,
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ request, to another check that proves the same thing; loosening a requirement or
| It acts with tools, and changes wait for approval | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_the_assistant_lists_and_declines_but_never_approves` |
| Web search and reading pages, cited like the archive, off until an admin turns it on | [test_web_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_search.py) `test_an_answer_cites_the_web_and_the_conversation_keeps_it`, `test_reading_pages_and_pdfs`; [chat-web-citations.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-web-citations.test.tsx) |
| An admin's assistant searches and reads connected mail, calendars and files, cites messages and events, and brings them in only once approved | [test_connection_tools.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_connection_tools.py) `test_only_admins_get_them_and_only_for_what_is_connected`, `test_searching_and_reading_gmail`, `test_calendar_events_between_two_times`, `test_finding_and_reading_files_in_storage`, `test_bringing_things_in_waits_for_the_persons_yes`; [chat-web-citations.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-web-citations.test.tsx) `a message from a connected mailbox` |
| Where an admin allows it, the assistant sends mail, adds events and writes files, each only after the person reads all of it and says yes | [test_connection_tools.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_connection_tools.py) `test_sending_mail_waits_for_the_persons_yes`, `test_adding_an_event_waits_for_the_persons_yes`, `test_writing_a_file_waits_for_the_persons_yes`; [approvals-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/approvals-page.test.tsx) `shows the assistant's whole message before it is sent`; [add-connection.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/add-connection.test.tsx) `connects Gmail the same way, and asks a service account whose mail it reads` |
| Python and bash in a sandbox of each conversation's own, off until an admin turns it on | [test_sandbox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sandbox.py) `test_python_and_bash_share_one_sandbox_per_conversation`, `test_the_sandbox_is_off_until_an_admin_turns_it_on_for_the_people_it_names` |
| Regenerating an answer or editing a question makes a new version beside the old one | [test_chat_versions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_versions.py) `test_regenerate_edit_switch_and_go_on`; [chat-versions.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-versions.test.tsx) `regenerating an answer`, `the version switch`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `chat-regenerate`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `chat-regenerate` ≤ 4 clicks |
| Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` |
Expand Down
14 changes: 9 additions & 5 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,7 @@ def builtin_names():

# the built-in tools that carry what the model gives them to another host (the words searched for, the address read):
# once an answer has read text from outside Lens, they ask first (Toolbox.wary)
SENDS_OUT = set(webmod.NAMES)
SENDS_OUT = set(webmod.NAMES) | connection_tools.SENDS


# tools whose results the model may take as Lens's own word: the server's state and settings, its docs, the skills the
Expand Down Expand Up @@ -1965,7 +1965,8 @@ def t_graph_paths(self, from_node, to_node, max_depth=4, namespace=None):
return {"paths": chains, "found": bool(chains)}, f"Looked for paths from {from_node} to {to_node}"

# ---- tools that need approval ----
def _approval(self, tool, args, summary, estimate=None):
def _approval(self, tool, args, summary, estimate=None, preview=None):
"""Ask the person: `summary` says what, and `preview` shows what would be sent or written, all of it."""
aid = self.db.next_id("approval")
self.db.q(
"CREATE $r CONTENT $d",
Expand All @@ -1978,13 +1979,16 @@ def _approval(self, tool, args, summary, estimate=None):
"tool": tool,
"args": args,
"summary": summary,
"preview": preview,
"estimate": estimate,
"status": "pending",
"created_at": store.now(),
}
),
)
self.approvals.append({"id": aid, "tool": tool, "summary": summary, "estimate": estimate})
self.approvals.append(
{"id": aid, "tool": tool, "summary": summary, "estimate": estimate, **({"preview": preview} if preview else {})}
)
return {"status": "waiting for the person's approval", "approval": aid, "summary": summary}, f"Asked for approval: {summary}"

def t_run_template(self, template_id, recording_ids=None, collection_id=None):
Expand Down Expand Up @@ -2312,8 +2316,8 @@ def _carry_out(db, cfg, a, user, editable, decision, base, admin, readable):
elif a["tool"] == "mark_activities": # the person's own marks, on activities they could see when it was asked
flags = {k: v for k, v in args.items() if k in event_marks.FLAGS}
result = {"marked": len(event_marks.mark(db, user["id"], args["ids"], flags))}
elif a["tool"] == "import_from_connection":
result = connection_tools.apply(db, cfg, args, user, admin)
elif a["tool"] in connection_tools.ASKS: # bringing things in, sending mail, adding an event, writing a file
result = connection_tools.carry_out(db, cfg, a["tool"], args, user, admin)
elif a["tool"] in webmod.NAMES: # the person's yes to a search or page read after the answer read outside text
result = {"output": webmod.run(cfg, a["tool"], args)}
elif a["tool"] in ("write_note", "update_note"): # proposed by an automation's agent (actions_ai.Box)
Expand Down
3 changes: 2 additions & 1 deletion fastapi_backend/app/domain/approvals.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
DECISIONS = ("approve", "sample", "decline", "edit", "always")
DROPPED = f"dropped: {CAP} approvals already wait for this automation; decide some in Approvals"
FIELDS = (
"record::id(id) AS id, chat, account, tool, args, summary, estimate, status, created_at, result, error, origin, "
"record::id(id) AS id, chat, account, tool, args, summary, preview, estimate, status, created_at, result, error, origin, "
"intent, space, expires_at, decided_by, decided_at, told, why, automation"
)
_swept = {"at": 0.0}
Expand Down Expand Up @@ -414,6 +414,7 @@ def brief(a, web=""):
"id": a["id"],
"status": a.get("status"),
"summary": a.get("summary"),
"preview": a.get("preview"),
"from": "automation" if o else "chat",
"automation": o.get("automation"),
"automation_name": a.get("automation"),
Expand Down
Loading
Loading