Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,13 +87,14 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
nothing is lost. Each answer keeps a record of what produced it (the model, its tools, how long it took and what it
cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine
(opt-in), any message in one click. Long conversations are summarised before they fill the model's context, and
`/clear` and `/compact` start afresh or summarise now. Conversations title themselves after their first answer, unless
`/clear` and `/compact` start afresh or summarise now. It answers questions about Lens itself (how to back up,
whether Telegram is on) from its docs and this server, citing both. Conversations title themselves after their first answer, unless
you name them. Each namespace can have its own assistant with
its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in
Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you
email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages),
[mail, calendars and files](docs/assistant.md#mail-calendars-and-files), [code sandbox](docs/assistant.md#code-sandbox), [answer records](docs/assistant.md#what-each-answer-records),
[chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md);
[questions about Lens](docs/assistant.md#questions-about-lens-itself), [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md), [speech providers](docs/speech-providers.md);
[checks](docs/requirements.md#ask-and-get-cited-answers))
- **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python),
skills, hooks and MCP servers, by code, on the canvas or just by asking. Remote MCP servers connect by address, and
Expand Down Expand Up @@ -167,8 +168,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
content types or topics, acting only on a sure answer, and an Entity filter node keeps the entities you want. Anything that reaches outside Lens or changes more than what
set it off asks first, until you let it run on its own: on the Approvals page or, for a change inside Lens, with a yes
in the chat it told you in; agents can decline an approval but never give one. Each automation has limits and budgets,
and agents make, try and run them over MCP and in the assistant, or draft one from a sentence; the studio to draw them
in the web app is planned. ([Processing](docs/processing.md#workflows-and-the-canvas),
and agents make, try and run them over MCP and in the assistant, or draft one from a sentence. An automation's
workflow is drawn and tried on the workflow canvas, with its actions in the palette; the studio's builder is
planned. ([Processing](docs/processing.md#workflows-and-the-canvas),
[converting assets](docs/processing.md#converting-assets), [notifications](docs/notifications.md),
[automations](docs/automations.md),
[approvals](docs/automations.md#approvals), [timeline](docs/timeline.md); [checks](docs/requirements.md#automate-it))
Expand Down Expand Up @@ -238,8 +240,6 @@ still loading. ([Components](docs/components.md),

These are open pull requests, not yet part of Lens:

- **The assistant** answering questions about Lens itself, from its docs and this server
([#400](https://github.com/adeelahmad/lense/pull/400)).
- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
- **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
Expand All @@ -252,7 +252,8 @@ These are open pull requests, not yet part of Lens:

Next, and not in a pull request yet; each links to where it is tracked:

- **The automations studio:** drawing automations in the web app ([automations](docs/automations.md)).
- **The automations studio:** a When, If and Then builder for automations, with their runs and costs, in the web app
([automations](docs/automations.md)).
- **Timeline:** what each item caused, and Approve and Decline on waiting approvals ([timeline](docs/timeline.md)).
- **Notes:** nesting a note under its project or area without a click, pages in the graph, and wiki addresses for
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
Expand Down
15 changes: 14 additions & 1 deletion docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ merged is called built.
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
| The event workflow scope: what an automation does, drawn with action nodes and tried on an activity (`app/domain/automation_nodes.py`) | built |
| The decide node: the decision model picks a branch, one port per option and `unsure` ([The decide node](#the-decide-node)) | built |
| Drawing an automation's workflow on the canvas: the actions in the palette, their settings from each action's params, Decide's ports, the leaf rule and a try on a kind's sample ([On the canvas](#on-the-canvas)) | built |
| The action registry with risk from each action's effect, and one action, `log_line` (`app/domain/actions.py`) | built |
| Telling a person: the `notify` action and `channels.tell` (their own chat, a notification target, a Matterbridge room) | built |
| The model and the agent: the `ask_model`, `call_tool` and `ask_agent` actions, and prompts with outside text marked | built |
Expand Down Expand Up @@ -567,6 +568,19 @@ lists each one with its params, effect, risk, status (`done`, `waiting`, `schedu
back. Text from outside Lens (`lens.untrusted` on the activity, or what a node passed on when its output says
`untrusted`, like a web page's answer) that reaches an action's params marks its intent `untrusted`.

#### On the canvas

Built: an event workflow opens on the workflow canvas (`/workflows/<id>`) like any other. It starts from **What
happened**, the run context. The palette lists every action in the catalog under **Actions**; dropping one adds an
action node set to it, and its settings are a form drawn from the action's params (a template field takes `{{ }}`),
with a badge saying whether it runs on its own or asks you first, and **Wait first** for a delay. **Decide** sits
under AI: write its question and its options one a line as `name: what it means`, and each option becomes an output
port, with `unsure` last. The canvas says what the backend would refuse before you publish: an action not chosen or
a required param left empty, a wait that isn't like `30m`, a decide node without a question or with fewer than two
options, and the leaf rule (nothing connected after an action that reaches outside Lens or waits). **Try on a
sample** runs the graph on the sample of the kind you pick: nothing is kept or sent, each node shows what it passed
on, and each action says what it would do (run at once, ask you first or wait for its time).

#### The decide node

Built (`match.check_decide`, `match.run_decide`). A decide node asks the decision model (Jev, then Laya, then the
Expand Down Expand Up @@ -1245,7 +1259,6 @@ held run until you pick.
channel; web push; private chats through Matterbridge (it can't tell them from rooms); with several people let in,
"me" is whoever last wrote privately, until each chat user has a Lens account of their own; telling which links in a
message the person's template wrote, so they stay when the run read outside text.
- The canvas draws and edits event workflows (today they are saved and tried through the API).
- Variables and templates: secrets as variables (sealed, never rendered into prompts); typed variable declarations
with defaults per automation; a time zone of each person's own, rather than the triggers'; wrapping outside text
the same way in `tool_nodes.ask_model` (pipeline workflows) and extensions' prompt tools; variables in the
Expand Down
4 changes: 4 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,6 +751,10 @@ A pipeline runs a workflow as a Workflow step, pinned to the workflow's publishe
means "runs after", and the graph is put in order (ties left to right) and kept as the version's steps, so runs and
Activity work as before. A pipeline saved as a list is drawn as a chain.

An automation's workflow (scope `event`) is drawn on the same canvas, from What happened (the run) to Action and
Decide nodes, with every action in the catalog listed in the palette and its settings drawn from the action's params
([On the canvas](automations.md#on-the-canvas)).

## Content types

Every resource is video, audio, image or text (transcripts, documents, web pages, emails and calendar events are
Expand Down
2 changes: 2 additions & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` |
| Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` |
| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again`, `test_what_tools_returned_is_cut_to_fit_the_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app`; [chat-summary-mark.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-summary-mark.test.tsx) `says how many messages the summary stands for, and shows it on request` |
| Questions about Lens itself answered from its docs and this server, citing both, for people and agents over MCP | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_assistant_answers_about_lens_itself`, `test_admins_see_where_each_setting_is`, `test_agents_ask_lens_over_mcp` |
| A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` |
| Telegram and Slack built in, with voice notes and files | [test_telegram.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_telegram.py) `test_it_answers_private_chats_and_what_is_said_to_it_in_groups`, `test_voice_notes_are_heard`, `test_files_are_held_for_the_assistant_to_import`; [test_slack.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_slack.py) `test_it_answers_direct_messages_mentions_and_the_slash_command`, `test_voice_clips_are_heard_and_files_held` |
| Other chat rooms through Matterbridge | [test_bridge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_bridge.py) `test_it_answers_what_is_said_to_it_and_nothing_else`; [test_matterbridge_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_matterbridge_run.py) `test_chat_networks_are_set_in_the_app_and_become_matterbridges_config`, `test_the_bridge_talks_to_the_matterbridge_lens_runs`; [matterbridge.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/matterbridge.yml) `Lens writes a config, and Matterbridge starts with it` |
Expand Down Expand Up @@ -198,6 +199,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Joins wait for an automation's triggers together: all of them, a number of times, one that doesn't come in time, or a batch, on timers that outlast a restart | [test_match_joins.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_match_joins.py) `test_all_waits_for_every_trigger_within_the_window_on_the_same_key`, `test_count_fires_once_per_crossing_then_counts_again`, `test_absent_waits_with_a_timer_and_the_missing_one_cancels_it`, `test_batch_collects_then_runs_once`, `test_two_writers_at_once_lose_nothing`; [test_automation_conditions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_conditions.py) `test_an_absent_join_waits_on_a_timer_and_switching_off_drops_it` |
| Each automation has limits and budgets | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_the_if_limits_and_dry_runs`, `test_a_run_over_budget_is_held_for_the_persons_pick` |
| Agents make, try and run automations over MCP and in the assistant, or draft one from a sentence | [test_automation_tools.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_automation_tools.py) `test_agents_save_try_and_run_automations_over_mcp`, `test_the_assistant_saves_at_once_and_asks_to_switch_on_or_run`; [test_automations.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_automations.py) `test_an_automation_is_drafted_from_a_sentence` |
| An automation's workflow is drawn and tried on the canvas: its actions in the palette, their settings drawn from each action's params, Decide's ports and the leaf rule | [workflow-canvas.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-canvas.test.ts) `give a decide node a port per option, then unsure`, `keep the leaf rule: nothing after an action that reaches outside Lens or waits`, `say what the backend would reject in their settings`; [automation-nodes-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/automation-nodes-ui.test.tsx) `sets an action up from the catalog, its params drawn from its schema`; [schema-form.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/schema-form.test.tsx) `opens the variable picker on {{ and puts the pick in with Enter` |

## See what it costs

Expand Down
89 changes: 89 additions & 0 deletions nextjs-frontend/__tests__/automation-nodes-ui.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
import { fireEvent, render, screen } from "@testing-library/react";
import "@testing-library/jest-dom";
import { useState } from "react";

import { ActionSettings, DecideSettings } from "@/components/workflows/action-settings";
import type { ActionDef } from "@/components/workflows/workflow-model";

const actions: ActionDef[] = [
{
name: "notify",
label: "Send a message",
group: "Messages",
description: "Tells you in your own chat app",
params: {
type: "object",
properties: { to: { type: "string", title: "To", default: "me" }, text: { type: "string", title: "Message" } },
},
required: ["text"],
effect: "depends",
},
{
name: "http_request",
label: "Call a web address",
group: "Web",
params: { type: "object", properties: { url: { type: "string", title: "Address" } } },
required: ["url"],
effect: "out",
},
];

function Harness({
which,
spy,
start,
}: {
which: "action" | "decide";
spy: jest.Mock;
start: Record<string, unknown>;
}) {
const [c, setC] = useState(start);
const set = (patch: Record<string, unknown>) => {
const next = { ...c, ...patch };
for (const k of Object.keys(next)) if (next[k] === undefined) delete next[k];
spy(next);
setC(next);
};
return which === "action" ? (
<ActionSettings config={c} set={set} actions={actions} nodeId="n1" variables={[{ path: "event.summary" }]} />
) : (
<DecideSettings config={c} set={set} />
);
}

describe("an automation's nodes on the canvas", () => {
it("sets an action up from the catalog, its params drawn from its schema", () => {
const spy = jest.fn();
render(<Harness which="action" spy={spy} start={{ action: "notify", params: {} }} />);
expect(screen.getByText("Tells you in your own chat app")).toBeInTheDocument();
expect(screen.getByText("Asks you first when it reaches other people")).toBeInTheDocument();
fireEvent.change(screen.getByLabelText(/^Message/), { target: { value: "New: {{ event.summary }}" } });
expect(spy).toHaveBeenLastCalledWith({ action: "notify", params: { text: "New: {{ event.summary }}" } });
fireEvent.change(screen.getByLabelText(/^Wait first/), { target: { value: "30m" } });
expect(spy).toHaveBeenLastCalledWith(expect.objectContaining({ delay: "30m" }));
});

it("clears the params when the action changes, and says an outside action asks first", () => {
const spy = jest.fn();
render(<Harness which="action" spy={spy} start={{ action: "notify", params: { text: "hi" } }} />);
fireEvent.change(screen.getByLabelText(/^Action/), { target: { value: "http_request" } });
expect(spy).toHaveBeenLastCalledWith({ action: "http_request", params: {} });
expect(screen.getByText("Asks you first")).toBeInTheDocument();
expect(screen.getByLabelText(/^Address/)).toHaveValue("");
});

it("writes a decide node's options one a line, as name: what it means", () => {
const spy = jest.fn();
render(<Harness which="decide" spy={spy} start={{ question: "", options: { yes: "", no: "" } }} />);
fireEvent.change(screen.getByLabelText(/^Question/), { target: { value: "Does it need me today?" } });
fireEvent.change(screen.getByLabelText(/^Options/), {
target: { value: "urgent: needs an answer today\nlater: can wait\n" },
});
fireEvent.change(screen.getByLabelText(/^Sure enough at/), { target: { value: "0.8" } });
expect(spy).toHaveBeenLastCalledWith({
question: "Does it need me today?",
options: { urgent: "needs an answer today", later: "can wait" },
min_confidence: 0.8,
});
});
});
Loading
Loading