Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,8 @@ When the configured model supports function calling, chat becomes an agent.
and its other names, hiding one, or adding one to a namespace's fixed list) and changing topics (merging,
renaming, describing or deleting one; [Topics](topics.md)) don't happen directly.
They become approval cards (with the batch estimate) that the person approves, approves on a sample, or declines
(`POST /api/v1/approvals/<id>`). Viewers only get the read tools.
(`POST /api/v1/approvals/<id>`). Viewers only get the read tools. When an [automation](automations.md#the-model-and-the-agent)
uses the assistant, every change it would make is an approval of yours, notes included, with the run as its origin.
- **Check sources** (`POST /api/v1/chats/<id>/messages/<id>/check`) re-checks every cited claim against the lines it
cites, and lists sentences that cite nothing.
- **Fallback:** a model that can't call tools falls back to search-and-answer, with a notice. A model that answers
Expand Down
30 changes: 29 additions & 1 deletion docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ merged is called built.
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
| The action registry with risk from each action's effect, and one action, `log_line` (`app/domain/actions.py`) | built |
| Telling a person: the `notify` action and `channels.tell` (their own chat, a notification target, a Matterbridge room) | built |
| The model and the agent: the `ask_model`, `call_tool` and `ask_agent` actions, and prompts with outside text marked | built |
| Settings → Automations: how often followers look and how long the event log is kept | built |
| Storing and running automations, schedules and timers, joins, the other actions, approvals, variables, the timeline page, the studio | planned |

Expand Down Expand Up @@ -250,12 +251,16 @@ An effect may be worked out from the params (`notify` to "me" is `self`, to a ro
worked out, it counts as `out`. An action's own `risk` function may make it stricter, never looser.

An action acts for the automation's person (`run_as`), with their permissions as they are when it runs
(`actions.person`): an automation whose person was switched off or removed does nothing.
(`actions.person`), within the automation's namespaces (`actions.reach`): an automation whose person was switched off
or removed does nothing. What it writes is by the automation itself, `automation:<id>` (`actions.actor`).

| Action | Does | Effect | Output |
| --- | --- | --- | --- |
| `log_line` | puts a line (`text`, `level` info or warning) in the run's log | read | `{line, level}` |
| `notify` | sends `text` (and a `link`; Lens's own paths become full links) `to` `me` (your own chat app), `target:<id>` (one of the namespace's [notification targets](notifications.md)) or `gateway:<name>` (a Matterbridge room) | self for `me`, else out | `{to, text, sent, failed}`: where it went |
| `ask_model` | asks the model a `prompt` (with your own `system` instructions and `model` if you like); with `json_schema` (a JSON Schema of type object) the reply is that object | read | `{text}`, or the object the schema describes |
| `call_tool` | calls one of the assistant's tools (`tool`, with `args`) as the automation | read: a change becomes an approval | `{tool, result, summary, approvals}` |
| `ask_agent` | hands what happened (`about`, by default the event) to the agent with your `instructions`, limited to some `tools` if you like, for up to `max_steps` rounds (6, at most 12) | read: changes become approvals | `{answer, chat, approvals, steps}` |

#### Telling a person

Expand All @@ -273,6 +278,26 @@ they stay. The same goes for notification targets and Matterbridge rooms. A noti
automation's namespaces, switched on, and the automation's person must be able to edit that namespace; a Matterbridge
room needs the bridge on and answering as the automation's person (or the person to be an admin).

#### The model and the agent

`ask_model`, `call_tool` and `ask_agent` use the assistant's tools as the automation, with its person's permissions
within the automation's namespaces, their extensions and no server tools. Nobody reads along while an automation runs,
so every change asks the person: a tool that changes something makes an approval (notes included, which the assistant
in chat writes at once), and a tool that could carry data out of Lens (an extension's web or server tool) asks first
whatever the run read. The approvals are the person's (`GET /api/v1/approvals`), with the run as their `origin`
(`{automation, run, node}`); the output lists their ids, so a later node can tell you what waits.

`ask_agent` saves the exchange as one of your conversations, titled "<automation>: <what happened>", with the
activity's page as what it was asked from, so you can open it, approve what it proposed there and carry on. Model calls
count in the activity ledger under the run, and stop once a budget they count for (the namespace's, or everything's)
is used up.

Params marked `"format": "prompt"` in an action's schema (`ask_model`'s `prompt`, `ask_agent`'s `instructions`) are
prompts: they are rendered with `actions.render_prompt(text, ctx, cfg)`, which marks as data (`<untrusted-…>`) what
the run read from outside Lens before the model sees it: each activity's `lens.untrusted` fields (an email's subject
and body) and everything earlier steps passed on (a model's reply, a web page). Your own words in the prompt stay as
written. `ask_agent` passes the activity it is about the same way.

## Refine later

- More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`.
Expand All @@ -286,3 +311,6 @@ room needs the bridge on and answering as the automation's person (or the person
channel; web push; private chats through Matterbridge (it can't tell them from rooms); with several people let in,
"me" is whoever last wrote privately, until each chat user has a Lens account of their own; telling which links in a
message the person's template wrote, so they stay when the run read outside text.
- The model and the agent: per-node model picks in the studio; streaming an agent's steps into the run's log as they
happen; a namespace assistant's memory for automations (`remember` is left out today); approving an agent's
proposals from the chat app it told you in; count the automation's own budget once budgets take `automation:<id>`.
92 changes: 88 additions & 4 deletions fastapi_backend/app/domain/actions.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,28 @@

When an action node runs, it renders its params and makes an intent (automation_spec.check_intent); `perform` carries
an intent out, or with `dry` says what it would do and changes nothing. An action acts for the automation's person
(`person`), with their permissions as they are when it runs. `log_line` ships here; the other actions are registered by
a module per group, loaded with this one: actions_messages.py (notify).
(`person`), with their permissions as they are when it runs (`reach`), and what it writes is by the automation itself
(`actor`). Params marked `"format": "prompt"` are prompts: the engine renders them with `render_prompt`, which wraps
what the run read from outside Lens as data. `log_line` ships here; the other actions are registered by a module per
group, loaded with this one: actions_messages.py (notify), actions_ai.py (ask_model, call_tool, ask_agent).
"""

from __future__ import annotations

import copy
import logging
import re
from collections.abc import Callable
from dataclasses import dataclass

from . import auth
from . import auth, store, templates, untrusted
from . import automation_spec as spec_

NAME_RX = re.compile(r"^[a-z][a-z0-9_]{0,40}$")
GROUPS = ("Messages", "Lens", "Assets", "Web", "AI", "Run")
TYPES = {"string": str, "integer": int, "number": (int, float), "boolean": bool, "array": list, "object": dict}
TEMPLATE = "{{"
PROMPT = "prompt" # the JSON Schema format of a param that is a prompt (render_prompt)
log = logging.getLogger("lens.automations")


Expand Down Expand Up @@ -215,6 +219,86 @@ def person(db, intent, user=None):
return {"id": a["id"], "email": a["email"], "name": a.get("name") or a["email"], "admin": bool(a.get("admin"))}


def actor(db, intent):
"""Who an intent's changes are by: the automation itself, {id: automation:<id>, email: "automation: <name>",
name}, as import hooks act as import_hook:<id>; a draft's dry run is automation:draft."""
aid = intent.get("automation")
if aid is None:
return {"id": "automation:draft", "email": "automation: draft", "name": "Draft"}
try:
name = (db.one("SELECT name FROM $r", r=store.R("automation", int(aid))) or {}).get("name")
except Exception: # noqa: BLE001 - automations aren't stored yet, or the row is gone: named by its id
name = None
name = name or f"Automation {aid}"
return {"id": f"automation:{aid}", "email": f"automation: {name}", "name": name}


def reach(db, intent, account):
"""(readable, editable): the namespaces an intent may read and change, the person's roles now within the
automation's namespaces (all of the person's when it names none)."""
roles = auth.roles(db, account)
spaces = set(intent.get("spaces") or [])
readable = {s for s in roles if not spaces or s in spaces}
return readable, {s for s in readable if auth.allows(roles, s, "editor")}


# ---------- prompts ----------
def prompt_params(spec):
"""The names of an action's params that are prompts (format "prompt")."""
return [k for k, p in (spec.params.get("properties") or {}).items() if p.get("format") == PROMPT]


def _wrap(v, source, cfg):
if isinstance(v, str):
return untrusted.wrap(v, source, cfg)
if isinstance(v, list):
return [_wrap(x, source, cfg) for x in v]
if isinstance(v, dict):
return {k: _wrap(x, source, cfg) for k, x in v.items()}
return v


def _wrapped(a, cfg):
"""An activity with what its `lens.untrusted` paths hold marked as data."""
out = copy.deepcopy(a)
source = f"{a.get('kind') or 'activity'} from {(a.get('actor') or {}).get('name') or 'a source'}"
for path in (a.get("lens") or {}).get("untrusted") or []:
*head, last = str(path).split(".")
node = out
for k in head:
node = (
node[int(k)]
if isinstance(node, list) and k.isdigit() and int(k) < len(node)
else node.get(k)
if isinstance(node, dict)
else None
)
if node is None:
break
if isinstance(node, dict) and last in node:
node[last] = _wrap(node[last], source, cfg)
elif isinstance(node, list) and last.isdigit() and int(last) < len(node):
node[int(last)] = _wrap(node[int(last)], source, cfg)
return out


def render_prompt(text, ctx, cfg=None):
"""A prompt param rendered against the run context (templates.render_body), with what the run read from outside
Lens marked as data first (untrusted.wrap): each activity's `lens.untrusted` paths and everything earlier steps
passed on (a model's reply, a web page). The person's own words stay as written. ValueError naming the problem."""
safe = dict(ctx or {})
if isinstance(safe.get("event"), dict):
safe["event"] = _wrapped(safe["event"], cfg)
if isinstance(safe.get("events"), dict):
safe["events"] = {k: _wrapped(a, cfg) if isinstance(a, dict) else a for k, a in safe["events"].items()}
if isinstance(safe.get("steps"), dict):
safe["steps"] = _wrap(safe["steps"], "an earlier step of this automation", cfg)
try:
return templates.render_body(str(text or ""), safe, "prompt")
except templates.TemplateProblem as e:
raise ValueError(f"the prompt: {e}") from None


def describe(name, params):
"""What an action would do, in words."""
spec = get(name)
Expand Down Expand Up @@ -257,4 +341,4 @@ def _log_line(db, cfg, intent, user, dry):
)


from . import actions_messages # noqa: E402, F401 - registers notify; loaded with the catalog
from . import actions_ai, actions_messages # noqa: E402, F401 - they register their actions; loaded with the catalog
Loading
Loading