From 6a07a986df7bdd96d62bb166a9341a12289641ba Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 07:18:59 +0000 Subject: [PATCH] feat(timeline): the agent answers when tagged in a comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A comment on the timeline or on an asset that tags @lens (or the assistant's own name) gets the agent's answer as a comment by Lens on the same thread. It answers as the person who tagged it, with their tools, in a conversation of theirs titled after the item (one per thread); every change it proposes waits for their approval and the reply links where to decide. Its model calls run in the ledger against that conversation and the item's namespace, and without a model or within a used-up budget the reply says so. Settings › AI assistant › Answer when tagged in a comment (ai.answer_comments, on) turns it off. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Ln25ykE6uBDqx21WBU94SX --- README.md | 3 +- changelog.d/comment-agent.md | 5 + docs/assistant.md | 6 + docs/requirements.md | 1 + docs/timeline.md | 21 +- .../app/api/v1/routes/activities.py | 8 +- fastapi_backend/app/api/v1/routes/comments.py | 21 +- fastapi_backend/app/domain/comment_agent.py | 229 ++++++++++++++++++ fastapi_backend/app/domain/comments.py | 18 +- fastapi_backend/app/domain/events_content.py | 7 +- fastapi_backend/app/domain/store.py | 1 + fastapi_backend/app/schemas/comments.py | 3 +- .../tests/api/test_comment_agent.py | 110 +++++++++ .../__tests__/settings-web-search.test.tsx | 1 + .../__tests__/timeline-social-model.test.ts | 11 +- nextjs-frontend/app/openapi-client/sdk.gen.ts | 5 +- .../app/openapi-client/types.gen.ts | 8 +- .../components/settings/bodies.tsx | 1 + nextjs-frontend/components/settings/model.ts | 7 + .../components/timeline/social-model.ts | 16 ++ .../components/timeline/social.tsx | 27 ++- nextjs-frontend/openapi.json | 12 +- 22 files changed, 485 insertions(+), 36 deletions(-) create mode 100644 changelog.d/comment-agent.md create mode 100644 fastapi_backend/app/domain/comment_agent.py create mode 100644 fastapi_backend/tests/api/test_comment_agent.py diff --git a/README.md b/README.md index 7074f3a25..01d12c5bd 100644 --- a/README.md +++ b/README.md @@ -184,7 +184,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p warnings) is kept as a stream of activities in W3C Activity Streams form, which agents read through the API, MCP and the assistant, and people review on the Timeline page (and Latest on Home), marking what they've seen, reacting with an emoji and commenting in threads everyone who sees the item shares (agents too, over MCP), and asking the - assistant about any item from the item. Automations act + assistant about any item from the item; a comment that tags @lens gets the agent's answer in the thread, with + every change it proposes waiting for a yes. Automations act on that stream, on a schedule, at a time or when their own webhook is called, alone or with their triggers joined (all of them, a number of times, one that doesn't come in time, or a batch): they test conditions (text, patterns, numbers, lists, times, days, whether a value changed and whether something like it happened before in their diff --git a/changelog.d/comment-agent.md b/changelog.d/comment-agent.md new file mode 100644 index 000000000..7d8ef36a9 --- /dev/null +++ b/changelog.d/comment-agent.md @@ -0,0 +1,5 @@ +**Tag @lens in a comment and the agent answers in the thread.** On the timeline or on an asset's page, a comment +with `@lens` (or the assistant's own name) in it gets the agent's answer as a comment by Lens on the same thread, +as the person who tagged it, with their tools, in a conversation of theirs titled after the item; every change it +proposes waits for their approval, and the reply links where to decide. Off with Settings → AI assistant → Answer +when tagged in a comment ([Asking the agent in a comment](docs/timeline.md#asking-the-agent-in-a-comment)). diff --git a/docs/assistant.md b/docs/assistant.md index 29a3b472a..c355a4512 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -766,6 +766,12 @@ A chat room can be given to a namespace's assistant too ([chat rooms](chat-rooms Refine later: a decision model choosing what's worth remembering, similar memories merged. +## Asking the assistant in a comment + +Tag **@lens** (or the assistant's own name) in a comment on the timeline or on an asset's page and it answers in the +thread, as you, with your tools, in a conversation of yours it opens for the thread; changes it proposes wait for +your yes as they do in chat ([Asking the agent in a comment](timeline.md#asking-the-agent-in-a-comment)). + ## Extending the assistant: tools, skills, hooks and plugins Anyone who can edit can add to what the assistant does. Each addition is an *extension*: versioned, kept private or diff --git a/docs/requirements.md b/docs/requirements.md index 4255bfbf7..0e52eacf3 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -236,6 +236,7 @@ request, to another check that proves the same thing; loosening a requirement or | Ask the assistant about a timeline item from the item (its button, or `a` on its line), in the chat beside the page, with the item's id, line and fields as what the question is about | [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `asking about an item hands the page chat its id, line and fields, by button or the a key`; [page-chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/page-chat.test.tsx) `opens with what a page's own Ask about this hands it, and sends it as the highlighted part` | | The Timeline page and Latest on Home, marking what you've seen | [test_timeline_marks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_timeline_marks.py) `test_marks_are_set_cleared_and_filter_the_timeline`; [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `Home's Latest card lists what you haven't reviewed` | | Reactions and comments on timeline activities, shared by everyone who sees them, through the API, MCP and the page | [test_timeline_social.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_timeline_social.py) `test_reactions_are_shared_counted_and_taken_back`, `test_comments_on_an_activity_are_a_thread_everyone_who_sees_it_joins`, `test_comments_on_a_logged_activity_follow_its_audience`, `test_agents_react_and_comment_too`; [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `reactions and comments: a line sums its items', a 👍 goes to the server, and Enter sends a comment` | +| Tagging @lens in a comment on the timeline or an asset gets the agent's answer in the thread, as the person who tagged it, with every change waiting for their approval, off with `ai.answer_comments` | [test_comment_agent.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_comment_agent.py) `test_the_agent_answers_a_tagged_comment_in_the_thread`, `test_without_a_model_or_within_a_used_up_budget_the_reply_says_so`; [timeline-social-model.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-social-model.test.ts) `the agent's reply links its addresses` | | Automations act on the stream, on a schedule, at a time or when their own webhook is called | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_each_new_activity_runs_each_automation_it_matches_once`, `test_a_schedule_runs_its_automation_when_its_timer_fires`, `test_a_call_to_a_webhook_runs_its_automation`; [test_timer_triggers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_timer_triggers.py) `test_an_at_trigger_fires_once_then_reads_done` | | They test conditions, then tell you, ask the model or the agent, use a tool, call a web address with sealed credentials, run Lens work, or change assets and notes | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_the_if_limits_and_dry_runs`; [test_actions_messages.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_messages.py) `test_notify_tells_me_in_my_chat_app`; [test_actions_ai.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_ai.py) `test_ask_model_answers_in_text_or_as_the_fields_a_schema_names`, `test_ask_agent_works_on_what_happened_and_saves_the_conversation`, `test_call_tool_reads_as_the_person_and_changes_only_by_asking_them`; [test_actions_web.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_web.py) `test_credentials_in_headers_are_sealed_to_the_automation_and_the_host_and_never_shown`; [test_actions_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_run.py) `test_run_workflow_queues_it_on_the_asset_as_the_automation`, `test_run_pipeline_processes_the_asset_again`; [test_actions_lens.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_lens.py) `test_set_field_is_kept_in_the_assets_history_by_the_automation`, `test_write_note_writes_a_page_adds_to_a_things_page_or_notes_an_asset` | | What reaches outside Lens, or changes more than what set it off, asks first: on the Approvals page or with a yes in chat; agents decline but never approve | [test_automation_spec.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_spec.py) `test_risk_follows_the_effect`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_approve_carries_the_action_out_and_the_run_records_it`, `test_agents_over_mcp_list_and_decline_but_never_approve`; [test_approvals_chat.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals_chat.py) `test_a_yes_in_my_chat_decides_an_action_inside_lens`; [approvals-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/approvals-page.test.tsx) `groups what waits, puts the first in focus, and Enter approves it` | diff --git a/docs/timeline.md b/docs/timeline.md index 46b34d214..5f256fd52 100644 --- a/docs/timeline.md +++ b/docs/timeline.md @@ -15,7 +15,7 @@ Status: in progress. The studio's timeline is built in steps; only what is merge | Reactions and comments on any item, shared by everyone who sees it; `react_to_activity`, `activity_comments` and `comment_on_activity` for agents | built | | Ask about an item: talk to the assistant about it from the item, in the chat beside the page | built | | What each item caused (automation runs, approvals), and Approve and Decline on waiting approvals | planned | -| Tagging the agent in a comment, so it answers in the thread | planned | +| Tagging the agent in a comment, so it answers in the thread | built | | Sources, sensors, people and agents as actors in the graph; a For you order learned from what each person reacts to, comments on and hides | planned | ## What shows @@ -95,6 +95,20 @@ them, and each one is an activity of its own, so an automation (or an agent) can are the thread, as [comments on a resource](api.md#comments) are. `GET /api/v1/activities` carries everyone's reactions on the page in `reactions` and each item's comment count in `comments`, by id. +## Asking the agent in a comment + +A comment that tags **@lens** (or the assistant's own name from Settings → Chat rooms, say `@jarvis`) gets an answer +from the agent in the same thread, on the timeline and on an asset's page alike: "@lens what is this one about?", +"@lens write a note of the action items". The agent answers as the person who tagged it, with their namespaces and +their tools, in a conversation of theirs titled after the item (one per thread, so a later tag on the thread carries +on in it); they find it on the Chat page and can go on there. It reads the item the thread is on and the comments on +it so far, as data, and answers with its sources. It only reads: every change it proposes (a note, a move, a message +out) waits for that person's yes, and the reply says so with a link to decide. The reply is a comment by **Lens** +(marked with a sparkle), which the person who asked can delete and nobody can edit. Its model calls run in the +ledger against their conversation and the item's namespace; when no model is set up or the namespace's budget is +used up, the reply says so instead. **Settings → AI assistant → Answer when tagged in a comment** +(`ai.answer_comments`, on) turns it off; without a tag nothing is asked of the model. + ## Review marks Each person marks activities for themselves alone; nobody else sees or sets them, admins included. @@ -129,7 +143,8 @@ Marks change nothing else: an activity you hid still triggers automations and st - MCP, on the feed: `react_to_activity` (`id`, `emoji`, `on`) reacts for the person it acts for, `activity_comments` (`id`) reads an item's threads, and `comment_on_activity` (`id`, `text`, `parent`) writes a comment or a reply, with a token that has the write scope. The `timeline` page carries `reactions` and `comments` too. Comments are - what people wrote: agents read them as data, never as instructions. + what people wrote: agents read them as data, never as instructions. A comment an agent writes with `@lens` in it + gets the assistant's answer like anyone's, as the person the token acts for. ## How marks, reactions and comments are kept @@ -156,3 +171,5 @@ counts are two grouped queries. Nothing of this costs a model call. the person is removed; a tidy could drop them. Reactions and comments on such activities likewise. - Who gave each reaction, on hover; editing and resolving comments from the timeline (the API has both). - A `comment.added` on an activity links to the timeline, not yet to the item itself. +- The agent's reply in a thread is plain text with links; the answer's excerpts could open at their moment, as in + chat. Tagging a namespace's own assistant by its name, with its memory. diff --git a/fastapi_backend/app/api/v1/routes/activities.py b/fastapi_backend/app/api/v1/routes/activities.py index 775b8a532..1b42226c8 100644 --- a/fastapi_backend/app/api/v1/routes/activities.py +++ b/fastapi_backend/app/api/v1/routes/activities.py @@ -12,7 +12,7 @@ from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer, domain_errors from app.api.v1.routes.comments import _comments_out -from app.domain import auth, comments, event_marks, event_reactions, events, events_content, store +from app.domain import auth, comment_agent, comments, event_marks, event_reactions, events, events_content, store from app.schemas.activities import ( ActivityKind, ActivityMark, @@ -268,6 +268,7 @@ def comment( audience=lens.get("audience"), owner=lens.get("owner"), ) + comment_agent.wake(db, cfg, cid, text) return _comments_out(db, [comments.get(db, cid)], _rank(acl, user, a), user)[0] @@ -275,7 +276,8 @@ def comment( def comment_on_activity(activity_id: str, body: ActivityCommentCreate, user: Writer, acl: Acl, db: Db, cfg: Cfg) -> Comment: """Comment on an activity, or with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone who sees the activity can, up to 1,000 each; everyone who sees it reads them, and each one is a `comment.added` - activity. On an asset's arrival the comment goes on the asset itself.""" + activity. On an asset's arrival the comment goes on the asset itself. A comment that tags @lens gets the agent's + answer on the thread.""" return comment(db, cfg, viewer(db, user, acl), user, acl, activity_id, body.text, body.parent) @@ -288,7 +290,7 @@ def update_activity_comment(activity_id: str, cid: int, body: CommentUpdate, use rank = _rank(acl, user, a) if body.text is None and body.resolved is None: raise HTTPException(400, "change the text, or resolve or reopen the thread") - if body.text is not None and c["account"] != user.id: + if body.text is not None and (c["account"] != user.id or c.get("author") == "assistant"): raise HTTPException(403, "only its writer can change a comment") if body.resolved is not None and c["account"] != user.id and rank < auth.ROLES["editor"]: raise HTTPException(403, "only its writer or an editor of the namespace can resolve a thread") diff --git a/fastapi_backend/app/api/v1/routes/comments.py b/fastapi_backend/app/api/v1/routes/comments.py index 558a07799..16047302d 100644 --- a/fastapi_backend/app/api/v1/routes/comments.py +++ b/fastapi_backend/app/api/v1/routes/comments.py @@ -10,8 +10,8 @@ from fastapi import APIRouter, HTTPException -from app.api.deps import Access, Acl, CurrentUser, Db, Principal, Writer, domain_errors -from app.domain import auth, comments, highlights, store +from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer, domain_errors +from app.domain import auth, comment_agent, comments, highlights, store from app.domain.store import DB from app.schemas.comments import Comment, CommentCreate, CommentUpdate, Highlight, HighlightCreate, HighlightUpdate from app.schemas.common import Ok @@ -56,7 +56,8 @@ def _comments_out(db: DB, rows: list[dict[str, Any]], rank: int, user: Principal out = [] for c in rows: by, res = people.get(c["account"], {}), people.get(int(c["resolved_by"]), {}) if c.get("resolved_by") is not None else {} - mine = c["account"] == user.id + agent = c.get("author") == "assistant" # the agent's answer, for the person it names: theirs to delete, not to edit + mine = c["account"] == user.id and not agent out.append( Comment( id=c["id"], @@ -71,14 +72,15 @@ def _comments_out(db: DB, rows: list[dict[str, Any]], rank: int, user: Principal resolved_by=res.get("email"), resolved_by_name=res.get("name") or None, resolved_at=c.get("resolved_at"), - created_by=by.get("email"), - created_by_name=by.get("name") or None, + created_by=None if agent else by.get("email"), + created_by_name="Lens" if agent else by.get("name") or None, + by_agent=agent, created_at=c.get("created_at"), updated_at=c.get("updated_at"), edited_at=c.get("edited_at"), mine=mine, can_resolve=c.get("parent") is None and (mine or editor), - can_delete=mine or owner, + can_delete=mine or owner or (agent and c["account"] == user.id), ) ) return out @@ -93,13 +95,14 @@ def list_comments(rid: int, user: CurrentUser, acl: Acl, db: Db) -> list[Comment @router.post("/comments") -def create_comment(rid: int, body: CommentCreate, user: Writer, acl: Acl, db: Db) -> Comment: +def create_comment(rid: int, body: CommentCreate, user: Writer, acl: Acl, db: Db, cfg: Cfg) -> Comment: """Comment on a moment or passage (`t0`–`t1`, with the `quote` picked in the text) or on the whole resource, or with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone who can read the resource - can, up to 1,000 each on a resource.""" + can, up to 1,000 each on a resource. A comment that tags @lens gets the agent's answer on the thread.""" rec = _recording(acl, db, rid) with domain_errors(): cid = comments.create(db, rid, rec["space"], user.id, body.text, body.t0, body.t1, body.quote, body.parent, rec.get("duration_ms")) + comment_agent.wake(db, cfg, cid, body.text) return _comments_out(db, [comments.get(db, cid)], _rank(acl, rec), user)[0] @@ -112,7 +115,7 @@ def update_comment(rid: int, cid: int, body: CommentUpdate, user: Writer, acl: A rank = _rank(acl, rec) if body.text is None and body.resolved is None: raise HTTPException(400, "change the text, or resolve or reopen the thread") - if body.text is not None and c["account"] != user.id: + if body.text is not None and (c["account"] != user.id or c.get("author") == "assistant"): raise HTTPException(403, "only its writer can change a comment") if body.resolved is not None and c["account"] != user.id and rank < auth.ROLES["editor"]: raise HTTPException(403, "only its writer or an editor of the resource can resolve a thread") diff --git a/fastapi_backend/app/domain/comment_agent.py b/fastapi_backend/app/domain/comment_agent.py new file mode 100644 index 000000000..42197d3b1 --- /dev/null +++ b/fastapi_backend/app/domain/comment_agent.py @@ -0,0 +1,229 @@ +"""The agent in a comment thread (docs/timeline.md, Asking the agent in a comment): a comment on the timeline, or on +an asset, that tags @lens (or the assistant's own name) gets an answer from the agent in the same thread. + +The agent answers as the person who tagged it, with their permissions, through the assistant's tool loop +(chat.tool_answer) in a conversation of theirs titled after the item, so they can open it and carry on there. It reads +the item the thread is on (the activity, marked as data where it holds text from outside Lens) and the thread so far. +It only reads: every change it proposes becomes an approval the person decides, and the reply says so with a link. +The reply is a comment by the assistant (`author` assistant) on the same thread, with its sources, so everyone who +sees the item reads it. Its model calls run in the ledger against the conversation and the item's namespace, and a +used-up budget stops it before it starts. `ai.answer_comments` (on) turns it off. +""" + +from __future__ import annotations + +import logging +import re +import threading + +from . import actions_ai, activity, auth, bridge, budgets, chat, comments, events, llm, store + +R = store.R +log = logging.getLogger(__name__) +NAME = "lens" +THREAD_MAX = 20 # earlier comments of the thread the agent reads +LINE_MAX = 600 # characters of each +MAX_STEPS = 6 +NOTE = ( + "\n\nYou were tagged in a comment on {where} in Lens by {person}; everyone who sees the item reads the thread. " + "Answer their comment briefly and plainly, in the thread's own words, citing what you found. Every change you " + "propose waits for their approval; say so when you propose one." +) + + +def names(cfg): + """The names a comment tags the agent by: lens, and the assistant's own name (Settings › Chat rooms).""" + own = ((cfg.get("bridge") or {}).get("name") or "").strip().lower() + return [NAME] + ([own] if own and own != NAME else []) + + +def _rx(cfg): + alts = "|".join(re.escape(n) for n in names(cfg)) + return re.compile(rf"(?`), so the same thread shows on the asset's page and on the timeline; otherwise `recording` is null. `audience` and `owner` are the activity's, so the comment's own activity (comment.added) is seen -by the same people. +by the same people. A comment that tags @lens gets the agent's answer on the thread, a comment with `author` +assistant (comment_agent.py); the thread's first comment keeps that conversation's id in `chat`. """ from __future__ import annotations @@ -24,8 +25,8 @@ TEXT_MAX = 5000 ABOUT_MAX = 300 FIELDS = ( - "record::id(id) AS id, recording, activity, about, space, account, parent, text, t0, t1, quote, resolved ?? false AS resolved, " - "resolved_by, resolved_at, created_at, updated_at, edited_at" + "record::id(id) AS id, recording, activity, about, space, account, author, chat, parent, text, t0, t1, quote, " + "resolved ?? false AS resolved, resolved_by, resolved_at, created_at, updated_at, edited_at" ) @@ -131,11 +132,12 @@ def create( about=None, audience=None, owner=None, + author=None, ): """A new comment, or with `parent` a reply on that comment's thread; its id. On a resource (`rid`), or on an activity of the timeline (`activity`, with `about`, its summary; `rid` then is the asset it is the arrival of, or - None). ValueError for no text, a moment that doesn't fit the resource, a parent that isn't a comment on it, or too - many comments.""" + None). `author` assistant: the agent wrote it, answering `account` (comment_agent.py). ValueError for no text, a + moment that doesn't fit the resource, a parent that isn't a comment on it, or too many comments.""" text = _text(text) rid = int(rid) if rid is not None else None if rid is None and not activity: @@ -162,6 +164,7 @@ def create( "about": " ".join(str(about or "").split())[:ABOUT_MAX] or None, "space": space, "account": account, + "author": "assistant" if author == "assistant" else None, "parent": parent, "text": text, "t0": t0, @@ -177,6 +180,11 @@ def create( return cid +def set_chat(db, cid, chat_id): + """Keep the conversation the agent answers this thread in, on its first comment.""" + db.q("UPDATE $r SET chat = $c", r=R("comment", int(cid)), c=int(chat_id)) + + def update(db, cid, text): """Change its text (kept as `edited_at`). ValueError for no text.""" t = store.now() diff --git a/fastapi_backend/app/domain/events_content.py b/fastapi_backend/app/domain/events_content.py index c852e91d9..9fdaba553 100644 --- a/fastapi_backend/app/domain/events_content.py +++ b/fastapi_backend/app/domain/events_content.py @@ -1075,7 +1075,8 @@ def _and_more(names, most=1): # ---------- comments ---------- COMMENT_FIELDS = ( - "id AS rid, record::id(id) AS id, recording, activity, about, audience, owner, space, account, parent, text, t0, t1, quote, created_at" + "id AS rid, record::id(id) AS id, recording, activity, about, audience, owner, space, account, author, parent, text, t0, t1, " + "quote, created_at" ) CONTENT_MAX = 1000 # characters of a comment an activity carries @@ -1087,7 +1088,7 @@ def _comment_activities(db, rows): return [] rids = list({r["recording"] for r in rows if _num(r.get("recording")) is not None}) assets = {a["id"]: a for a in db.rows(f"SELECT {ASSET_FIELDS} FROM recording WHERE id IN $r", r=[R("recording", i) for i in rids])} - who = actors(db, [(r.get("account"), r.get("space")) for r in rows]) + who = actors(db, [(_note_label(r, "account"), r.get("space")) for r in rows]) types = _content_types(db) out = [] for r in rows: @@ -1115,7 +1116,7 @@ def _comment_activities(db, rows): "t1": _num(r.get("t1")), } ) - actor = who[(r.get("account"), r.get("space"))] + actor = who[(_note_label(r, "account"), r.get("space"))] out.append( events.build( f"comment-{r['id']}", diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 4c909f7f6..b6e83841f 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -315,6 +315,7 @@ def labels(): "untrusted_asks": True, # after reading it, tools that send data out or change the server ask first (ai_tools.py) "browser": True, # people may connect their browser's Lens side panel for the assistant to use (browser.py) "browser_steps": 25, # at most this many tool steps in an answer that uses the browser + "answer_comments": True, # a comment that tags @lens gets the agent's answer on its thread (comment_agent.py) "injection_check": "patterns", # flag text that reads like instructions to an AI: patterns, classifier, model, off "disabled_tools": [], "max_steps": 6, diff --git a/fastapi_backend/app/schemas/comments.py b/fastapi_backend/app/schemas/comments.py index 79e761be1..acf47b6aa 100644 --- a/fastapi_backend/app/schemas/comments.py +++ b/fastapi_backend/app/schemas/comments.py @@ -28,7 +28,8 @@ class Comment(ResponseModel): resolved_by_name: str | None = None resolved_at: str | None = None created_by: str | None = Field(None, description="its writer's email") - created_by_name: str | None = Field(None, description="its writer's name, when they gave one") + created_by_name: str | None = Field(None, description="its writer's name, when they gave one; Lens for the agent's answer") + by_agent: bool = Field(False, description="the agent wrote it, answering a comment that tagged it") created_at: str | None = None updated_at: str | None = None edited_at: str | None = Field(None, description="when its text last changed") diff --git a/fastapi_backend/tests/api/test_comment_agent.py b/fastapi_backend/tests/api/test_comment_agent.py new file mode 100644 index 000000000..8caf8b19f --- /dev/null +++ b/fastapi_backend/tests/api/test_comment_agent.py @@ -0,0 +1,110 @@ +"""The agent in a comment thread (docs/timeline.md, Asking the agent in a comment): a comment that tags @lens gets the +agent's answer on the thread, as the person who asked, with their permissions; what it proposes waits for their +approval, the exchange is a conversation of theirs, and the answer is a comment by Lens everyone on the thread reads.""" + +from __future__ import annotations + +from app.domain import activity, chat, comment_agent, comments, store +from tests import fake_llm +from tests.api._assist import start_llm +from tests.api.test_setup_assistant import call +from tests.helpers import login, make_user, seed + +URL = "/api/v1/activities" + + +def _env(client, db, cfg, folder, monkeypatch): + ids = seed(db, cfg, folder) + me = make_user(db, "me@x.io", "my password 1", roles={"pods": "editor"}) + other = make_user(db, "other@x.io", "other password 1", roles={"pods": "viewer"}) + answered = [] + # the agent answers on a thread of its own; here at once, so the test reads the reply + monkeypatch.setattr(comment_agent, "later", lambda db_, cfg_, cid: answered.append(comment_agent.answer(db_, cfg_, cid)) or True) + return { + "ids": [f"create-recording-{i}" for i in ids], + "rids": ids, + "me": me, + "other": other, + "h": login(client, "me@x.io", "my password 1"), + "ho": login(client, "other@x.io", "other password 1"), + "answered": answered, + } + + +def test_the_agent_answers_a_tagged_comment_in_the_thread(client, db, cfg, folder, monkeypatch): + srv = start_llm(cfg) + try: + llm = fake_llm.Handler + env = _env(client, db, cfg, folder, monkeypatch) + pods1 = env["ids"][0] + h = env["h"] + llm.tool_script = [ + {"content": "", "tool_calls": [call(1, "search_transcripts", {"query": "trip"})]}, + {"content": "", "tool_calls": [call(2, "write_note", {"namespace": "pods", "title": "Trip", "body": "Friday"})]}, + {"content": "It's about the trip; I proposed a note."}, + ] + # a plain comment wakes nothing; one that tags @lens gets an answer on its thread, by Lens + client.post(f"{URL}/{pods1}/comments", json={"text": "Worth a listen"}, headers=h) + assert env["answered"] == [] + c = client.post(f"{URL}/{pods1}/comments", json={"text": "@Lens what is this one about?"}, headers=h).json() + assert len(env["answered"]) == 1 + got = client.get(f"{URL}/{pods1}/comments", headers=env["ho"]).json() + reply = got[-1] + assert reply["parent"] == c["id"] and reply["by_agent"] is True and reply["created_by_name"] == "Lens" + assert reply["created_by"] is None and reply["mine"] is False and reply["can_delete"] is False + assert reply["text"].startswith("It's about the trip; I proposed a note.") + assert "Waiting for approval in Lens: Write the note “Trip” in pods" in reply["text"] + # the person it answered may delete it; nobody edits it + mine = client.get(f"{URL}/{pods1}/comments", headers=h).json()[-1] + assert mine["can_delete"] is True and mine["mine"] is False + assert client.patch(f"{URL}/{pods1}/comments/{reply['id']}", json={"text": "x"}, headers=h).status_code == 403 + # the exchange is a conversation of the person's, titled after the item, that the thread keeps + root = comments.get(db, c["id"]) + conv = chat.get(db, root["chat"], env["me"]) + assert conv["title"].startswith("Timeline: ") and conv["scope"] == {"namespaces": ["pods"]} + q, a = chat.history(db, root["chat"]) + assert q["content"] == "what is this one about?" and q["meta"] == {"via": "comment", "options": {"comment": c["id"]}} + assert q["context"]["url"] == f"/resources/{env['rids'][0]}" + assert [s["tool"] for s in a["steps"]] == ["search_transcripts", "write_note"] + assert activity.totals(db, f"chat_message:{a['id']}")["calls"] == 3 # its model calls are in the ledger + appr = db.one("SELECT tool, account, chat FROM approval") + assert appr == {"tool": "write_note", "account": env["me"], "chat": root["chat"]} + # the model was told where it was tagged, and read the thread so far and the item as data + first = next(b for b in llm.seen if b.get("tools")) + system, asked = first["messages"][0]["content"], first["messages"][-1]["content"] + assert "tagged in a comment on the timeline in Lens by me;" in system + assert asked.startswith("what is this one about?") and "The comments on it so far:\n- me: Worth a listen" in asked + assert "What happened, as the automation saw it" in asked + # the answer is a comment.added by Lens, and tagging again carries the same conversation on + added = client.get(URL, params={"kinds": "comment.added"}, headers=h).json()["items"] + assert added[0]["actor"]["id"] == "lens:assistant" and added[0]["object"]["inReplyTo"] == f"lens:comment/{c['id']}" + llm.tool_script = [{"content": "Friday, it says."}] + client.post(f"{URL}/{pods1}/comments", json={"text": "@lens when?", "parent": c["id"]}, headers=h) + assert len(chat.history(db, root["chat"])) == 4 + assert [x["text"] for x in client.get(f"{URL}/{pods1}/comments", headers=h).json()][-2:] == ["@lens when?", "Friday, it says."] + assert "- Lens: It's about the trip" in llm.seen[-1]["messages"][-1]["content"] + # off in Settings: nothing answers; the agent never answers itself + cfg["ai"]["answer_comments"] = False + monkeypatch.undo() + assert comment_agent.wake(db, cfg, c["id"], "@lens again") is False + cfg["ai"]["answer_comments"] = True + assert comment_agent.answer(db, cfg, reply["id"]) is None + finally: + fake_llm.Handler.tool_script = [] + srv.shutdown() + + +def test_without_a_model_or_within_a_used_up_budget_the_reply_says_so(client, db, cfg, folder, monkeypatch): + env = _env(client, db, cfg, folder, monkeypatch) + rid = env["rids"][0] + h = env["h"] + # on the asset's page too, with the asset as the conversation's page + cfg["llm"]["model"] = None + c = client.post(f"/api/v1/recordings/{rid}/comments", json={"text": "hey @lens, summarise?"}, headers=h).json() + got = client.get(f"/api/v1/recordings/{rid}/comments", headers=h).json() + assert got[-1]["by_agent"] and got[-1]["parent"] == c["id"] and "no model is set up" in got[-1]["text"] + assert comment_agent.mentioned(cfg, "ask @lens.") and not comment_agent.mentioned(cfg, "lens@x.io or @lensing") + cfg["bridge"]["name"] = "Jarvis" + assert comment_agent.mentioned(cfg, "@jarvis?") and comment_agent.question(cfg, "@Jarvis when is it? ") == "when is it?" + assert comment_agent.question(cfg, "@lens") == "What do you make of this?" + assert store.R("comment", c["id"]) # the thread stays the person's diff --git a/nextjs-frontend/__tests__/settings-web-search.test.tsx b/nextjs-frontend/__tests__/settings-web-search.test.tsx index ceb4d3a36..5c458b8e7 100644 --- a/nextjs-frontend/__tests__/settings-web-search.test.tsx +++ b/nextjs-frontend/__tests__/settings-web-search.test.tsx @@ -25,6 +25,7 @@ const web = (values: Record) => tools: true, browser: true, refine_notes: true, + answer_comments: true, organise_notes: true, compact: true, auto_title: true, diff --git a/nextjs-frontend/__tests__/timeline-social-model.test.ts b/nextjs-frontend/__tests__/timeline-social-model.test.ts index 605231ede..daa67fb97 100644 --- a/nextjs-frontend/__tests__/timeline-social-model.test.ts +++ b/nextjs-frontend/__tests__/timeline-social-model.test.ts @@ -1,4 +1,4 @@ -import { applyReaction, commentsLabel, sumReactions } from "@/components/timeline/social-model"; +import { applyReaction, commentsLabel, linked, sumReactions } from "@/components/timeline/social-model"; test("a reaction moves the count by one and keeps the palette's order", () => { const list = [{ emoji: "❤️", count: 2, mine: false }]; @@ -30,3 +30,12 @@ test("a line sums its items' reactions, yours when you gave it on any", () => { expect(commentsLabel(1)).toBe("1 comment"); expect(commentsLabel(4)).toBe("4 comments"); }); + +test("the agent's reply links its addresses", () => { + expect(linked("Done.\nDecide in Lens: Write the note\nhttps://lens.local/approvals?id=3 (soon)")).toEqual([ + { text: "Done.\nDecide in Lens: Write the note\n" }, + { text: "https://lens.local/approvals?id=3", url: "https://lens.local/approvals?id=3" }, + { text: " (soon)" }, + ]); + expect(linked("no links here")).toEqual([{ text: "no links here" }]); +}); diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 08c55726a..b2ea0725e 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -4404,7 +4404,8 @@ export class Activities { * * Comment on an activity, or with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone * who sees the activity can, up to 1,000 each; everyone who sees it reads them, and each one is a `comment.added` - * activity. On an asset's arrival the comment goes on the asset itself. + * activity. On an asset's arrival the comment goes on the asset itself. A comment that tags @lens gets the agent's + * answer on the thread. */ public static commentOnActivity( options: Options, @@ -6887,7 +6888,7 @@ export class Comments { * * Comment on a moment or passage (`t0`–`t1`, with the `quote` picked in the text) or on the whole resource, or * with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone who can read the resource - * can, up to 1,000 each on a resource. + * can, up to 1,000 each on a resource. A comment that tags @lens gets the agent's answer on the thread. */ public static createComment( options: Options, diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 948e31fbf..0d56e248b 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -4801,9 +4801,15 @@ export type Comment = { /** * Created By Name * - * its writer's name, when they gave one + * its writer's name, when they gave one; Lens for the agent's answer */ created_by_name?: string | null; + /** + * By Agent + * + * the agent wrote it, answering a comment that tagged it + */ + by_agent?: boolean; /** * Created At */ diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index f265c39da..5f875282a 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -2982,6 +2982,7 @@ function AiBody({ ctx }: { ctx: BodyCtx }) { + "')\]]+/g; + +/** A comment's text split into plain runs and the web addresses in it, so the agent's "Decide in Lens: …" lines + * and sources open with a click. */ +export function linked(text: string): { text: string; url?: string }[] { + const out: { text: string; url?: string }[] = []; + let at = 0; + for (const m of text.matchAll(URL_RX)) { + if (m.index! > at) out.push({ text: text.slice(at, m.index) }); + out.push({ text: m[0], url: m[0] }); + at = m.index! + m[0].length; + } + if (at < text.length) out.push({ text: text.slice(at) }); + return out; +} diff --git a/nextjs-frontend/components/timeline/social.tsx b/nextjs-frontend/components/timeline/social.tsx index f9a2e5e76..cf46aa67c 100644 --- a/nextjs-frontend/components/timeline/social.tsx +++ b/nextjs-frontend/components/timeline/social.tsx @@ -1,13 +1,13 @@ "use client"; -import { MessageSquare, Reply, SmilePlus, Trash2 } from "lucide-react"; +import { MessageSquare, Reply, SmilePlus, Sparkles, Trash2 } from "lucide-react"; import { useState, type KeyboardEvent } from "react"; import type { Comment, Reaction } from "@/app/openapi-client/types.gen"; import { threads, type Thread } from "@/components/recording/comments-model"; import { writer } from "@/components/recording/notes-model"; import type { Activity } from "@/components/timeline/model"; -import { EMOJIS, EMOJI_LABEL, commentsLabel } from "@/components/timeline/social-model"; +import { EMOJIS, EMOJI_LABEL, commentsLabel, linked } from "@/components/timeline/social-model"; import { useActivityCommentActions, useActivityComments } from "@/components/timeline/use-social"; import { Button, IconButton } from "@/components/ui/button"; import { Textarea } from "@/components/ui/field"; @@ -189,7 +189,10 @@ function CommentLine({ return (
  • - {writer(c)} + + {c.by_agent && } + {writer(c)} + {relative(c.created_at)} {c.edited_at && · edited} @@ -209,7 +212,19 @@ function CommentLine({ )} -

    {c.text}

    +

    + {c.by_agent + ? linked(c.text).map((part, i) => + part.url ? ( + + {part.text} + + ) : ( + {part.text} + ), + ) + : c.text} +

  • ); } @@ -294,7 +309,9 @@ export function CommentThread({ a, bump }: { a: Activity; bump: (id: string, by: send(text)} diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 11a48684f..685370a73 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -5684,7 +5684,7 @@ "activities" ], "summary": "Comment On Activity", - "description": "Comment on an activity, or with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone\nwho sees the activity can, up to 1,000 each; everyone who sees it reads them, and each one is a `comment.added`\nactivity. On an asset's arrival the comment goes on the asset itself.", + "description": "Comment on an activity, or with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone\nwho sees the activity can, up to 1,000 each; everyone who sees it reads them, and each one is a `comment.added`\nactivity. On an asset's arrival the comment goes on the asset itself. A comment that tags @lens gets the agent's\nanswer on the thread.", "operationId": "comment_on_activity", "parameters": [ { @@ -13811,7 +13811,7 @@ "comments" ], "summary": "Create Comment", - "description": "Comment on a moment or passage (`t0`\u2013`t1`, with the `quote` picked in the text) or on the whole resource, or\nwith `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone who can read the resource\ncan, up to 1,000 each on a resource.", + "description": "Comment on a moment or passage (`t0`\u2013`t1`, with the `quote` picked in the text) or on the whole resource, or\nwith `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone who can read the resource\ncan, up to 1,000 each on a resource. A comment that tags @lens gets the agent's answer on the thread.", "operationId": "create_comment", "parameters": [ { @@ -44181,7 +44181,13 @@ } ], "title": "Created By Name", - "description": "its writer's name, when they gave one" + "description": "its writer's name, when they gave one; Lens for the agent's answer" + }, + "by_agent": { + "type": "boolean", + "title": "By Agent", + "description": "the agent wrote it, answering a comment that tagged it", + "default": false }, "created_at": { "anyOf": [