diff --git a/CHANGELOG.md b/CHANGELOG.md index 435ba5864..e043c5b82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased +- **Deep research.** Give the assistant a goal (`/research `, `research: true` on a question, `POST + /api/v1/chats//research`, its `start_research` tool, or MCP) and it plans the research: up to three follow-up + questions, the sub-questions, where it looks (the archive, the graph, notes, the web, the code sandbox), the tool + calls it may make, the report's outline, and a labelled cost estimate. Approve the plan as it is or edited, revise + it, or say "yes" in a chat room; an API key or an app only declines it unless `research.agents_skip_review` is on. + It runs in the background, one run per person at a time, within its tool calls, `research.max_minutes` and your + budgets, survives a restart, and stops when asked; the cited report comes back in the conversation, its claims + checked against their sources, and saves as a note or makes a podcast episode. The web app's plan and progress + cards are in progress ([Deep research](docs/assistant.md#deep-research)). - **Read any message aloud.** Every question and answer in a conversation has a speaker button: one click reads it aloud by the speech model and voice picked in Settings → AI assistant → Voice (else the browser's own speech), another stops it. One message plays at a time, long answers go in pieces under every provider's limit, and each diff --git a/docs/assistant.md b/docs/assistant.md index 0e1cccded..05471c65b 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -346,6 +346,7 @@ tools it may use, how many tool calls it may make, and which namespaces it reads - `changes` (proposals that become approvals); - `files` (importing files and web pages); - `code` (the [code sandbox](#code-sandbox): Python and bash, once an admin turns it on); + - `research` ([deep research](#deep-research): `start_research` plans research that waits for your approval); - `extensions`; - `server` (admins); - `browser`; @@ -472,6 +473,72 @@ driver); the Podman driver for hosts without Docker; GPUs; a direct sandbox API assistant); one real run against OpenShell in CI (the tests use a fake gateway with OpenShell's own client); showing an image the sandbox shared inline in the answer, not only as a download; the 20-file cap as a setting. +## Deep research + +Built in the API, MCP, chat rooms and the assistant's tools; the web app's plan and progress cards are in progress. +Give the assistant a goal that needs many look-ups, and it comes back with a plan; once you approve it, the research +runs in the background and a cited report comes back in the conversation. + +- **Asking.** Start a question with `/research` (`/research what changed in the capsid work this year?`), send it with + `research: true` (`POST /api/v1/chats//messages`), call `POST /api/v1/chats//research {goal}`, or ask the + assistant for research: its `start_research` tool (the `research` [tool group](#choosing-tools-namespaces-and-a-limit)) + works from chat, voice and [chat rooms](chat-rooms.md). It needs a language model that uses tools. +- **The plan.** One model call reads the goal and writes up to three follow-up questions, each with answers to pick + from, and a plan: a title, the sub-questions it will answer, where it will look (`archive`, `graph`, `notes`, `web`, + `code`: only the sources this person has, and of those only what the conversation's choice of tools keeps), the + most tool calls it may make, and the report's outline. With it comes an estimate of what it will cost, always + labelled an estimate: from what each tool call of past research runs cost in the activity ledger, else its likely + tokens at the model's price in Settings (no figure when the model has no price). The plan is the answer to the + question, and it waits as an approval of the assistant's (tool `research`). +- **Approve, edit or revise.** Approve it as it is, or with your edits (`POST /api/v1/approvals/` with + `decision: approve` and `params`: any of `title`, `sub_questions`, `sources`, `max_tool_calls`, `report_outline`), in + the conversation or the Approvals page; or send your answers and edits back for one more planning call (`POST + /api/v1/research//revise {answers, plan}`), which changes the plan and its approval in place. In a chat room, + "yes" approves the plan as it is. Decline it and nothing runs. `review: false` on `POST /chats//research` starts + it at once, as your own pick. +- **Agents only decline.** An API key, an app or an agent (MCP, the assistant) can decline a plan but not approve it, + as for every approval, unless an admin turns on `research.agents_skip_review` (off by default): then an agent's plan + starts without review. The `research` settings are saved with `PUT /api/v1/settings/research`; their place in + Settings → AI assistant is in progress. +- **The run.** Once approved, one process runs it at a time (a claim it keeps with a heartbeat every 10 seconds; + another process takes it over after two minutes of silence and goes on from the last finished step), in stages: + - *gather*: each sub-question in turn, with the assistant's tool loop, its share of the tool calls that are left, + and the plan's sources' tools alone (read tools only). Searches and page reads on the web, and code runs that + reach the web, don't ask each time: approving the plan approved them. Any other tool that changes something + still asks. + - *write*: one call writes the report in Markdown, following the outline, citing [n] across every finding, + numbered as one list. + - *check*: each cited claim is checked against what it cites ([check the sources](#chat)); one that isn't + supported is rewritten once, or marked "(not supported by its sources)". + - *done*: the report is an answer in the conversation (`kind: research`), after its last message, with its + [record](#what-each-answer-records) (`research` is its id; its tokens and cost are the whole run's). +- **Limits.** A person has one run going at a time; the next waits its turn. `research.max_tool_calls` (40) caps a + plan's tool calls, and `research.max_minutes` (20) a run's time: past it, the report is written from what it found + and says which sub-questions it didn't get to (outcome `limit`). `research.model` picks another model for research + (empty: the conversation's). +- **Budgets.** Before each stage the [budgets](budgets.md) of the conversation's namespaces and `all` are checked. One + that is over holds the run (`held`, saying why) until the budget allows it, or until you let it go on anyway (`POST + /api/v1/research//resume`); a budget that skips stops it with what it found. +- **Progress and stopping.** `GET /api/v1/research/` gives its status (planned, declined, queued, running, held, + done, stopped, failed), stage, and progress: sub-questions done, sources found, tool calls, minutes, and tokens and + cost so far. `GET /api/v1/research?chat_id=` lists a conversation's. `POST /api/v1/research//stop` stops it + after the step it's on (at once while it waits), and what it found so far is saved in the conversation as its + answer, marked stopped. +- **After.** `POST /api/v1/research//note` saves the report as a [note](notes.md) in the namespace it looked in + (`namespace` when there are several), its archive citations as mention links to what they cite, web ones as links, + and a link back to the conversation (`@[Research: title](chat:12)`), which only you see named. `POST + /api/v1/research//podcast` makes a [podcast episode](podcasts.md) from the assets the report cites, about its + goal. The report reads aloud and exports like any answer. +- **Agents.** MCP has `start_research` (a new conversation unless `chat_id` is given; `namespaces`, `tools`, + `max_tool_calls`) and `research_status` (the plan and progress, and once it's done the report with its sources and + their links). +- **Cost.** Planning is one call; gathering makes at most the plan's tool calls, each round a call; writing and + checking are two or three calls. Every call is in the activity ledger for the conversation and `research:`. + +Refine later: sub-questions in parallel, research on a schedule, sharing a report with a namespace's members, research +as an automation action, a report template people can edit, and the report posted back into the chat room it was +asked in (today the room gets the plan, and the report is in the conversation in Lens). + ## Third-party content and prompt injection The assistant reads text nobody vouched for: what was said in recordings, documents, text read off screens, web pages diff --git a/docs/database.md b/docs/database.md index ef27b36fc..de85f27c8 100644 --- a/docs/database.md +++ b/docs/database.md @@ -90,7 +90,9 @@ Installs keep their data across updates (running `install.sh` again, a new image indexes arrive through `SCHEMA` on the next start. A field that is only added needs no step when its readers treat it as missing on older rows: a chat message's record (`meta`, [What each answer records](assistant.md#what-each-answer-records)) is one, and messages from before it simply have -none. Anything else, such as rewriting stored values, moving data between +none. [Deep research](assistant.md#deep-research) is another: its tables (`research`, and `research_run` for the claim +of the process running one) are new, a message's `kind` is missing on older ones, and notes accept `chat` links +without rewriting any. Anything else, such as rewriting stored values, moving data between tables, or changing or removing an index that already exists, is a named step in `app/domain/migrations.py`: * Every process that opens the database (the API, workers, `lens watch`, `lens` commands) runs the steps the database diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 6e500e406..8345abbd7 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -36,6 +36,7 @@ from app.api.v1.routes import namespaces as namespace_routes from app.api.v1.routes import notebook as note_routes from app.api.v1.routes import recordings as recording_routes +from app.api.v1.routes import research as research_routes from app.api.v1.routes import routines as routine_routes from app.api.v1.routes import search as search_routes from app.api.v1.routes import topics as topic_routes @@ -62,6 +63,7 @@ nsfw, rdf, render, + research, rss, settings, sources, @@ -87,7 +89,7 @@ AutomationTry, AutomationUpdate, ) -from app.schemas.chats import ChatSummary +from app.schemas.chats import ChatScope, ChatSummary from app.schemas.entities import EntityHide, EntityList, EntityRename, EntityRetype, EntityUpdate from app.schemas.notebook import NotePage from app.schemas.search import SearchResults @@ -2768,6 +2770,109 @@ def decline_approval(ctx: Context, id: int) -> dict[str, Any]: return {"approval": id, "status": out.get("status")} +# ---------- deep research (docs/assistant.md#deep-research) ---------- +@tool( + "start_research", + "Start deep research", + "Research a goal for the person: one planning call makes a plan (the sub-questions, where it looks: the archive, the " + "graph, notes, the web or the code sandbox, the tool calls it may make, the report's outline) and up to three " + "follow-up questions. The plan waits for the person's approval in Lens (url), unless the admin let agents start " + "research without review (research.agents_skip_review); approved, it runs in the background and its cited report " + "comes back in the conversation: follow it with research_status. Needs a token with the write scope.", + Arg("goal", "string", "what to research, in the person's words", required=True, max_length=4000), + Arg("chat_id", "integer", "the conversation to research in; a new one when left out", minimum=1), + Arg("namespaces", "string[]", "only these namespaces (a new conversation's scope)"), + Arg("tools", "string[]", "what it may use: tool groups (archive, graph, notes, web, code) and tools; else the conversation's choice"), + Arg("max_tool_calls", "integer", "the most tool calls it may make (research.max_tool_calls at most)", minimum=1, maximum=500), + writes=True, +) +def start_research( + ctx: Context, goal: str, chat_id: int | None, namespaces: list[str] | None, tools: list[str] | None, max_tool_calls: int | None +) -> dict[str, Any]: + _writer(ctx, "start research") + if not research.available(ctx.cfg): + raise ToolError("research needs a language model that uses tools: an admin sets one up in Settings › AI assistant") + asked = [ctx.ns] if ctx.ns else namespaces + scope = chat_routes._scope(ctx.acl, ctx.db, ChatScope(namespaces=asked)) if asked else None + if chat_id is not None: + c = chat_routes._own_chat(ctx.db, chat_id, ctx.user) + scope = scope if scope is not None else c.get("scope") + else: + chat_id = chat.create(ctx.db, ctx.user.id, goal[:80], scope) + c = chat.get(ctx.db, chat_id, ctx.user.id) + box = research_routes.toolbox(ctx.db, ctx.cfg, ctx.user, ctx.acl, chat_id, scope, None, ctx.request) + if tools is not None and box.unknown(tools): + raise ToolError(f"no tool or tool group {box.unknown(tools)[0]!r}") + box = research_routes.toolbox( + ctx.db, ctx.cfg, ctx.user, ctx.acl, chat_id, scope, tools if tools is not None else c.get("tools"), ctx.request + ) + skip = research.settings(ctx.cfg)["agents_skip_review"] + out = research.start( + ctx.db, + ctx.cfg, + box, + " ".join(goal.split()), + "api", + model=research.model_for(ctx.cfg), + max_tool_calls=max_tool_calls, + options={"tools": tools, "max_tool_calls": max_tool_calls}, + review=not skip, + ) + if out.get("error"): + raise ToolError(out["error"]) + r = research.view(ctx.db, out["research"]) + return { + "research": r["id"], + "chat_id": chat_id, + "status": r["status"], + "plan": r["plan"], + "questions": r["questions"], + "estimate": r["estimate"], + "approval": r["approval"], + "url": f"{ctx.web}/chat/{chat_id}", + "next": "running: follow it with research_status" + if r["status"] in ("queued", "running") + else "waiting for the person to approve the plan in Lens (url); follow it with research_status", + } + + +@tool( + "research_status", + "Research status", + "Where a deep research run stands: its plan and status (planned: waiting for the person's approval; queued, " + "running, held by a budget, done, stopped, failed), its stage (gather, write, check, done) and progress " + "(sub-questions done, sources found, tool calls and cost so far). Once it's done (or stopped), the report in " + "Markdown with its numbered sources, each with a link.", + Arg("id", "integer", "the research (start_research's)", required=True, minimum=1), +) +def research_status(ctx: Context, id: int) -> dict[str, Any]: # noqa: A002 - the argument's name + try: + r = research.mine(ctx.db, id, ctx.user.id) + except KeyError: + raise ToolError(f"no research {id} of yours") from None + out = research.view(ctx.db, r) + out["url"] = f"{ctx.web}/chat/{r['chat']}" + if r.get("answer"): + m = ctx.db.one("SELECT content, passages FROM $r", r=R("chat_message", r["answer"])) or {} + ok = None if ctx.user.admin else set(ctx.acl.spaces()) + spaces = { + x["id"]: x["space"] + for x in ctx.db.rows( + "SELECT record::id(id) AS id, space FROM recording WHERE id IN $ids", + ids=[R("recording", p["recording_id"]) for p in m.get("passages") or [] if p.get("recording_id") is not None], + ) + } + sources = [] + for p in m.get("passages") or []: + if p.get("source") == "web": + sources.append({"n": p["n"], "title": p.get("title"), "url": p.get("url"), "site": p.get("site")}) + elif p.get("recording_id") in spaces and (ok is None or spaces[p["recording_id"]] in ok): + sources.append({"n": p["n"], "title": p.get("title"), "at": p.get("time"), "url": ctx.link(p["recording_id"], p.get("t0"))}) + out["report"] = m.get("content") + out["sources"] = sources + return out + + @tool( "event_kinds", "Kinds of activity", diff --git a/fastapi_backend/app/api/v1/router.py b/fastapi_backend/app/api/v1/router.py index 701471d63..344457eb4 100644 --- a/fastapi_backend/app/api/v1/router.py +++ b/fastapi_backend/app/api/v1/router.py @@ -55,6 +55,7 @@ rdf, recordings, requests, + research, routines, search, searches, @@ -136,6 +137,7 @@ extensions, marketplaces, chats, + research, browser, collections, batches, diff --git a/fastapi_backend/app/api/v1/routes/chats.py b/fastapi_backend/app/api/v1/routes/chats.py index e7de390d7..040995d09 100644 --- a/fastapi_backend/app/api/v1/routes/chats.py +++ b/fastapi_backend/app/api/v1/routes/chats.py @@ -2,7 +2,7 @@ A question streams back server-sent events: ``branch`` (a new version: regenerated, edited, or beside others), ``scoped`` (the namespace a conversation over everything was narrowed -to) or ``suggested`` (namespaces to offer when it isn't clear, new ones included), ``step`` (a tool the assistant used), ``approval`` (work it proposed that waits for you), ``notice``, ``passages`` (the numbered excerpts), ``token`` (answer text), ``error``, ``meta`` (the answer's record), ``done`` and, after a first answer, ``title`` (the title Lens gave the conversation). +to) or ``suggested`` (namespaces to offer when it isn't clear, new ones included), ``step`` (a tool the assistant used), ``research`` (a research goal's plan, research.py), ``approval`` (work it proposed that waits for you), ``notice``, ``passages`` (the numbered excerpts), ``token`` (answer text), ``error``, ``meta`` (the answer's record), ``done`` and, after a first answer, ``title`` (the title Lens gave the conversation). The assistant only ever sees, and cites, what the asker can read. """ @@ -35,6 +35,7 @@ chat_export, convert, llm, + research, sandbox, store, uploads, @@ -427,6 +428,71 @@ def _ev(name: str, data: Any) -> str: return f"event: {name}\ndata: {json.dumps(sign_urls(data, full=True), default=str)}\n\n" +def _research_reply( + cid: int, + c: dict[str, Any], + goal: str, + said: str, + under: Any, + model: str | None, + tools: list[str] | None, + body: MessageCreate, + own_scope: dict[str, Any] | None, + user: Principal, + acl: Access, + db: DB, + cfg: dict[str, Any], + request: Request, +) -> StreamingResponse: + """A question that is a research goal (`research: true`, or /research ): its answer is a plan to approve + (research.start). Streams research (the plan, as GET /research/{id}), approval, passages, token (the plan as + text), meta and done; error when the plan couldn't be made.""" + if not research.available(cfg): + raise HTTPException(400, "research needs a language model that uses tools (Settings › AI assistant)") + scope = own_scope if own_scope is not None else c.get("scope") + + def run() -> Iterator[str]: + box = ai_tools.Toolbox( + db, + cfg, + user.as_audit(), + set(acl.roles), + set(acl.editable()), + scope, + cid, + request.app.state.archive.base, + user.admin, + only=tools, + ) + if c["title"] == "New conversation": + db.q("UPDATE $r SET title = $t", r=R("chat", cid), t=goal[:80]) + via = "voice" if body.via == "voice" else "web" if user.via == "access" else "api" + options = {"model": body.model, "tools": body.tools, "max_tool_calls": body.max_tool_calls, "scope": own_scope} + out = research.start( + db, + cfg, + box, + goal, + via, + said=said, + parent=under, + model=research.model_for(cfg, model), + max_tool_calls=body.max_tool_calls, + options=options, + ) + if out.get("error"): + yield _ev("error", {"message": out["error"]}) + else: + yield _ev("research", research.view(db, out["research"])) + yield _ev("approval", out["approval"]) + yield _ev("passages", []) + yield _ev("token", {"text": out["text"]}) + yield _ev("meta", out["meta"]) + yield _ev("done", {"message": out["message"]}) + + return StreamingResponse(run(), media_type="text/event-stream", headers=SSE_HEADERS) + + @router.post( "/chats/{cid}/messages", response_class=StreamingResponse, @@ -455,6 +521,10 @@ async def send_message(cid: int, body: MessageCreate, user: Writer, acl: Acl, db `/clear` makes the model start afresh (the messages stay to read) and `/compact [what to keep]` folds the conversation so far into its summary now; each streams cleared or compacted, then its reply as an answer. + With `research` (or /research ) the content is a research goal: the answer is a plan to approve (streamed as + `research`, then `approval`), which runs in the background once approved (POST /chats/{cid}/research, GET + /research/{id}). + With `browser` (the session of a browser's Lens side panel, POST /browser/sessions), the assistant can use that browser while the side panel is connected: open pages, read them, click, type and download files into the conversation (docs/browser-assistant.md).""" @@ -508,6 +578,9 @@ def run() -> Iterator[str]: tools = c.get("tools") limit = _limit(cfg, body.max_tool_calls) if body.max_tool_calls is not None else _most(cfg, c.get("max_tool_calls")) own_scope = await run_in_threadpool(_scope, acl, db, body.scope) if body.scope is not None else None + goal = (q if body.research else research.command(q)) if not files and body.edit is None and again is None else None + if goal: # a research goal: the answer is a plan to approve (research.py) + return _research_reply(cid, c, goal, q, under, model, tools, body, own_scope, user, acl, db, cfg, request) record = chat.Record(cfg, model, picked=tools, tool_calls=limit) # what produced the answer (from here: it times it) retitle = chat.lens_titles(c) # whether a first question titles the conversation (nobody has named it) page = body.context.model_dump() if body.context else None @@ -750,15 +823,16 @@ def _via(user: Principal) -> str: return "web" if user.via == "access" else user.via -def approval_view(a: dict, via: str = "web") -> Any: +def approval_view(a: dict, via: str = "web", cfg: dict[str, Any] | None = None) -> Any: """An approval as the API shows it: an automation's intent with credentials hidden, and what `via` can do.""" - return {**a, "intent": approvals.shown(a.get("intent")), "told": a.get("told") or [], "can": approvals.can(a, via)} + return {**a, "intent": approvals.shown(a.get("intent")), "told": a.get("told") or [], "can": approvals.can(a, via, cfg)} @router.get("/approvals") def list_approvals( user: CurrentUser, db: Db, + cfg: Cfg, chat_id: int | None = None, status: Literal["pending", "running", "done", "failed", "declined", "expired"] | None = None, origin: Literal["automation", "chat"] | None = None, @@ -770,7 +844,7 @@ def list_approvals( actions. Filter by `status` (pending is waiting), `origin` (automation or chat), one `automation` or one chat (`chat_id`); page with `before`.""" rows = approvals.listing(db, user.id, status, origin, automation, chat_id, before, limit) - return [approval_view(a, _via(user)) for a in rows] + return [approval_view(a, _via(user), cfg) for a in rows] @router.get("/approvals/count") @@ -780,12 +854,12 @@ def count_approvals(user: CurrentUser, db: Db) -> ApprovalCount: @router.get("/approvals/{aid}") -def get_approval(aid: int, user: CurrentUser, db: Db) -> Approval: +def get_approval(aid: int, user: CurrentUser, db: Db, cfg: Cfg) -> Approval: """One of your approvals.""" a = approvals.get(db, aid) if not a or a["account"] != user.id: raise HTTPException(404, "not found") - return approval_view(a, _via(user)) + return approval_view(a, _via(user), cfg) @router.post("/approvals/{aid}") diff --git a/fastapi_backend/app/api/v1/routes/notebook.py b/fastapi_backend/app/api/v1/routes/notebook.py index 3872b1d2c..89733e213 100644 --- a/fastapi_backend/app/api/v1/routes/notebook.py +++ b/fastapi_backend/app/api/v1/routes/notebook.py @@ -78,8 +78,9 @@ def _out(acl: Access, db: DB, p: dict[str, Any], user: Principal) -> NotePage: out_links = [] for x in links: t = found.get(x["target"]) - seen = t is not None and t.get("space") in readable - out_links.append({**x, "name": t["name"] if seen else None, "namespace": names.get(t["space"]) if seen else None}) + # a conversation (chat:12) is named only for the person whose conversation it is + seen = t is not None and (t.get("account") == user.id if x["target"].startswith("chat:") else t.get("space") in readable) + out_links.append({**x, "name": t["name"] if seen else None, "namespace": names.get(t.get("space")) if seen else None}) writer = db.one("SELECT email FROM $r", r=store.R("account", p["created_by"])) if p.get("created_by") else None targets = [f"page:{p['id']}"] + ([p["about"]] if p.get("about") else []) return NotePage( diff --git a/fastapi_backend/app/api/v1/routes/research.py b/fastapi_backend/app/api/v1/routes/research.py new file mode 100644 index 000000000..02e382393 --- /dev/null +++ b/fastapi_backend/app/api/v1/routes/research.py @@ -0,0 +1,196 @@ +"""Deep research (app/domain/research.py, docs/assistant.md#deep-research): a goal gets a plan to approve; approved, +it runs in the background and its cited report comes back in the conversation. Its plan's approval is decided like +any other (POST /approvals/{id}); these routes start one, follow it, stop it and use its report.""" + +from __future__ import annotations + +from typing import Any + +from fastapi import APIRouter, HTTPException, Request + +from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer +from app.api.v1.routes import podcasts as podcast_routes +from app.api.v1.routes.chats import _own_chat, _scope, _tools, _via +from app.domain import ai_tools, auth, chat, llm, research, store +from app.domain.store import DB +from app.schemas.podcasts import PodcastCreate, PodcastSelection, PodcastStarted +from app.schemas.research import ( + Research, + ResearchNote, + ResearchNoteMade, + ResearchPodcast, + ResearchRevise, + ResearchStart, + ResearchStarted, + ResearchStop, +) + +router = APIRouter(tags=["research"]) + + +def toolbox( + db: DB, + cfg: dict[str, Any], + user: Principal, + acl: Access, + cid: int, + scope: dict[str, Any] | None, + tools: list[str] | None, + request: Request, +) -> ai_tools.Toolbox: + """The person's tools in the conversation `cid`, as research plans with them.""" + return ai_tools.Toolbox( + db, + cfg, + user.as_audit(), + set(acl.roles), + set(acl.editable()), + scope, + cid, + request.app.state.archive.base, + user.admin, + only=tools, + ) + + +def _mine(db: DB, rid: int, user: Principal) -> dict[str, Any]: + try: + return research.mine(db, rid, user.id) + except KeyError: + raise HTTPException(404, "not found") from None + + +def view(db: DB, r: dict[str, Any]) -> Research: + return Research.model_validate(research.view(db, r)) + + +@router.post("/chats/{cid}/research") +def start_research(cid: int, body: ResearchStart, user: Writer, acl: Acl, db: Db, cfg: Cfg, request: Request) -> ResearchStarted: + """Start research on a goal in a conversation: the goal is saved as a question and one model call plans it (up to + three follow-up questions with answers to pick, the sub-questions, where it looks, the tool calls it may make, the + report's outline, and an estimate of the cost). The plan waits as an approval (POST /approvals/{id}: approve it as + it is, or with `params` its edits; or decline), or revise it (POST /research/{id}/revise). With `review` false it + starts at once: your own pick when signed in; an API key or an app needs research.agents_skip_review. The same as a + question with `research: true`, or /research .""" + c = _own_chat(db, cid, user) + if not research.available(cfg): + raise HTTPException(400, "research needs a language model that uses tools (Settings › AI assistant)") + via = _via(user) + if not body.review and not research.agents_may_approve(cfg, via): + raise HTTPException(403, "starting research without review is the person's own pick (research.agents_skip_review)") + scope = _scope(acl, db, body.scope) if body.scope is not None else c.get("scope") + pre = toolbox(db, cfg, user, acl, cid, scope, None, request) + tools = _tools(pre, body.tools) if body.tools is not None else c.get("tools") + if body.max_tool_calls is not None and body.max_tool_calls > research.settings(cfg)["max_tool_calls"]: + raise HTTPException( + 400, f"max_tool_calls is a whole number from 1 to {research.settings(cfg)['max_tool_calls']} (research.max_tool_calls)" + ) + box = toolbox(db, cfg, user, acl, cid, scope, tools, request) + if not research.sources_of(box): + raise HTTPException(400, "none of the tools you picked can look things up for research") + kept = c.get("model") # the conversation's model, while it's still offered + model = research.model_for(cfg, kept if kept and kept in chat.model_choices(cfg) else None) + if c["title"] == "New conversation": + db.q("UPDATE $r SET title = $t", r=store.R("chat", cid), t=body.goal[:80]) + out = research.start( + db, + cfg, + box, + " ".join(body.goal.split()), + "web" if via == "web" else "api", + model=model, + max_tool_calls=body.max_tool_calls, + options={"tools": body.tools, "max_tool_calls": body.max_tool_calls, "scope": scope if body.scope is not None else None}, + review=body.review, + ) + if out.get("error"): + raise HTTPException(400, out["error"]) + return ResearchStarted( + research=view(db, out["research"]), question=out["question"], message=out["message"], text=out["text"], meta=out["meta"] + ) + + +@router.get("/research") +def list_research(user: CurrentUser, db: Db, chat_id: int | None = None, limit: int = 50) -> list[Research]: + """Your research, newest first; with `chat_id`, that conversation's (to show each plan's or run's card).""" + where = "account = $a" + (" AND chat = $c" if chat_id is not None else "") + rows = db.rows( + f"SELECT *, record::id(id) AS id FROM research WHERE {where} ORDER BY id DESC LIMIT $n", + a=user.id, + c=chat_id, + n=max(1, min(limit, 200)), + ) + return [view(db, r) for r in rows] + + +@router.get("/research/{rid}") +def get_research(rid: int, user: CurrentUser, db: Db) -> Research: + """A research run: its plan, status and stage (gather, write, check, done), progress (sub-questions done, sources + found, tool calls and cost so far, from the activity ledger) and, once it's done, the report's message + (`answer`, in the conversation). What it waits for when a budget holds it (`held`), and why it ended early.""" + return view(db, _mine(db, rid, user)) + + +@router.post("/research/{rid}/revise") +def revise_research(rid: int, body: ResearchRevise, user: Writer, acl: Acl, db: Db, cfg: Cfg, request: Request) -> Research: + """Send your answers to the follow-up questions and your edits to the plan back for one more planning call; the + plan and its approval change in place. Only while the plan waits for approval.""" + r = _mine(db, rid, user) + box = toolbox(db, cfg, user, acl, r["chat"], r.get("scope"), r.get("tools"), request) + try: + return view(db, research.revise(db, cfg, box, r, body.answers, body.plan)) + except (ValueError, llm.LLMError) as e: + raise HTTPException(400, str(e)) from None + + +@router.post("/research/{rid}/stop") +def stop_research(rid: int, user: Writer, db: Db, cfg: Cfg) -> ResearchStop: + """Stop a run: it stops after the step it's on and keeps what it found, saved in the conversation as its answer, + marked stopped (one that hasn't started, or waits on a budget, stops now). Works whichever process runs it.""" + _mine(db, rid, user) + return ResearchStop(stopping=research.request_stop(db, cfg, rid)) + + +@router.post("/research/{rid}/resume") +def resume_research(rid: int, user: Writer, db: Db, cfg: Cfg) -> Research: + """Let a run that a budget holds go on anyway (your pick, as "run it once" is for a held routine run).""" + _mine(db, rid, user) + try: + return view(db, research.go_on(db, cfg, rid, user.as_audit())) + except ValueError as e: + raise HTTPException(409, str(e)) from None + + +@router.post("/research/{rid}/note") +def research_note(rid: int, user: Writer, acl: Acl, db: Db, body: ResearchNote | None = None) -> ResearchNoteMade: + """Save the report as a note: in the namespace it looked in (`namespace` when there are several), its archive + citations as links to the assets they cite, web citations as links, and a link back to the conversation (shown to + you alone). Needs editor access there.""" + r = _mine(db, rid, user) + try: + out = research.to_note(db, r, user.id, set(acl.editable()), (body or ResearchNote()).namespace) + except PermissionError as e: + raise HTTPException(403, str(e)) from None + except ValueError as e: + raise HTTPException(400, str(e)) from None + auth.audit(db, user.as_audit(), "research.note", f"note_page:{out['page']}", {"research": rid}) + return ResearchNoteMade(**out) + + +@router.post("/research/{rid}/podcast", status_code=202) +def research_podcast(rid: int, user: Writer, acl: Acl, db: Db, cfg: Cfg, body: ResearchPodcast | None = None) -> PodcastStarted: + """Make a podcast episode from the assets the report cites (POST /podcasts with them picked): about the research + goal unless `prompt` says otherwise, titled after the report.""" + r = _mine(db, rid, user) + body = body or ResearchPodcast() + picked = research.podcast_sources(db, r, acl.readable()) + if not picked: + raise HTTPException(400, "the report cites no assets from the archive to make an episode from") + ask = PodcastCreate( + selection=PodcastSelection(recordings=picked[:100]), + prompt=body.prompt or (r.get("goal") or "")[:2000] or None, + length=body.length, + style=body.style, + title=body.title or (r.get("plan") or {}).get("title"), + ) + return podcast_routes.start(acl, user, db, cfg, ask) diff --git a/fastapi_backend/app/domain/ai_tools.py b/fastapi_backend/app/domain/ai_tools.py index 71b56d481..bba22e734 100644 --- a/fastapi_backend/app/domain/ai_tools.py +++ b/fastapi_backend/app/domain/ai_tools.py @@ -33,6 +33,7 @@ ops_tools, recsets, render, + research as researchmod, sandbox as sandboxmod, search as searchmod, browser as browsermod, @@ -541,6 +542,7 @@ def builtin_names(): + AUTOMATION_TOOLS + browsermod.TOOLS + sandboxmod.TOOLS + + researchmod.TOOLS ) return {t[0] for t in every + webmod.TOOLS} | {"use_skill"} @@ -564,7 +566,14 @@ def builtin_names(): "decline_approval", } | { t[0] - for t in TOOLS + ops_tools.ADMIN_TOOLS + ops_tools.FILE_TOOLS + AUTHOR_TOOLS + MEMORY_TOOLS + AUTOMATION_TOOLS + sandboxmod.TOOLS + for t in TOOLS + + ops_tools.ADMIN_TOOLS + + ops_tools.FILE_TOOLS + + AUTHOR_TOOLS + + MEMORY_TOOLS + + AUTOMATION_TOOLS + + sandboxmod.TOOLS + + researchmod.TOOLS if t[4] } @@ -620,6 +629,7 @@ def builtin_names(): }, "files": {"label": "Importing files", "tools": [t[0] for t in ops_tools.FILE_TOOLS]}, "code": {"label": "Code sandbox", "tools": [t[0] for t in sandboxmod.TOOLS]}, + "research": {"label": "Deep research", "tools": [t[0] for t in researchmod.TOOLS]}, "extensions": {"label": "Extensions and skills", "tools": [t[0] for t in AUTHOR_TOOLS] + ["use_skill"]}, "server": {"label": "The server", "tools": [t[0] for t in ops_tools.ADMIN_TOOLS]}, "browser": {"label": "Your browser", "tools": [t[0] for t in browsermod.TOOLS]}, @@ -644,6 +654,8 @@ def _why_missing(group, cfg): return "the code sandbox is off", "sandbox.enabled" if group == "code" and sandboxmod.settings(cfg)["who"] == "admins": return "for admins", "sandbox.who" + if group == "research" and researchmod.unavailable(cfg): + return researchmod.unavailable(cfg) if group == "extensions" and not cfg["ai"].get("extensions", True): return "switched off by an admin", "ai.extensions" if all(t in off for t in GROUPS[group]["tools"]): @@ -680,6 +692,8 @@ def catalog(box): offered |= {t for t in GROUPS["browser"]["tools"] if t not in off} if sandboxmod.offered(box.cfg, box.admin, bool(box.editable)): # every conversation of theirs has its sandbox offered |= {t for t in GROUPS["code"]["tools"] if t not in off} + if researchmod.available(box.cfg): # every conversation of theirs can start research + offered |= {t for t in GROUPS["research"]["tools"] if t not in off} # code that can reach another host can carry out what it's given runs_out = {"run_python", "run_bash"} if sandboxmod.sends_out(box.cfg) else set() built = { @@ -692,6 +706,7 @@ def catalog(box): + AUTOMATION_TOOLS + sandboxmod.TOOLS + webmod.TOOLS + + researchmod.TOOLS } out = [] for name, g in GROUPS.items(): @@ -732,7 +747,9 @@ class Toolbox(ops_tools.OpsTools): `account`: the person an automation acts for, when `user` is the automation itself (automation:, the by of what it writes): their approvals, extensions, view of the timeline and marks on it; `origin` ({automation, run, node}) is kept on the approvals it asks for. `only`: the groups (GROUPS, an extension's mcp: or ext:) - and tools the question was given (None: every tool this person may use; an admin's switch still wins).""" + and tools the question was given (None: every tool this person may use; an admin's switch still wins). + `allowed_out`: tools that send data out of Lens and don't ask first even after the answer read outside text, + because the person already approved them for this work (an approved research plan's web search, research.py).""" def __init__( self, @@ -753,9 +770,11 @@ def __init__( account=None, origin=None, only=None, + allowed_out=(), ): self.db, self.cfg, self.user, self.chat, self.said = db, cfg, user, chat_id, said self.only = None if only is None else set(only) + self.allowed_out = set(allowed_out or ()) self.account, self.origin = user["id"] if account is None else account, origin # where this answer has read text from outside Lens so far (`outside`: before any tool ran, such as the page it # was asked from or a chat room's message): once it has, tools that send data out of Lens or change the server @@ -870,6 +889,8 @@ def specs(self): tools += sandboxmod.TOOLS if self.web: tools += webmod.TOOLS + if self.chat is not None and researchmod.available(self.cfg): + tools += researchmod.TOOLS if self.ext: tools += self.ext.tool_specs(can_act or self.admin) keep = self.chosen() @@ -1102,7 +1123,7 @@ def _sandbox(self, fn, *args): def _sandbox_run(self, kind, source, files): source = str(source or "") files = [str(f) for f in ([files] if isinstance(files, str) else files or [])] - if sandboxmod.sends_out(self.cfg) and self.wary(): + if sandboxmod.sends_out(self.cfg) and self.wary() and f"run_{kind}" not in self.allowed_out: # code that reaches another host can carry out what the answer read: after text from outside Lens, which # may have put it there, a run asks first first = next((x.strip() for x in source.splitlines() if x.strip()), "")[:80] @@ -1142,8 +1163,9 @@ def t_import_sandbox_file(self, path, namespace=None): def _web(self, tool, args, what, own): """A web search or page read, as (what it found, None), or (None, the approval it waits for): the words or the address leave the machine, so once this answer has read text from outside Lens, which may have put them there, - it asks first, unless the person wrote them (`own`) or already approved the same call in this conversation.""" - if own or not self.wary(): + it asks first, unless the person wrote them (`own`), already approved the same call in this conversation, or + approved this tool for the work it's doing (allowed_out: an approved research plan's).""" + if own or not self.wary() or tool in self.allowed_out: return webmod.run(self.cfg, tool, args), None rows = ( self.db.rows( @@ -1201,6 +1223,25 @@ def t_web_read(self, url, find=None): "web_read", args, out ) + # ---- deep research (research.py): a plan to approve, then a run in the background ---- + def t_start_research(self, goal): + goal = " ".join(str(goal or "").split()) + if not goal: + raise ValueError("give the goal to research") + if self.chat is None: + raise ValueError("research runs in a conversation") + model = researchmod.model_for(self.cfg, (self.db.one("SELECT model FROM $r", r=R("chat", self.chat)) or {}).get("model")) + r = researchmod.propose(self.db, self.cfg, self, goal, model) + out = { + "status": "waiting for the person's approval", + "research": r["id"], + "approval": r["approval"], + "plan": r["plan"], + "questions": r["questions"], + "estimate": r["estimate"], + } + return out, f"Planned {researchmod.summary(r['plan'])}; it waits for the person's approval" + # ---- extensions, from chat ---- def _ext_named(self, name): for g in extensions.visible(self.db, self.me): @@ -2158,6 +2199,8 @@ def approve(db, cfg, aid, user, editable, decision="approve", base=None, admin=F a = approvals.get(db, aid) if not a or a["status"] != "pending": raise ValueError("nothing to approve") + if a["tool"] == "research": # a research plan: approved (as it is, or with `params` its edits) or declined + return researchmod.decide(db, cfg, a, user, decision, params, via) if via == "agent" and decision != "decline": raise PermissionError("approving is the person's own pick: an agent can only decline") if a.get("intent"): diff --git a/fastapi_backend/app/domain/approvals.py b/fastapi_backend/app/domain/approvals.py index fc3cd61a1..be98ac046 100644 --- a/fastapi_backend/app/domain/approvals.py +++ b/fastapi_backend/app/domain/approvals.py @@ -260,11 +260,21 @@ def edited(intent, new): return out -def can(row, via="web"): +def can(row, via="web", cfg=None): """What the person can do with a waiting approval from where they are (`via`: web for a signed-in session, chat for - their own chat app, token or oauth for an API key or an app, agent for MCP and the assistant, which only decline).""" + their own chat app, token or oauth for an API key or an app, agent for MCP and the assistant, which only decline). + A research plan is approved by the person only (edited in the web app), unless research.agents_skip_review is on + in `cfg`.""" pending = row.get("status") == "pending" intent = row.get("intent") + if row.get("tool") == "research": + skip = bool(((cfg or {}).get("research") or {}).get("agents_skip_review")) + return { + "approve": pending and (via in ("web", "chat") or skip), + "decline": pending, + "edit": pending and via == "web", + "always": False, + } if not intent: return {"approve": pending and via != "agent", "decline": pending, "edit": False, "always": False} person = via in ("web", "chat") diff --git a/fastapi_backend/app/domain/bridge.py b/fastapi_backend/app/domain/bridge.py index a2c829c07..f845846ab 100644 --- a/fastapi_backend/app/domain/bridge.py +++ b/fastapi_backend/app/domain/bridge.py @@ -38,7 +38,21 @@ import urllib.error import urllib.request -from . import activity, ai_tools, auth, auto_scope, channels, chat, graph_history, llm, matterbridge, ns_assistant, store, untrusted +from . import ( + activity, + ai_tools, + auth, + auto_scope, + channels, + chat, + graph_history, + llm, + matterbridge, + ns_assistant, + research, + store, + untrusted, +) from .store import R log = logging.getLogger(__name__) @@ -330,6 +344,25 @@ def _answer(db, cfg, base, account, cid, q, files, act_on, outside): past = chat.path(db, cid) # a room goes on from the head (the version showing, should it be switched in the app) summary = chat.memory(db, cfg, cid, past) scope = (db.one("SELECT scope FROM $r", r=R("chat", cid)) or {}).get("scope") or None + goal = research.command(q) if not files else None + if goal and research.available(cfg): # /research : a plan, approved with a "yes" here + box = ai_tools.Toolbox( + db, + cfg, + {"id": account["id"], "email": account["email"]}, + readable, + editable, + scope, + cid, + base, + account["admin"], + said=q, + outside=list(outside) + ai_tools.read_before(past, summary), + ) + out = research.start(db, cfg, box, goal, "room", said=q) + if out.get("error"): + return "I couldn't plan that: " + out["error"], [], set() + return out["text"], [out["approval"]], set() chat.add(db, cid, "user", q, attachments=list(files) or None, meta=chat.question_record("room")) # the answer's id up front: every call made for it names it in the ledger, and its record sums them mid = db.next_id("chat_message") @@ -392,9 +425,9 @@ def said(words, cited, outcome, **kw): # ---------- approving in the room ---------- # what can be approved by a reply in the room: changes to the entities and topics (history keeps them, and merges and -# deletes can be undone) and new namespaces; settings, imports, batch runs, extensions and forgetting are approved in -# the web app -ROOM_SAFE = {"propose_entity_change", "propose_topic_change", "create_namespace"} +# deletes can be undone), new namespaces and a research plan as it is (it only reads, within the plan, and can be +# stopped); settings, imports, batch runs, extensions and forgetting are approved in the web app +ROOM_SAFE = {"propose_entity_change", "propose_topic_change", "create_namespace", "research"} YES_RX = re.compile(r"^(?:yes|y|yep|approve|approved|ok|okay|do it|go ahead)(?:\s+#?(\d+|all))?\s*[.!]*$", re.I) NO_RX = re.compile(r"^(?:no|n|nope|decline|cancel|don't)(?:\s+#?(\d+|all))?\s*[.!]*$", re.I) diff --git a/fastapi_backend/app/domain/chat.py b/fastapi_backend/app/domain/chat.py index b6eb6cbaf..c7f1212e0 100644 --- a/fastapi_backend/app/domain/chat.py +++ b/fastapi_backend/app/domain/chat.py @@ -375,7 +375,7 @@ def history(db, cid, ids=None): """A conversation's messages, oldest first: every version's, or only those with `ids` (a path).""" return db.rows( "SELECT record::id(id) AS id, role, content, passages, created_at, stopped ?? false AS stopped, steps ?? [] AS steps, " - "attachments ?? [] AS attachments, notice, error, check, model, context, parent, meta FROM chat_message WHERE chat = $c" + "attachments ?? [] AS attachments, notice, error, check, model, context, parent, meta, kind FROM chat_message WHERE chat = $c" + (" AND id IN $ids" if ids is not None else "") + " ORDER BY id", c=cid, @@ -510,13 +510,15 @@ def add( parent=HEAD, mid=None, meta=None, + kind=None, ): """Save a message under `parent` (by default the head; None: a first message) and make it the head; an answer keeps the tool steps it took, any notice (e.g. the model can't use tools) and error; a question asked from a page keeps what it shared of it (shared_context). A question an automation asked keeps its `origin` ({automation, run, node}), so it isn't taken for a person's message (`chat.message`). `mid`: an id taken earlier with db.next_id("chat_message"), after the parent was saved (an answer's, so the ledger rows of the calls made for it - name it). `meta` is its record (Record; a question's: how it was sent).""" + name it). `meta` is its record (Record; a question's: how it was sent). `kind`: research for a research report, + research_plan for the plan it waits on (research.py).""" parent = head(db, cid) if parent == HEAD else parent mid = mid or db.next_id("chat_message") db.q( @@ -539,6 +541,7 @@ def add( "origin": origin or None, "parent": parent, "meta": meta or None, + "kind": kind, } ), ) diff --git a/fastapi_backend/app/domain/notebook.py b/fastapi_backend/app/domain/notebook.py index b4ab881a8..7faedcccb 100644 --- a/fastapi_backend/app/domain/notebook.py +++ b/fastapi_backend/app/domain/notebook.py @@ -10,6 +10,8 @@ @[Weekly call](recording:12) @[Ada Lovelace](entity:5) @[Plans](page:3) #[Capsid design](topic:9) @ is for resources, people and other pages; # is for topics, the namespace's vocabulary (topics.py). Links are kept as note_link rows, so a page shows its backlinks and the graph sees them. +@[Research: capsids](chat:12) links a conversation (a research report's note links back to the one it came from), +which is named and linked only for the person whose conversation it is. The editor may also keep its own document state (`doc`, opaque: a BlockSuite/Yjs snapshot) next to the Markdown, which holds what Markdown can't, like drawings on the edgeless canvas. A change to the Markdown alone (the assistant's, say) @@ -40,7 +42,9 @@ PLACES = ("project", "area", "resource", "archive") AUTHORS = ("person", "assistant") VIEWS = ("page", "edgeless") # how the editor shows it: a document, or the endless canvas (drawings, diagrams) -KINDS = ("recording", "entity", "topic", "collection", "speaker", "page") # what a page can link to +# what a page can link to; a conversation (chat, as a research report's note links back to it) shows as a link only +# to the person whose conversation it is +KINDS = ("recording", "entity", "topic", "collection", "speaker", "page", "chat") ABOUT = ("recording", "entity", "topic", "collection", "speaker") # what can have a page of its own TITLE_MAX = 200 SUMMARY_MAX = 300 @@ -877,7 +881,8 @@ def backlinks(db, sid, targets): def labels(db, targets): - """Current names for link targets: {"recording:12": "Weekly call"}; the ones that are gone are left out.""" + """Current names for link targets: {"recording:12": "Weekly call"}; the ones that are gone are left out. A + conversation's (chat:12) has its `account` instead of a `space`: only that person sees its name.""" by_kind = {} for t in targets: kind, _, key = t.partition(":") @@ -887,6 +892,12 @@ def labels(db, targets): for kind, keys in by_kind.items(): table = "note_page" if kind == "page" else kind name = {"recording": "title ?? path", "page": "title", "speaker": "name ?? label", "topic": "label"}.get(kind, "name") + if kind == "chat": + for r in db.rows( + "SELECT record::id(id) AS id, account, title AS name FROM chat WHERE id IN $ids", ids=[R("chat", k) for k in keys] + ): + out[f"chat:{r['id']}"] = {"name": r.get("name"), "account": r.get("account")} + continue for r in db.rows( f"SELECT record::id(id) AS id, space, {name} AS name FROM {table} WHERE id IN $ids", ids=[R(table, k) for k in keys] ): diff --git a/fastapi_backend/app/domain/research.py b/fastapi_backend/app/domain/research.py new file mode 100644 index 000000000..e1f052b53 --- /dev/null +++ b/fastapi_backend/app/domain/research.py @@ -0,0 +1,1197 @@ +"""Deep research (docs/assistant.md, "Deep research"): give a goal, approve a plan, get a cited report back. + +A goal (`research: true` on a question, `/research `, POST /chats/{cid}/research, the MCP tool or the +assistant's own `start_research`) gets one planning call: up to three follow-up questions, each with answers to pick +from, and a plan (the sub-questions it will answer, the sources it will look in, the tool calls it may make, the +report's outline). The plan is saved as a `research` row and waits as an approval of the assistant's (tool +`research`), with an estimate of what it will cost. The person approves it as it is or edited (Enter in the chat, +the Approvals page, "yes" in a chat app), or revises it (their answers and edits go back for one more planning call). +An API key or an app only declines it, unless research.agents_skip_review lets agents start runs without review. + +Approved, it runs in the background, claimed by one process (CREATE-as-claim on `research_run:`, a heartbeat +every HEARTBEAT seconds, taken over after STALE seconds of silence and resumed from the last finished stage), in +stages: + + gather each sub-question answered with the tool loop (chat.tool_answer), its share of the tool calls, and a + toolbox of the plan's sources only, whose outbound tools (web search, a sandbox that reaches the web) + don't ask again: the person approved them with the plan (Toolbox.allowed_out). Other change tools still + ask. + write one call: the report in Markdown, following the outline, citing [n] across every finding, renumbered + check chat.check_sources on the report; unsupported claims are rewritten once, or marked + done the report saved in the conversation as an answer of kind `research`, with its record (chat.Record) + +Every call counts in the ledger for the conversation and `research:`. Before each stage the budgets of the +scope's namespaces and `all` are checked: one over holds the run (on_over ask) until the budget allows or the person +lets it go on, or stops it with what it has (skip). research.max_minutes stops a run with what it found; Stop +(POST /research/{id}/stop) stops it after the step it's on, like stopping an answer. A person has one run going at a +time; the next waits its turn. + +After: save the report as a note (archive citations become mention links, web ones links, and it links back to the +conversation), read it aloud, make a podcast episode from its archive sources, or export the conversation. + +Refine later: sub-questions in parallel, research on a schedule, sharing a report with namespace members, research as +an automation action, a report template people can edit, the report posted back into the chat room it was asked in. +""" + +from __future__ import annotations + +import datetime as dt +import logging +import re +import threading +import time +import uuid + +from . import activity, auth, budgets, chat, llm, notebook, store, telemetry, untrusted + +log = logging.getLogger("lens") +R = store.R + +STATUSES = ("planned", "declined", "queued", "running", "held", "done", "stopped", "failed") +ACTIVE = ("queued", "running", "held") # approved and not finished +STAGES = ("gather", "write", "check", "done") +SOURCES = ("archive", "graph", "notes", "web", "code") +# the tools a run may use for each source: reading only (code that runs in the sandbox reads and works things out) +SOURCE_TOOLS = { + "archive": ["search_transcripts", "find_resources", "find_similar", "list_recordings", "read_transcript", "recording_outputs"], + "graph": [ + "find_entities", + "entity_mentions", + "entity_timeline", + "find_topics", + "topic_recordings", + "graph_neighbours", + "speaker_stats", + "graph_schema", + "graph_query", + "graph_related", + "graph_paths", + ], + "notes": ["find_notes", "read_note"], + "web": ["web_search", "web_read"], + "code": ["run_python", "run_bash", "sandbox_files"], +} +SOURCE_LABELS = {"archive": "the archive", "graph": "the graph", "notes": "notes", "web": "the web", "code": "the code sandbox"} +QUESTIONS_MAX, ANSWERS_MAX, SUBS_MAX, OUTLINE_MAX = 3, 4, 8, 10 +HEARTBEAT = 10 # seconds between a run's heartbeats +STALE = 120 # seconds of silence after which another process takes a run over +SWEEP_SECONDS = 30 # how often the scheduler looks for runs no process is on +LOOKBACK = 20 # past runs an estimate is drawn from + +# the start_research tool (ai_tools.GROUPS "research"): it plans, and the plan waits for the person's approval +TOOLS = [ + ( + "start_research", + "Plan deep research on a goal that needs many look-ups (several sub-questions, more than one source): you get " + "a plan with follow-up questions, which waits for the person's approval; once approved it runs in the " + "background and the cited report comes back in this conversation. Use it when the person asks for research or " + "a report, not for a quick question.", + {"goal": {"type": "string", "description": "what to research, in the person's words"}}, + ["goal"], + True, + ) +] + +PLAN_SCHEMA = { + "type": "object", + "required": ["questions", "plan"], + "properties": { + "questions": { + "type": "array", + "items": { + "type": "object", + "required": ["question", "answers"], + "properties": {"question": {"type": "string"}, "answers": {"type": "array", "items": {"type": "string"}}}, + }, + }, + "plan": { + "type": "object", + "required": ["title", "goal", "sub_questions", "sources", "max_tool_calls", "report_outline"], + "properties": { + "title": {"type": "string"}, + "goal": {"type": "string"}, + "sub_questions": {"type": "array", "items": {"type": "string"}}, + "sources": {"type": "array", "items": {"type": "string", "enum": list(SOURCES)}}, + "max_tool_calls": {"type": "integer"}, + "report_outline": {"type": "array", "items": {"type": "string"}}, + }, + }, + }, +} +PLAN_SYSTEM = ( + "You plan deep research for a person. From their goal, write a plan: a short title, the goal in one sentence, " + "the sub-questions that together answer it (3 to {subs}, each answerable by looking things up), the sources to " + "look in (only from: {sources}), how many tool calls it may make in all (at most {calls}; about 3 to 5 per " + "sub-question) and the report's outline (its section headings). Ask follow-up questions (at most 3, each with 2 " + "to 4 short suggested answers) only where the answer would change the plan; none when the goal is clear. When " + "you're given answers and edits to an earlier plan, follow them." +) +FIND = ( + "Research goal: {goal}\n\nAnswer this part of it: {sub}\n\nLook it up with the tools, then answer in a short " + "paragraph or a few bullets, citing what you found as [n]. Say plainly what you couldn't find." +) +WRITE_SYSTEM = ( + "You write research reports. From the findings below, write the report in Markdown: a title (# heading), a short " + "summary, then the outline's sections (## headings), then what is still open. Cite every claim with the [n] numbers " + "of the excerpts it rests on, and only those; say so where the findings don't cover a section. Don't make up " + "anything the findings don't say." +) +REWRITE_SCHEMA = { + "type": "object", + "required": ["rewrites"], + "properties": { + "rewrites": { + "type": "array", + "items": { + "type": "object", + "required": ["claim", "text"], + "properties": {"claim": {"type": "string"}, "text": {"type": "string"}}, + }, + } + }, +} +REWRITE_SYSTEM = ( + "These sentences of a report aren't supported by the excerpts they cite. Rewrite each so that it says only what " + "its excerpts support, keeping its [n] citations; give an empty text for one the excerpts don't support at all." +) +UNSUPPORTED = " *(not supported by its sources)*" + + +class Stopped(Exception): + """The run stops here: asked to, out of time, or another process took it over.""" + + def __init__(self, why): + super().__init__(why) + self.why = why + + +# ---------- settings ---------- +def settings(cfg): + r = cfg.get("research") or {} + return { + "max_tool_calls": int(r.get("max_tool_calls") or 40), + "max_minutes": int(r.get("max_minutes") or 20), + "model": r.get("model") or None, + "agents_skip_review": bool(r.get("agents_skip_review")), + } + + +def unavailable(cfg): + """Why research can't run, and the setting that would change it, or None: it needs a language model that uses + tools.""" + if not llm.configured(cfg): + return "no language model is set up", "llm.model" + if not cfg["ai"].get("tools"): + return "the assistant's tools are switched off", "ai.tools" + return None + + +def available(cfg): + """Whether research can run: a language model that uses tools.""" + return unavailable(cfg) is None + + +def agents_may_approve(cfg, via): + """Whether a pick made `via` (web, chat, token, oauth or agent) may approve a plan: the person's own, signed in or + in their chat app; an API key, an app or an agent only with research.agents_skip_review.""" + return via in ("web", "chat") or settings(cfg)["agents_skip_review"] + + +# ---------- planning ---------- +def sources_of(box): + """The sources this person's toolbox (as the conversation and their choice of tools have it) can look in.""" + offered = {s["function"]["name"] for s in box.specs()} + return [s for s in SOURCES if offered & set(SOURCE_TOOLS[s])] + + +def _text(v, n): + return " ".join(str(v or "").split())[:n] + + +def clean_plan(cfg, plan, sources, goal=None): + """A plan as a run takes it: its sub-questions, sources (of those this person has), tool calls (1 to + research.max_tool_calls) and outline, trimmed. ValueError for one with no sub-question or no source.""" + plan = plan or {} + subs = [_text(s, 300) for s in plan.get("sub_questions") or [] if _text(s, 300)][:SUBS_MAX] + if not subs: + raise ValueError("a plan has at least one sub-question") + picked = [s for s in dict.fromkeys(plan.get("sources") or []) if s in sources] + if not picked: + if plan.get("sources"): + raise ValueError(f"a plan looks in {', '.join(sources) or 'nothing you have'}") + picked = sources[:1] + if not picked: + raise ValueError("you have no tools research can look things up with") + most = settings(cfg)["max_tool_calls"] + try: + calls = int(plan.get("max_tool_calls") or 0) + except (TypeError, ValueError): + raise ValueError("max_tool_calls is a whole number") from None + calls = max(len(subs), min(most, calls or 4 * len(subs))) + calls = min(calls, most) + goal = _text(plan.get("goal") or goal, 1000) + return { + "title": _text(plan.get("title") or goal, 120) or "Research", + "goal": goal, + "sub_questions": subs, + "sources": picked, + "max_tool_calls": calls, + "report_outline": [_text(s, 200) for s in plan.get("report_outline") or [] if _text(s, 200)][:OUTLINE_MAX], + } + + +def _questions(raw): + out = [] + for q in raw or []: + text = _text((q or {}).get("question"), 300) + if text: + out.append({"question": text, "answers": [_text(a, 120) for a in (q.get("answers") or []) if _text(a, 120)][:ANSWERS_MAX]}) + return out[:QUESTIONS_MAX] + + +def model_for(cfg, chat_model=None): + """The model research uses: research.model, else the conversation's, else the configured one (None).""" + return settings(cfg)["model"] or chat_model or None + + +def plan(cfg, goal, sources, scope=None, model=None, previous=None, answers=None): + """One planning call: {questions, plan}. `previous` (a plan) with `answers` ({question: answer}) revises it.""" + if not llm.configured(cfg): + raise llm.LLMError("research needs a language model: set one up in Settings › AI assistant") + s = settings(cfg) + system = PLAN_SYSTEM.format(subs=SUBS_MAX, sources=", ".join(sources), calls=s["max_tool_calls"]) + untrusted.rules(cfg) + parts = [f"Goal: {goal}"] + if (scope or {}).get("namespaces"): + parts.append("It looks only in these namespaces: " + ", ".join(scope["namespaces"])) + if previous: + lines = "\n".join(f"- {s}" for s in previous.get("sub_questions") or []) + parts.append( + f"The earlier plan, as the person edited it:\nTitle: {previous.get('title')}\nSub-questions:\n{lines}\n" + f"Sources: {', '.join(previous.get('sources') or [])}\nTool calls: {previous.get('max_tool_calls')}\n" + f"Outline: {'; '.join(previous.get('report_outline') or [])}" + ) + if answers: + parts.append("Their answers:\n" + "\n".join(f"- {q}: {a}" for q, a in answers.items() if a)) + out = llm.json_out(cfg, system, "\n\n".join(parts), PLAN_SCHEMA, model=model) + return {"questions": [] if answers else _questions(out.get("questions")), "plan": clean_plan(cfg, out.get("plan"), sources, goal)} + + +def estimate(db, cfg, p, model=None): + """What a plan will likely cost, always labelled an estimate: from the ledger's past research runs (what each tool + call cost them, times this plan's), else its tokens at the model's price in Settings. {usd, tokens, tool_calls, + basis, estimate}.""" + calls = int(p["max_tool_calls"]) + past = db.rows( + "SELECT record::id(id) AS id, used, finished_at FROM research WHERE status = 'done' ORDER BY finished_at DESC LIMIT $n", n=LOOKBACK + ) + usd = tokens = used = 0.0 + runs = 0 + for r in past: + n = (r.get("used") or {}).get("tool_calls") or 0 + if n: + t = activity.totals(db, f"research:{r['id']}") + usd, tokens, used, runs = usd + t["cost_usd"], tokens + t["tokens_in"] + t["tokens_out"], used + n, runs + 1 + if runs: + return { + "usd": round(usd / used * calls, 4), + "tokens": int(tokens / used * calls), + "tool_calls": calls, + "basis": f"what each tool call of {runs} past research run{'s' if runs != 1 else ''} cost", + "estimate": True, + } + subs = len(p["sub_questions"]) + tin, tout = 3000 + calls * 2500 + 8000 + 4000, subs * 500 + 2500 + 800 # planning, gathering, writing, checking + name = model or cfg["llm"].get("model") + cost = telemetry.cost(name, tin, tout, telemetry.prices(cfg)) + return { + "usd": round(cost, 4) if cost is not None else None, + "tokens": tin + tout, + "tool_calls": calls, + "basis": f"about {tin + tout:,} tokens at {name}'s price in Settings" + if cost is not None + else f"about {tin + tout:,} tokens ({name} has no price in Settings)", + "estimate": True, + } + + +def summary(p): + n = len(p["sub_questions"]) + where = ", ".join(SOURCE_LABELS[s] for s in p["sources"]) + return f"Research: {p['title']} ({n} sub-question{'s' if n != 1 else ''}, in {where}, up to {p['max_tool_calls']} tool calls)" + + +def create(db, cfg, box, goal, made, message=None, model=None, rid=None): + """Save a plan ({questions, plan}) as a `research` row of the conversation `box` is in (`rid`: its id, taken + before planning so the planning call counts for it), and the approval it waits for (box._approval). Returns the + row.""" + rid = rid or db.next_id("research") + p = made["plan"] + est = estimate(db, cfg, p, model) + t = store.now() + db.q( + "CREATE $r CONTENT $d", + r=R("research", rid), + d=store.clean( + { + "chat": box.chat, + "message": message, + "account": box.account, + "goal": _text(goal, 2000), + "status": "planned", + "plan": p, + "questions": made["questions"], + "scope": box.scope or None, + "tools": sorted(box.only) if box.only is not None else None, + "model": model, + "estimate": est, + "revisions": 0, + "created_at": t, + "updated_at": t, + } + ), + ) + box._approval("research", {"research": rid, "plan": p}, summary(p), est) + aid = box.approvals[-1]["id"] + db.q("UPDATE $r SET approval = $a", r=R("research", rid), a=aid) + return get(db, rid) + + +def propose(db, cfg, box, goal, model=None): + """The assistant's start_research: a plan for `goal` in the conversation `box` is in, waiting for approval (the + planning call counts for the research too). Returns the row.""" + rid = db.next_id("research") + with activity.scope(db, f"research:{rid}", cfg=cfg): + made = plan(cfg, goal, sources_of(box), box.scope, model) + return create(db, cfg, box, goal, made, model=model, rid=rid) + + +COMMAND = re.compile(r"^/research\s+(.+)$", re.S | re.I) + + +def command(text): + """The goal of `/research `, else None.""" + m = COMMAND.match((text or "").strip()) + return " ".join(m.group(1).split()) if m else None + + +def _money(usd): + return f"${usd:,.4f}".rstrip("0").rstrip(".") if usd < 1 else f"${usd:,.2f}" + + +def plan_text(r): + """The plan as the conversation shows it, for people reading it in a chat app or without the plan card.""" + p, est = r["plan"], r.get("estimate") or {} + where = ", ".join(SOURCE_LABELS[s] for s in p["sources"]) + cost = f", about {_money(est['usd'])} (an estimate)" if est.get("usd") is not None else "" + lines = [f"Here's a plan to research **{p['title']}**:", ""] + lines += [f"{i}. {q}" for i, q in enumerate(p["sub_questions"], 1)] + lines += ["", f"It looks in {where}, with up to {p['max_tool_calls']} tool calls{cost}."] + if p.get("report_outline"): + lines.append("The report: " + "; ".join(p["report_outline"]) + ".") + if r.get("questions"): + lines += ["", "A few questions, so it fits what you need:"] + lines += [f"- {q['question']}" + (f" ({' / '.join(q['answers'])})" if q.get("answers") else "") for q in r["questions"]] + lines += ["", "Approve it to start, or answer the questions or edit it and revise it."] + return "\n".join(lines) + + +def start(db, cfg, box, goal, via="web", said=None, parent=chat.HEAD, model=None, max_tool_calls=None, options=None, review=True): + """A research goal asked in the conversation `box` is in: the question saved (`said`: as typed, e.g. with + /research), one planning call, the plan saved and its approval, and the plan as the answer (kind research_plan). + With review False the person's own pick (via web or chat; an agent's only with research.agents_skip_review) + approves it at once. Returns {research, approval, question, message, text, meta}, with `error` when the plan + couldn't be made (saved as the answer's error).""" + cid = box.chat + question = chat.add(db, cid, "user", said or goal, parent=parent, meta=chat.question_record(via, research=True, **(options or {}))) + rid, mid = db.next_id("research"), db.next_id("chat_message") + record = chat.Record(cfg, model, box.scope, picked=sorted(box.only) if box.only is not None else None, research=rid) + wrote = (model or cfg["llm"].get("model")) if llm.configured(cfg) else None + try: + with activity.scope(db, f"chat:{cid}", f"chat_message:{mid}", f"research:{rid}", cfg=cfg): + made = plan(cfg, goal, sources_of(box), box.scope, model) + if max_tool_calls: + made["plan"]["max_tool_calls"] = max(1, min(int(max_tool_calls), settings(cfg)["max_tool_calls"])) + except (llm.LLMError, ValueError) as e: + meta = record.meta(db, "error", f"chat_message:{mid}") + chat.add(db, cid, "assistant", "(no plan)", [], error=str(e), model=wrote, parent=question, mid=mid, meta=meta) + return {"error": str(e), "question": question, "message": mid, "meta": meta} + r = create(db, cfg, box, goal, made, message=question, model=model, rid=rid) + text = plan_text(r) + meta = record.meta(db, "answered", f"chat_message:{mid}") + chat.add(db, cid, "assistant", text, [], model=wrote, parent=question, mid=mid, meta=meta, kind="research_plan") + db.q("UPDATE $r SET plan_message = $m", r=R("research", rid), m=mid) + approval = box.approvals[-1] + if not review: + a = db.one("SELECT *, record::id(id) AS id FROM $r", r=R("approval", approval["id"])) + decide(db, cfg, a, box.user, "approve", None, via if via in ("web", "chat") else "agent") + return {"research": get(db, rid), "approval": approval, "question": question, "message": mid, "text": text, "meta": meta} + + +def get(db, rid): + r = db.one("SELECT *, record::id(id) AS id FROM $r", r=R("research", int(rid))) + if not r: + raise KeyError(rid) + return r + + +def mine(db, rid, account): + """A research row of this person's, or KeyError.""" + r = get(db, rid) + if r.get("account") != account: + raise KeyError(rid) + return r + + +def view(db, r): + """A research row as the API shows it, with what it has spent so far (the ledger rows that name it).""" + spent = activity.totals(db, f"research:{r['id']}") + findings = r.get("findings") or [] + p = r.get("plan") or {} + sources = {(x.get("url") or "", x.get("recording_id"), x.get("t0"), x.get("page")) for f in findings for x in f.get("passages") or []} + return { + "id": r["id"], + "chat": r.get("chat"), + "message": r.get("message"), + "plan_message": r.get("plan_message"), + "answer": r.get("answer"), + "approval": r.get("approval"), + "status": r.get("status"), + "stage": r.get("stage"), + "goal": r.get("goal"), + "plan": p, + "questions": r.get("questions") or [], + "answers": r.get("answers") or {}, + "scope": r.get("scope") or {}, + "estimate": r.get("estimate"), + "progress": { + "sub_questions": len(p.get("sub_questions") or []), + "done": len(findings), + "sources": len(sources) + int(r.get("sources_now") or 0), + "tool_calls": int((r.get("used") or {}).get("tool_calls") or 0), + "cost_usd": spent["cost_usd"], + "tokens": spent["tokens_in"] + spent["tokens_out"], + "cost_estimated": spent["estimate"], + "minutes": round(float(r.get("ran_s") or 0) / 60, 1), + }, + "findings": [{"question": f["question"], "tool_calls": f.get("tool_calls") or 0} for f in findings], + "held": r.get("held"), + "error": r.get("error"), + "ended": r.get("ended"), + "revisions": r.get("revisions") or 0, + "notes": r.get("notes") or [], + "created_at": r.get("created_at"), + "started_at": r.get("started_at"), + "finished_at": r.get("finished_at"), + } + + +def revise(db, cfg, box, r, answers=None, edited=None): + """One more planning call with the person's answers and edits; the plan and its approval change in place.""" + if r["status"] != "planned": + raise ValueError(f"this research is {r['status']}: only a plan waiting for approval can be revised") + sources = sources_of(box) + previous = clean_plan(cfg, {**(r.get("plan") or {}), **(edited or {})}, sources, r.get("goal")) + answers = {_text(k, 300): _text(v, 300) for k, v in (answers or {}).items() if _text(k, 300)} + with activity.scope(db, f"chat:{r['chat']}", f"research:{r['id']}", cfg=cfg): + made = plan(cfg, r.get("goal") or previous["goal"], sources, r.get("scope"), r.get("model"), previous, answers) + p = made["plan"] + est = estimate(db, cfg, p, r.get("model")) + db.q( + "UPDATE $r SET plan = $p, questions = $q, answers = $a, estimate = $e, revisions += 1, updated_at = $t", + r=R("research", r["id"]), + p=p, + q=made["questions"], + a=answers or None, + e=est, + t=store.now(), + ) + db.q( + "UPDATE $r SET args = $args, summary = $s, estimate = $e WHERE status = 'pending'", + r=R("approval", r["approval"]), + args={"research": r["id"], "plan": p}, + s=summary(p), + e=est, + ) + return get(db, r["id"]) + + +# ---------- deciding ---------- +def decide(db, cfg, a, user, decision, params=None, via="web"): + """The person's pick on a plan's approval: approve (as it is, or with `params`: the plan edited; `edit` is the + same) or decline. An approved plan queues its run, which starts at once unless the person has one going.""" + decision = "approve" if decision == "edit" else decision + if decision not in ("approve", "decline"): + raise ValueError("a research plan is approved or declined") + r = get(db, a["args"]["research"]) + if decision == "approve" and not agents_may_approve(cfg, via): + raise PermissionError("approving a research plan is the person's own pick: an agent can only decline (research.agents_skip_review)") + p = r["plan"] + if decision == "approve" and params: # the plan as the person edited it, within the sources they have + box = toolbox(db, cfg, {**r, "plan": {**p, "sources": list(SOURCES)}}, account_of(db, r["account"])) + allowed = sources_of(box) if box else p["sources"] + p = clean_plan(cfg, {**p, **{k: v for k, v in params.items() if v is not None}}, allowed, r.get("goal")) + if not approvals_take(db, a["id"], "done" if decision == "approve" else "declined", user): + raise ValueError("nothing to approve") + if decision == "decline": + db.q("UPDATE $r SET status = 'declined', updated_at = $t", r=R("research", r["id"]), t=store.now()) + return {"status": "declined", "result": {"research": r["id"], "status": "declined"}} + queue(db, cfg, r["id"], p, user) + result = {"research": r["id"], "status": "queued"} + db.q("UPDATE $r SET result = $res", r=R("approval", a["id"]), res=result) + return {"status": "done", "result": result} + + +def approvals_take(db, aid, status, user): + """Claim the plan's approval for one decision: True when this one won (approvals._take).""" + from . import approvals + + return approvals._take(db, aid, status, user) + + +def queue(db, cfg, rid, p, user=None): + """An approved plan, waiting for its run: started now, unless the person already has one going.""" + db.q( + "UPDATE $r SET status = 'queued', plan = $p, stage = 'gather', approved_by = $u, approved_at = $t, updated_at = $t", + r=R("research", rid), + p=p, + u=(user or {}).get("email"), + t=store.now(), + ) + r = get(db, rid) + if not _busy(db, r["account"], rid): + launch(db, cfg, rid) + + +def _busy(db, account, rid=None): + """Whether the person has another run going (one at a time each).""" + return bool( + db.rows( + "SELECT id FROM research WHERE account = $a AND status IN ['running', 'held'] AND id != $r LIMIT 1", + a=account, + r=R("research", int(rid or 0)), + ) + ) + + +def launch(db, cfg, rid): + """Run it in the background, in this process, if it claims it.""" + threading.Thread(target=run, args=(db, cfg, rid), daemon=True, name=f"research {rid}").start() + + +def request_stop(db, cfg, rid): + """Ask a run to stop after the step it's on; a run that hasn't started (or waits on a budget) stops now. False when + it's already over.""" + r = get(db, rid) + if r["status"] in ("queued", "held"): + took = db.rows( + "UPDATE $r SET status = 'stopped', ended = 'stopped', finished_at = $t, updated_at = $t WHERE status IN ['queued', 'held'] RETURN id", + r=R("research", rid), + t=store.now(), + ) + if took and (r.get("findings") or []): + # what it found before it was held stays: saved as a report of its findings, without another model call + _save_found(db, cfg, get(db, rid)) + return bool(took) + if r["status"] == "planned": + return False + return bool(db.rows("UPDATE $r SET stop_requested = true WHERE status = 'running' RETURN id", r=R("research", rid))) + + +def go_on(db, cfg, rid, user=None): + """Let a run held by a budget go on this once (the person's pick, as budgets.py's "run it once").""" + took = db.rows( + "UPDATE $r SET status = 'queued', over_ok = true, held = NONE, updated_at = $t WHERE status = 'held' RETURN id", + r=R("research", rid), + t=store.now(), + ) + if not took: + raise ValueError("only a run held by a budget can go on") + launch(db, cfg, rid) + return get(db, rid) + + +# ---------- claiming a run ---------- +def _ago(seconds): + return (dt.datetime.now(dt.timezone.utc) - dt.timedelta(seconds=seconds)).isoformat(timespec="seconds") + + +def claim(db, rid, owner): + """Take a run for this process (CREATE-as-claim on research_run:): True when it's ours. A claim whose + heartbeat stopped STALE seconds ago is another process gone, so it's taken over.""" + rec = R("research_run", int(rid)) + for _ in range(2): + try: + db.q( + "CREATE $r CONTENT $d", + r=rec, + d={"research": int(rid), "owner": owner, "heartbeat_at": store.now(), "claimed_at": store.now()}, + ) + return True + except Exception as e: # noqa: BLE001 - only a claim that's already there means someone else has it + if "already exists" not in str(e): + raise + if not db.rows("DELETE $r WHERE heartbeat_at < $s RETURN BEFORE", r=rec, s=_ago(STALE)): + return False + return False + + +def beat(db, rid, owner): + """The run's heartbeat: False when the claim isn't ours any more (another process took it over).""" + return bool( + db.rows("UPDATE $r SET heartbeat_at = $t WHERE owner = $o RETURN id", r=R("research_run", int(rid)), t=store.now(), o=owner) + ) + + +def release(db, rid, owner): + db.q("DELETE $r WHERE owner = $o", r=R("research_run", int(rid)), o=owner) + + +def sweep(db, cfg, log=None): + """Runs no process is on: an approved one waiting its turn, one whose process went away (its claim gone stale) and + one a budget held, once the budget allows it. Each is started here, in the background.""" + rows = db.rows("SELECT record::id(id) AS id, account, status, created_at FROM research WHERE status IN $s ORDER BY id", s=list(ACTIVE)) + claimed = { + c["research"]: c + for c in db.rows("SELECT research, heartbeat_at FROM research_run WHERE research IN $ids", ids=[r["id"] for r in rows]) + } + going = {r["account"] for r in rows if r["status"] == "running" and r["id"] in claimed} + started = [] + for r in rows: + c = claimed.get(r["id"]) + if c and (c.get("heartbeat_at") or "") >= _ago(STALE): + continue # a process is on it + if r["status"] == "held": + full = get(db, r["id"]) + if _over(db, cfg, full): + continue + db.q("UPDATE $r SET status = 'queued', held = NONE WHERE status = 'held'", r=R("research", r["id"])) + elif r["account"] in going and r["status"] == "queued": + continue # their other run first + going.add(r["account"]) + started.append(r["id"]) + launch(db, cfg, r["id"]) + if started and log: + log(f"research: started {', '.join(map(str, started))}") + return started + + +# ---------- running ---------- +def account_of(db, aid): + a = auth.get_account(db, aid) + if not a or a.get("disabled") or a.get("removed_at"): + return None + return {"id": a["id"], "email": a["email"], "admin": bool(a.get("admin"))} + + +def toolbox(db, cfg, r, account, sub=None): + """The person's tools for the run: only the plan's sources' (and of those, what the conversation's choice of tools + keeps), and its outbound tools allowed without asking again (the plan approved them).""" + from . import ai_tools, sandbox + + if not account: + return None + roles = auth.roles(db, account) + readable, editable = set(roles), {s for s in roles if auth.allows(roles, s, "editor")} + user = {"id": account["id"], "email": account["email"]} + p = r.get("plan") or {} + box = ai_tools.Toolbox( + db, + cfg, + user, + readable, + editable, + r.get("scope"), + r.get("chat"), + None, + account["admin"], + said=f"{r.get('goal') or ''} {sub or ''}", + only=r.get("tools"), + ) + keep = box.chosen() + names = {t for s in p.get("sources") or [] for t in SOURCE_TOOLS.get(s, [])} + box.only = names if keep is None else names & keep + out = set() + if "web" in (p.get("sources") or []): + out |= set(SOURCE_TOOLS["web"]) + if "code" in (p.get("sources") or []) and sandbox.sends_out(cfg): + out |= {"run_python", "run_bash"} + box.allowed_out = out + return box + + +def _over(db, cfg, r): + """Why the run must wait for a budget, or None: the budgets of the scope's namespaces and `all` (budgets.check).""" + if r.get("over_ok"): + return None + names = {v: k for k, v in store.space_names(db).items()} + spaces = [f"space:{names[n]}" for n in (r.get("scope") or {}).get("namespaces") or [] if n in names] + got = budgets.check(db, cfg, spaces) + return None if got["go"] else got + + +class Run: + """One process's turn at a run: its claim, kept by a heartbeat every HEARTBEAT seconds (from a thread of its own, + so a long model call doesn't let the claim go stale), the time it has run, and what stops it.""" + + def __init__(self, db, cfg, rid, owner): + self.db, self.cfg, self.rid, self.owner = db, cfg, rid, owner + self.r = get(db, rid) + self.t0, self.ran = time.monotonic(), float(self.r.get("ran_s") or 0) + self.lost, self.done = False, threading.Event() + self.limit = settings(cfg)["max_minutes"] * 60 + threading.Thread(target=self._beat, daemon=True, name=f"research {rid} heartbeat").start() + + def _beat(self): + while not self.done.wait(HEARTBEAT): + try: + if not beat(self.db, self.rid, self.owner): + self.lost = True + return + self.db.q("UPDATE $r SET ran_s = $s", r=R("research", self.rid), s=round(self.elapsed(), 1)) + except Exception: # noqa: BLE001 - the next beat tries again; a claim that goes stale is taken over + log.warning("research %s: no heartbeat", self.rid, exc_info=True) + + def end(self): + self.done.set() + try: + self.db.q("UPDATE $r SET ran_s = $s", r=R("research", self.rid), s=round(self.elapsed(), 1)) + except Exception: # noqa: BLE001 - only the time it ran is lost + log.warning("research %s: couldn't save the time it ran", self.rid, exc_info=True) + + def elapsed(self): + return self.ran + time.monotonic() - self.t0 + + def tick(self): + """Between steps: whether another process took the run over, or the person asked it to stop.""" + if self.lost: + raise Stopped("lost") + row = self.db.one("SELECT stop_requested FROM $r", r=R("research", self.rid)) or {} + if row.get("stop_requested"): + raise Stopped("stopped") + + def out_of_time(self): + return self.elapsed() >= self.limit + + +def run(db, cfg, rid, owner=None): + """Claim the run and carry it out, from its last finished stage, until it's done, held, stopped or failed. False + when another process has it (or it isn't waiting for one).""" + owner = owner or uuid.uuid4().hex + r = get(db, rid) + if r["status"] not in ("queued", "running") or not claim(db, rid, owner): + return False + try: + r = get(db, rid) # as it is now we hold it + if r["status"] not in ("queued", "running"): + return False + if r["status"] == "queued" and _busy(db, r["account"], rid): + return False # their other run first; the sweep starts this one after it + db.q( + "UPDATE $r SET status = 'running', started_at = started_at ?? $t, updated_at = $t", + r=R("research", rid), + t=store.now(), + ) + turn = Run(db, cfg, rid, owner) + try: + with activity.scope(db, f"chat:{r['chat']}", f"research:{rid}", cfg=cfg): + _stages(db, cfg, turn) + finally: + turn.end() + return True + except Exception as e: # noqa: BLE001 - a run that breaks says why, keeps what it found, and frees the person's turn + log.exception("research %s failed", rid) + why = str(e) if isinstance(e, llm.LLMError) else f"{type(e).__name__}: {e}" + took = db.rows( + "UPDATE $r SET status = 'failed', error = $e, finished_at = $t, updated_at = $t WHERE status = 'running' RETURN id", + r=R("research", rid), + e=why[:500], + t=store.now(), + ) + if took: + try: + _save_found(db, cfg, get(db, rid), "error") + except Exception: # noqa: BLE001 - the run's row keeps its findings and the error + log.exception("research %s: couldn't save what it found", rid) + return True + finally: + release(db, rid, owner) + _next(db, cfg, rid) + + +def _next(db, cfg, rid): + """The person's next approved run, once this one is over.""" + r = db.one("SELECT account, status FROM $r", r=R("research", rid)) or {} + if r.get("status") in ("running", "held", "queued") or not r.get("account"): + return + nxt = db.one("SELECT record::id(id) AS id FROM research WHERE account = $a AND status = 'queued' ORDER BY id LIMIT 1", a=r["account"]) + if nxt: + launch(db, cfg, nxt["id"]) + + +def _hold(db, rid, got): + """A budget is over: the run waits (on_over ask), saying why, or stops with what it has (skip).""" + if got["action"] == "skip": + raise Stopped("budget") + db.q( + "UPDATE $r SET status = 'held', held = $h, updated_at = $t", + r=R("research", rid), + h={"why": got["why"], "budget": got["resource"]}, + t=store.now(), + ) + + +def _stages(db, cfg, run): + rid = run.rid + account = account_of(db, run.r["account"]) + if not account: + raise RuntimeError("the person who asked for it can't sign in any more") + try: + while True: + r = get(db, rid) + stage = r.get("stage") or "gather" + if stage == "done": + return + got = _over(db, cfg, r) + if got: + _hold(db, rid, got) + return + if stage == "gather": + _gather(db, cfg, run, r, account) + db.q("UPDATE $r SET stage = 'write'", r=R("research", rid)) + elif stage == "write": + _write(db, cfg, run, r) + db.q("UPDATE $r SET stage = 'check'", r=R("research", rid)) + elif stage == "check": + _check(db, cfg, run, r) + db.q("UPDATE $r SET stage = 'done'", r=R("research", rid)) + _finish(db, cfg, run, get(db, rid), "timeout" if r.get("ended") == "timeout" else None) + return + except activity.OverBudget as e: # the overall budget's hard limit, reached mid-stage: it waits, as before a stage + _hold(db, rid, {"action": "ask", "why": str(e), "resource": budgets.ALL}) + except Stopped as s: + if s.why == "lost": # another process has it now, and goes on from the last finished stage + return + db.q( + "UPDATE $r SET status = 'stopped', ended = $w, stop_requested = false, finished_at = $t, updated_at = $t", + r=R("research", rid), + w=s.why, + t=store.now(), + ) + _save_found(db, cfg, get(db, rid)) + + +def _model(db, cfg, r): + """The model the run uses: research.model, else the conversation's (while it's still offered), else the configured one.""" + if settings(cfg)["model"]: + return settings(cfg)["model"] + m = r.get("model") or (db.one("SELECT model FROM $r", r=R("chat", r["chat"])) or {}).get("model") + return m if m and m in chat.model_choices(cfg) else None + + +def _gather(db, cfg, run, r, account): + """Each sub-question not yet answered, with its share of the tool calls that are left.""" + p = r["plan"] + findings = list(r.get("findings") or []) + used = int((r.get("used") or {}).get("tool_calls") or 0) + model = _model(db, cfg, r) + most = int(cfg["ai"].get("max_tool_calls") or 50) + for i, sub in enumerate(p["sub_questions"]): + if i < len(findings): + continue + if run.out_of_time(): + db.q("UPDATE $r SET ended = 'timeout'", r=R("research", run.rid)) + return + run.tick() + left = max(0, p["max_tool_calls"] - used) + if left <= 0: + break + share = max(1, min(most, left // (len(p["sub_questions"]) - i))) + box = toolbox(db, cfg, r, account, sub) + q = FIND.format(goal=p["goal"], sub=sub) + record = chat.Record(cfg, model, r.get("scope"), picked=p["sources"], tool_calls=share, research=run.rid) + text, steps = "", [] + try: + for kind, data in chat.tool_answer(cfg, box, q, (), share, model, record=record, max_tool_calls=share): + if kind == "step": + steps.append(data) + db.q( + "UPDATE $r SET used = $u, sources_now = $s", + r=R("research", run.rid), + u={"tool_calls": used + record.used["tool_calls"]}, + s=len(box.refs), + ) + run.tick() + if run.out_of_time(): + break + else: + text = data + cited = box.cited(text) + except llm.ToolsUnsupported: # a model that can't call tools: answered from a search of the archive + passages = chat.retrieve(db, sub, box.readable, r.get("scope"), cfg=cfg) + text = llm.chat(cfg, chat.messages_for(q, passages, cfg=cfg), model=model) + cited = [x for x in chat.cited(text, passages) if x["used"]] + used += record.used["tool_calls"] + findings.append( + {"question": sub, "text": text or "(nothing found)", "passages": cited, "tool_calls": record.used["tool_calls"], "steps": steps} + ) + db.q( + "UPDATE $r SET findings = $f, used = $u, sources_now = 0, updated_at = $t", + r=R("research", run.rid), + f=findings, + u={"tool_calls": used}, + t=store.now(), + ) + if run.out_of_time(): + db.q("UPDATE $r SET ended = 'timeout'", r=R("research", run.rid)) + return + + +def _key(p): + return (p.get("url") or "", p.get("recording_id"), p.get("t0"), p.get("page")) + + +def renumber(findings): + """Every finding's citations as one numbered list: (the passages, each finding's text cited by their new numbers). + The same moment or page cited in two findings keeps one number.""" + out, at, texts = [], {}, [] + for f in findings: + local = {} + for x in f.get("passages") or []: + k = _key(x) + if k not in at: + at[k] = len(out) + 1 + out.append({**{k2: v for k2, v in x.items() if k2 != "used"}, "n": at[k]}) + local[x["n"]] = at[k] + texts.append(re.sub(r"\[(\d+)\]", lambda m: f"[{local[int(m.group(1))]}]" if int(m.group(1)) in local else "", f.get("text") or "")) + return out, texts + + +def _write(db, cfg, run, r): + """The report: one call, from every finding, following the outline.""" + p = r["plan"] + passages, texts = renumber(r.get("findings") or []) + run.tick() + if not passages and not any(t.strip() for t in texts): + report = f"# {p['title']}\n\nNothing turned up for this goal in {', '.join(SOURCE_LABELS[s] for s in p['sources'])}." + else: + found = "\n\n".join( + f"## {f['question']}\n" + untrusted.wrap(t, "research finding", cfg) for f, t in zip(r.get("findings") or [], texts) + ) + outline = "\n".join(f"- {s}" for s in p.get("report_outline") or []) or "- (your choice)" + user = f"Goal: {p['goal']}\nTitle: {p['title']}\nOutline:\n{outline}\n\nFindings:\n\n{found}\n\nExcerpts:\n\n" + ( + chat.excerpts(passages, cfg) or "(none)" + ) + if r.get("ended") == "timeout": + user += "\n\nThe research ran out of time before every sub-question was looked into: say which weren't." + report = llm.chat( + cfg, + [{"role": "system", "content": WRITE_SYSTEM + untrusted.rules(cfg)}, {"role": "user", "content": user}], + model=_model(db, cfg, r), + ) + db.q("UPDATE $r SET report = $t, passages = $p, updated_at = $n", r=R("research", run.rid), t=report, p=passages, n=store.now()) + + +def _check(db, cfg, run, r): + """Each cited claim of the report against its excerpts; an unsupported one is rewritten once, or marked.""" + report, passages = r.get("report") or "", r.get("passages") or [] + run.tick() + if not passages: + db.q("UPDATE $r SET check = $c", r=R("research", run.rid), c={"claims": 0, "supported": 0, "verdicts": [], "uncited": []}) + return + got = chat.check_sources(cfg, report, passages) + bad = [v["claim"] for v in got["verdicts"] if not v.get("supported") and v.get("claim") in report] + if bad: + run.tick() + try: + fixes = llm.json_out( + cfg, + REWRITE_SYSTEM + untrusted.rules(cfg), + f"Excerpts:\n\n{chat.excerpts(passages, cfg)}\n\nSentences:\n" + "\n".join(f"- {c}" for c in bad), + REWRITE_SCHEMA, + model=_model(db, cfg, r), + )["rewrites"] + except llm.LLMError: + fixes = [] + by = {f.get("claim"): (f.get("text") or "").strip() for f in fixes if isinstance(f, dict)} + for c in bad: + new = by.get(c) + report = report.replace(c, new if new and new != c else c + UNSUPPORTED, 1) + db.q("UPDATE $r SET report = $t, check = $c", r=R("research", run.rid), t=report, c={**got, "rewritten": len(bad)}) + + +def _record(db, cfg, r, outcome): + """The report's record (chat.Record): the model, the plan's sources and tool calls, what it used, how it ended, + and the tokens and cost of every call the run made (research:).""" + model = _model(db, cfg, r) + rec = chat.Record( + cfg, + model, + r.get("scope"), + picked=(r.get("plan") or {}).get("sources"), + tool_calls=(r.get("plan") or {}).get("max_tool_calls"), + research=r["id"], + ) + rec.offered = sorted({t for s in (r.get("plan") or {}).get("sources") or [] for t in SOURCE_TOOLS.get(s, [])}) + rec.used = {"rounds": len(r.get("findings") or []), "tool_calls": int((r.get("used") or {}).get("tool_calls") or 0)} + meta = rec.meta(db, outcome, f"research:{r['id']}") + meta["ms"] = int(float(r.get("ran_s") or 0) * 1000) + return meta + + +def _steps(r): + return [s for f in r.get("findings") or [] for s in f.get("steps") or []] + + +ANSWERING_WAIT = 120 # seconds a report waits for an answer being written in the conversation to end + + +def _where(db, cid): + """Where the report goes in the conversation: after its last message, once an answer being written there ends (a + question waiting for its answer isn't followed by the report).""" + end = time.monotonic() + ANSWERING_WAIT + while time.monotonic() < end and (db.one("SELECT answering FROM $r", r=R("chat", cid)) or {}).get("answering"): + time.sleep(1) + return chat.head(db, cid) + + +def _answer(db, cfg, r, text, passages, outcome, stopped=False): + """The report saved in the conversation, where it is now, as an answer of kind research with its record.""" + parent = _where(db, r["chat"]) + mid = db.next_id("chat_message") + meta = _record(db, cfg, r, outcome) + wrote = _model(db, cfg, r) or (cfg["llm"].get("model") if llm.configured(cfg) else None) + chat.add( + db, + r["chat"], + "assistant", + text, + chat.cited(text, passages), + stopped=stopped, + steps=_steps(r), + model=wrote, + parent=parent, + mid=mid, + meta=meta, + kind="research", + ) + db.q("UPDATE $r SET answer = $m, updated_at = $t", r=R("research", r["id"]), m=mid, t=store.now()) + return mid + + +def _finish(db, cfg, run, r, ended): + _answer(db, cfg, r, r.get("report") or "(no report)", r.get("passages") or [], "limit" if ended else "answered") + db.q("UPDATE $r SET status = 'done', finished_at = $t, updated_at = $t", r=R("research", run.rid), t=store.now()) + + +def _save_found(db, cfg, r, outcome="stopped"): + """A run that stopped (or broke off: `outcome` error) keeps what it found: its report when it got that far, else its + findings, saved as the answer without another model call.""" + if r.get("answer"): + return + passages, texts = renumber(r.get("findings") or []) + said = "Stopped" if outcome == "stopped" else f"It broke off ({r.get('error') or 'an error'})" + if r.get("report"): + text, passages = r["report"], r.get("passages") or passages + elif any(t.strip() for t in texts): + text = f"# {r['plan']['title']}\n\n{said}. What it found so far:\n\n" + "\n\n".join( + f"## {f['question']}\n{t}" for f, t in zip(r.get("findings") or [], texts) + ) + else: + text = f"# {r['plan']['title']}\n\n{said} before it found anything." + _answer(db, cfg, r, text, passages, outcome, stopped=outcome == "stopped") + + +# ---------- after: a note, a podcast ---------- +def mention(p): + """A citation as a note links it: an asset's moment as a mention, a web page as a link.""" + if p.get("source") == "web" and p.get("url"): + return f"[{_label(p.get('title') or p.get('site') or p['url'])}]({p['url']})" + if p.get("recording_id") is not None: + return f"@[{_label(p.get('title') or 'Asset')}](recording:{p['recording_id']})" + return "" + + +def _label(text): + return re.sub(r"[\[\]\n]", " ", str(text or "")).strip()[:200] or "source" + + +def note_body(r, report, passages, title): + """The report as a note: each [n] becomes the link to what it cites, and the note links back to the conversation.""" + by = {p["n"]: p for p in passages} + + def cite(m): + p = by.get(int(m.group(1))) + return f" ({mention(p)})" if p and mention(p) else "" + + body = re.sub(r"\s*\[(\d+)\]", cite, report or "") + return body.rstrip() + f"\n\n@[Research: {_label(title)}](chat:{r['chat']})\n" + + +def note_spaces(db, r, editable): + """The namespaces the note can go in: the scope's (or, when it's over everything, those of the archive it cites), + of those the person can edit.""" + names = store.space_names(db) + scoped = (r.get("scope") or {}).get("namespaces") or [] + if scoped: + ids = [sid for sid, n in names.items() if n in scoped] + else: + recs = [p["recording_id"] for p in r.get("passages") or [] if p.get("recording_id") is not None] + ids = sorted( + {x["space"] for x in db.rows("SELECT space FROM recording WHERE id IN $ids", ids=[R("recording", i) for i in recs])} + if recs + else set() + ) + if not ids: + ids = sorted(editable) + return [sid for sid in ids if sid in editable] + + +def to_note(db, r, account, editable, namespace=None): + """Save the report as a notebook page; its id and namespace. ValueError when it has none yet, or the namespace + must be picked (several are possible) or can't be written.""" + if not r.get("answer") and not r.get("report"): + raise ValueError("the report isn't ready yet") + report = _report_text(db, r) + choices = note_spaces(db, r, editable) + names = store.space_names(db) + if namespace: + sid = next((s for s, n in names.items() if n == namespace), None) + if sid is None or sid not in editable: + raise PermissionError(f"you can't write notes in {namespace}") + elif len(choices) == 1: + sid = choices[0] + elif not choices: + raise PermissionError("you can't write notes in any namespace this research looked in") + else: + raise ValueError("pick a namespace: " + ", ".join(names[s] for s in choices)) + title = (r.get("plan") or {}).get("title") or "Research" + pid = notebook.create(db, sid, account, title, note_body(r, report, r.get("passages") or [], title), author="assistant") + db.q("UPDATE $r SET notes = array::union(notes ?? [], [$n])", r=R("research", r["id"]), n=pid) + return {"page": pid, "namespace": names[sid]} + + +def _report_text(db, r): + """The report as it was saved in the conversation (the person may have stopped the run).""" + if r.get("answer"): + m = db.one("SELECT content FROM $r", r=R("chat_message", r["answer"])) or {} + if m.get("content"): + return m["content"] + return r.get("report") or "" + + +def podcast_sources(db, r, readable): + """The assets the report cites, that the person can still read: an episode's selection.""" + used = {p["n"] for p in chat.cited(_report_text(db, r), r.get("passages") or []) if p["used"]} + ids = list( + dict.fromkeys( + p["recording_id"] for p in r.get("passages") or [] if p.get("recording_id") is not None and (p["n"] in used or not used) + ) + ) + if not ids: + return [] + ok = { + x["id"] + for x in db.rows("SELECT record::id(id) AS id, space FROM recording WHERE id IN $ids", ids=[R("recording", i) for i in ids]) + if readable is None or x["space"] in readable + } + return [i for i in ids if i in ok] diff --git a/fastapi_backend/app/domain/routines.py b/fastapi_backend/app/domain/routines.py index 7f1291dea..c79253fb3 100644 --- a/fastapi_backend/app/domain/routines.py +++ b/fastapi_backend/app/domain/routines.py @@ -675,7 +675,8 @@ def start(db, cfg_fn, stop, log=None): """The scheduling threads, until `stop` is set: watched folders scanned when due (every sources.check_seconds), routines run when due (every CHECK_SECONDS), the scheduled backup taken when due, and automations' timers fired when due (timers.py). The API runs them with its background work; `lens worker` runs them too, since Docker and - the packages run the API without it.""" + the packages run the API without it. Research runs no process is on (research.py) are picked up here too.""" + from . import research # it loads the chat, which loads what routines run def every(seconds, fn, what): def loop(): @@ -696,6 +697,7 @@ def loop(): every(lambda: cfg_fn()["fedora"]["sync_seconds"], fedora.sync_due, "fedora"), every(lambda: notebook.QUIET_SECONDS / 2, notebook.organise_due, "notes"), every(lambda: CHECK_SECONDS, backups.run_due, "backups"), + every(lambda: research.SWEEP_SECONDS, research.sweep, "research"), timers.start(db, cfg_fn, stop, log=log), ] diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index 46550c592..a66f1ccd1 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -37,6 +37,7 @@ "ai": None, "sandbox": None, "web": None, + "research": None, "decisions": None, "components": None, "voice": None, @@ -512,6 +513,8 @@ def _check(section, key, value, default): return _sandbox_setting(key, value) if section == "web" and key != "search": return _web_setting(key, value) + if section == "research": + return _research_setting(key, value) if section == "components": return _component_setting(key, value) if section == "nsfw": @@ -1428,6 +1431,24 @@ def _web_setting(key, value): return value +RESEARCH_RANGES = {"max_tool_calls": (1, 500), "max_minutes": (1, 240)} + + +def _research_setting(key, value): + if key == "agents_skip_review": + if not isinstance(value, bool): + raise ValueError("research.agents_skip_review is true or false") + return value + if key == "model": + if value is not None and not (isinstance(value, str) and len(value.strip()) <= 200): + raise ValueError("research.model is a model's name") + return (value or "").strip() or None + lo, hi = RESEARCH_RANGES[key] + if not (isinstance(value, int) and not isinstance(value, bool) and lo <= value <= hi): + raise ValueError(f"research.{key} is a whole number from {lo} to {hi}") + return value + + def _share_setting(key, value): if key in ("enabled", "s3", "webdav", "add_files", "remote"): if not isinstance(value, bool): diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 586dfa744..d63a13b8d 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -322,6 +322,10 @@ def labels(): # searxng_url; brave and tavily need api_key. results: per search; read_chars: of a page's text the model reads; # render: a page that comes back nearly empty is loaded with Chromium (scripts run), when Chromium is installed "web": {"search": "off", "searxng_url": None, "api_key": None, "results": 5, "read_chars": 20000, "render": False}, + # deep research (research.py, docs/assistant.md#deep-research): the most tool calls a plan may make, the minutes + # a run may take before it writes its report with what it found, a model for it (null: the conversation's), and + # whether an agent (an API key, an app, MCP) may start a run without the person reviewing its plan + "research": {"max_tool_calls": 40, "max_minutes": 20, "model": None, "agents_skip_review": False}, # talking to Lens (voice.py): input auto uses the server's speech-to-text engine when it has one, else the # browser's; spoken answers come from tts_model (an OpenAI-compatible /audio/speech), else the browser reads them # stt: the engine that hears voice chat ("same": the transcription engine); tts_provider: who reads answers aloud @@ -1483,6 +1487,12 @@ def close(self): "DEFINE INDEX IF NOT EXISTS browser_action_session ON browser_action FIELDS session, state", # each conversation's code sandbox on the OpenShell gateway (sandbox.py): its name, settings and last use "DEFINE TABLE IF NOT EXISTS code_sandbox SCHEMALESS", + # deep research (research.py): a goal's plan, its run's progress and report; research_run: is the claim of + # the process carrying a run out, with its heartbeat + "DEFINE TABLE IF NOT EXISTS research SCHEMALESS", + "DEFINE INDEX IF NOT EXISTS research_account ON research FIELDS account, status", + "DEFINE INDEX IF NOT EXISTS research_chat ON research FIELDS chat", + "DEFINE TABLE IF NOT EXISTS research_run SCHEMALESS", # files a conversation's sandbox handed the person to download (sandbox.share), kept with the conversation "DEFINE TABLE IF NOT EXISTS chat_file SCHEMALESS", "DEFINE INDEX IF NOT EXISTS chat_file_chat ON chat_file FIELDS chat", diff --git a/fastapi_backend/app/schemas/chats.py b/fastapi_backend/app/schemas/chats.py index bb4c9d333..51f2df005 100644 --- a/fastapi_backend/app/schemas/chats.py +++ b/fastapi_backend/app/schemas/chats.py @@ -261,6 +261,9 @@ class ChatMessage(ResponseModel): parent: int | None = Field(None, description="the message it follows; null for a first question") versions: Versions | None = Field(None, description="it and the others said in its place, from regenerating or editing") meta: MessageRecord | None = Field(None, description="what produced it; none on messages from before it was recorded") + kind: Literal["research", "research_plan"] | None = Field( + None, description="research: a deep research report; research_plan: the plan it waits on (GET /research/{meta.research})" + ) class MessageDetail(ChatMessage): @@ -351,6 +354,11 @@ class MessageCreate(RequestModel): None, description="what this answer draws on, instead of the conversation's scope (which stays as it is); {} for everything you can read", ) + research: bool | None = Field( + None, + description="the content is a research goal: the reply is a plan to approve (streams `research`, then `approval`), " + "as /research and POST /chats/{cid}/research do", + ) class ToolInfo(ResponseModel): diff --git a/fastapi_backend/app/schemas/research.py b/fastapi_backend/app/schemas/research.py new file mode 100644 index 000000000..233c61763 --- /dev/null +++ b/fastapi_backend/app/schemas/research.py @@ -0,0 +1,138 @@ +"""Deep research: a goal's plan, its run in the background and its report (app/domain/research.py).""" + +from __future__ import annotations + +from typing import Any, Literal + +from pydantic import Field + +from app.schemas.chats import ChatScope, MessageRecord +from app.schemas.common import RequestModel, ResponseModel +from app.schemas.podcasts import Style + +Source = Literal["archive", "graph", "notes", "web", "code"] + + +class ResearchQuestion(ResponseModel): + question: str + answers: list[str] = Field(default_factory=list, description="suggested answers to pick from") + + +class ResearchPlan(ResponseModel): + title: str + goal: str + sub_questions: list[str] = Field(description="what it answers, one look-up at a time, in order") + sources: list[Source] = Field(description="where it looks: the archive, the graph, notes, the web, the code sandbox") + max_tool_calls: int = Field(description="the most tool calls the run makes in all (research.max_tool_calls at most)") + report_outline: list[str] = Field(default_factory=list, description="the report's sections") + + +class ResearchEstimate(ResponseModel): + usd: float | None = Field(None, description="what it will likely cost, from the prices in Settings; null without a price") + tokens: int | None = None + tool_calls: int | None = None + basis: str | None = Field(None, description="what the estimate is drawn from") + estimate: bool = True + + +class ResearchProgress(ResponseModel): + sub_questions: int = 0 + done: int = Field(0, description="sub-questions answered") + sources: int = Field(0, description="sources found so far") + tool_calls: int = 0 + cost_usd: float = Field(0.0, description="spent so far (the activity ledger rows of research:)") + tokens: int = 0 + cost_estimated: bool = False + minutes: float = Field(0.0, description="how long it has run") + + +class ResearchFinding(ResponseModel): + question: str + tool_calls: int = 0 + + +class ResearchHeld(ResponseModel): + why: str + budget: str | None = Field(None, description="the budget it waits for (all, space:)") + + +class Research(ResponseModel): + id: int + chat: int + message: int | None = Field(None, description="the question that asked for it") + plan_message: int | None = Field(None, description="the answer that shows the plan") + answer: int | None = Field(None, description="the report, once saved in the conversation") + approval: int | None = Field( + None, description="the plan's approval (POST /approvals/{id}: approve, with `params` its edits, or decline)" + ) + status: Literal["planned", "declined", "queued", "running", "held", "done", "stopped", "failed"] = Field( + description="planned (waiting for approval), declined, queued (approved, waiting its turn), running, held (by a " + "budget), done, stopped or failed" + ) + stage: Literal["gather", "write", "check", "done"] | None = None + goal: str | None = None + plan: ResearchPlan + questions: list[ResearchQuestion] = Field(default_factory=list, description="follow-up questions; answer them with revise") + answers: dict[str, str] = Field(default_factory=dict, description="the answers given to them") + scope: dict[str, Any] = Field(default_factory=dict) + estimate: ResearchEstimate | None = None + progress: ResearchProgress + findings: list[ResearchFinding] = Field(default_factory=list, description="the sub-questions answered so far") + held: ResearchHeld | None = None + error: str | None = None + ended: Literal["stopped", "timeout", "budget"] | None = Field( + None, description="why it ended early: stopped, out of time (research.max_minutes), or a budget" + ) + revisions: int = 0 + notes: list[int] = Field(default_factory=list, description="notes made from the report") + created_at: str | None = None + started_at: str | None = None + finished_at: str | None = None + + +class ResearchStart(RequestModel): + goal: str = Field(min_length=1, max_length=4000, description="what to research") + tools: list[str] | None = Field(None, max_length=100, description="what it may use (GET /chats/tools); null: the conversation's choice") + max_tool_calls: int | None = Field(None, ge=1, description="the most tool calls the plan may make (research.max_tool_calls at most)") + scope: ChatScope | None = Field(None, description="what it draws on, instead of the conversation's scope") + review: bool = Field( + True, + description="false: start the run at once, without the plan waiting for review (signed in; an API key or an app " + "only with research.agents_skip_review)", + ) + + +class ResearchStarted(ResponseModel): + research: Research + question: int = Field(description="the question saved in the conversation") + message: int = Field(description="the answer that shows the plan") + text: str = Field(description="the plan as the conversation shows it") + meta: MessageRecord | None = None + + +class ResearchRevise(RequestModel): + answers: dict[str, str] = Field(default_factory=dict, description="the follow-up questions' answers, by question") + plan: dict[str, Any] | None = Field( + None, description="the plan as edited: any of title, sub_questions, sources, max_tool_calls, report_outline" + ) + + +class ResearchNote(RequestModel): + namespace: str | None = Field(None, description="where the note goes; needed when the research looked in several") + + +class ResearchNoteMade(ResponseModel): + page: int + namespace: str + + +class ResearchPodcast(RequestModel): + prompt: str | None = Field(None, max_length=2000, description="an angle; else the research goal") + length: int = Field(10, ge=1, le=120, description="minutes (up to podcasts.max_minutes)") + style: Style = "deep-dive" + title: str | None = Field(None, max_length=200, description="else the report's title") + + +class ResearchStop(ResponseModel): + ok: bool = True + stopping: bool = Field(description="it stops after the step it's on (or now, when it hadn't started or waited on a budget)") diff --git a/fastapi_backend/tests/api/_assist.py b/fastapi_backend/tests/api/_assist.py index 54d52a079..1d39042c0 100644 --- a/fastapi_backend/tests/api/_assist.py +++ b/fastapi_backend/tests/api/_assist.py @@ -28,6 +28,7 @@ def start_llm(cfg): fake_llm.Handler.ignore_tool_choice = False fake_llm.Handler.title = None fake_llm.Handler.names, fake_llm.Handler.cypher_script = None, [] + fake_llm.Handler.research_plan = fake_llm.Handler.research_report = None fake_llm.Handler.seen = [] # what earlier tests on this worker sent isn't this test's srv, url = fake_llm.start() cfg["llm"].update(base_url=url, model="fake") diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index 819a7fd1b..134636f61 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -97,6 +97,7 @@ def test_clients_are_sent_to_sign_in(client, app, env): "unsubscribe_feed", "mark_activities", "decline_approval", + "start_research", "set_automation_vars", "save_automation", "run_automation", diff --git a/fastapi_backend/tests/api/test_research.py b/fastapi_backend/tests/api/test_research.py new file mode 100644 index 000000000..f8e7bc7a4 --- /dev/null +++ b/fastapi_backend/tests/api/test_research.py @@ -0,0 +1,583 @@ +"""Deep research (research.py, docs/assistant.md#deep-research): a goal gets a plan with follow-up questions, which +waits for the person's approval (in the web app, the inbox or a chat room; an API key or an app only declines it unless +research.agents_skip_review); approved, it runs in stages, claimed by one process at a time, within its tool calls, +minutes and budgets, and stops when asked; its report comes back checked against its sources, and becomes a note or a +podcast episode.""" + +from __future__ import annotations + +import json + +import pytest + +from app.api.v1.routes import podcasts as podcast_routes +from app.domain import activity, ai_tools, bridge, budgets, chat, research, store +from tests import fake_llm +from tests.api._assist import Assist, sse, start_llm +from tests.api.test_web_search import _Site, site # noqa: F401 - the fake web site, for a plan that looks on the web + +GOAL = "What happened with the capsid work?" +REPORT = "# Capsid research\n\nThe capsid model beat the human designs [1]. Samples ship on Friday [2]. Mars has capsids too [2]." + + +@pytest.fixture +def llm(cfg): + srv = start_llm(cfg) + yield fake_llm.Handler + srv.shutdown() + + +@pytest.fixture +def app(cfg, db, llm): + from app.main import create_app + + return create_app(cfg, db, background=False) + + +@pytest.fixture +def now(monkeypatch): + """A run carried out where it's launched, not in a thread of its own, so the test sees it end.""" + monkeypatch.setattr(research, "launch", lambda db, cfg, rid: research.run(db, cfg, rid)) + + +@pytest.fixture +def launched(monkeypatch): + """The runs launched, none of them carried out.""" + got: list[int] = [] + monkeypatch.setattr(research, "launch", lambda db, cfg, rid: got.append(rid)) + return got + + +def call(i, name, args): + return {"id": f"c{i}", "type": "function", "function": {"name": name, "arguments": json.dumps(args)}} + + +def archive(): + """The model's turns for the default plan's two sub-questions: one search of the archive each, then a finding.""" + return [ + {"content": "", "tool_calls": [call(1, "search_transcripts", {"query": "capsid model"})]}, + {"content": "The capsid model beat the human designs [1]."}, + {"content": "", "tool_calls": [call(2, "search_transcripts", {"query": "exploit benchmark"})]}, + {"content": "The exploit benchmark measures working exploits [1]."}, + ] + + +def new_chat(c, h): + return c.post("/api/v1/chats", headers=h, json={}).json()["id"] + + +def plan(c, h, cid, goal=GOAL, **body): + r = c.post(f"/api/v1/chats/{cid}/research", headers=h, json={"goal": goal, **body}) + assert r.status_code == 200, r.text + return r.json()["research"] + + +def approve(c, h, res, **body): + return c.post(f"/api/v1/approvals/{res['approval']}", headers=h, json={"decision": "approve", **body}) + + +def status(c, h, res): + return c.get(f"/api/v1/research/{res['id']}", headers=h).json() + + +def token(c, h): + return {"Authorization": f"Bearer {c.post('/api/v1/tokens', headers=h, json={'name': 'agent', 'scope': 'write'}).json()['token']}"} + + +def mcp(c, h, name, **args): + r = c.post("/mcp", headers=h, json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": name, "arguments": args}}) + assert r.status_code == 200, r.text + res = r.json()["result"] + assert res["isError"] is False, res + return res["structuredContent"] + + +def live(app): + """The settings the app's requests read (a copy of cfg, kept until a setting is saved).""" + return app.state.settings.current() + + +def set_up(s, section, **values): + """An admin saves settings, as in the web app.""" + ca, ha = s.cl["admin"] + r = ca.put(f"/api/v1/settings/{section}", headers=ha, json=values) + assert r.status_code == 200, r.text + + +def schema(body): + return ((body.get("response_format") or {}).get("json_schema") or {}).get("schema") or {} + + +def me(db, email="ed@x.io"): + return {**db.one("SELECT record::id(id) AS id, email FROM account WHERE email = $e", e=email), "admin": False} + + +def test_a_goal_gets_a_plan_that_waits_for_approval_and_can_be_revised(app, db, cfg, folder, new_client, llm): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + r = c.post(f"/api/v1/chats/{cid}/research", headers=h, json={"goal": GOAL}) + assert r.status_code == 200, r.text + got = r.json() + res = got["research"] + assert (res["status"], res["stage"], res["goal"]) == ("planned", None, GOAL) + assert res["plan"] == { + "title": "Capsid research", + "goal": GOAL, + "sub_questions": ["What did the capsid model do?", "When do the samples ship?"], + "sources": ["archive"], + "max_tool_calls": 6, + "report_outline": ["Findings", "Open questions"], + } + assert res["questions"] == [{"question": "Which period?", "answers": ["This year", "All time"]}] + # what it will likely cost, labelled an estimate (no past runs yet: its tokens, and the fake model has no price) + est = res["estimate"] + assert est["estimate"] is True and est["tool_calls"] == 6 and est["usd"] is None and "fake has no price" in est["basis"] + assert res["progress"] == { + "sub_questions": 2, + "done": 0, + "sources": 0, + "tool_calls": 0, + "cost_usd": 0.0, + "tokens": 0, + "cost_estimated": False, + "minutes": 0.0, + } + # the planner was told the sources this person has, and the most tool calls a plan may make + asked = next(b for b in llm.seen if {"questions", "plan"} <= set(schema(b).get("properties", {}))) + assert "only from: archive, graph, notes" in asked["messages"][0]["content"] and "at most 40" in asked["messages"][0]["content"] + assert asked["messages"][-1]["content"] == f"Goal: {GOAL}" + # the conversation: the goal as a question asked for research, then the plan as the answer it waits on + view = c.get(f"/api/v1/chats/{cid}", headers=h).json() + assert view["title"] == GOAL + q, a = view["messages"] + assert q["content"] == GOAL and q["meta"]["via"] == "web" and q["meta"]["options"]["research"] is True + assert a["id"] == got["message"] and a["kind"] == "research_plan" and a["content"] == got["text"] + assert a["meta"]["research"] == res["id"] and a["meta"]["outcome"] == "answered" + assert "1. What did the capsid model do?" in got["text"] and "- Which period? (This year / All time)" in got["text"] + # it waits as an approval of the assistant's: the person approves it (as it is or edited) or declines it + ap = c.get(f"/api/v1/approvals/{res['approval']}", headers=h).json() + assert ap["tool"] == "research" and ap["status"] == "pending" and ap["chat"] == cid + assert ap["summary"] == "Research: Capsid research (2 sub-questions, in the archive, up to 6 tool calls)" + assert ap["can"] == {"approve": True, "decline": True, "edit": True, "always": False} + assert ap["estimate"]["tool_calls"] == 6 + assert [x["id"] for x in c.get(f"/api/v1/research?chat_id={cid}", headers=h).json()] == [res["id"]] + # revised: the answers and the edits go back for one more planning call, and the plan and its approval change + before = len(llm.seen) + edits = {"sub_questions": ["What did the capsid model do?"], "max_tool_calls": 3} + r = c.post(f"/api/v1/research/{res['id']}/revise", headers=h, json={"answers": {"Which period?": "This year"}, "plan": edits}) + assert r.status_code == 200, r.text + rev = r.json() + sent = llm.seen[before]["messages"][-1]["content"] + assert "Sub-questions:\n- What did the capsid model do?\nSources: archive\nTool calls: 3" in sent + assert sent.endswith("Their answers:\n- Which period?: This year") + assert rev["plan"]["title"] == "Capsid research (revised)" and rev["questions"] == [] and rev["revisions"] == 1 + assert rev["answers"] == {"Which period?": "This year"} and rev["status"] == "planned" + a = db.one("SELECT args, summary FROM $r", r=store.R("approval", res["approval"])) + assert a["args"]["plan"]["title"] == "Capsid research (revised)" and a["summary"].startswith("Research: Capsid research (revised)") + # both planning calls count for the research, and the conversation + assert activity.totals(db, f"research:{res['id']}")["by_kind"]["out"]["calls"] == 2 + assert activity.totals(db, f"chat:{cid}")["by_kind"]["out"]["calls"] == 2 + # another person's research isn't there for them + cv, hv = s.cl["viewer"] + assert cv.get(f"/api/v1/research/{res['id']}", headers=hv).status_code == 404 + assert cv.post(f"/api/v1/research/{res['id']}/revise", headers=hv, json={}).status_code in (403, 404) + + +def test_a_question_marked_research_or_slash_research_streams_the_plan(app, db, cfg, folder, new_client, llm): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + ev = sse(c.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": f"/research {GOAL}"}).text) + assert ev["research"][0]["status"] == "planned" and ev["research"][0]["goal"] == GOAL + assert ev["approval"][0]["tool"] == "research" and ev["approval"][0]["id"] == ev["research"][0]["approval"] + assert ev["token"][0]["text"].startswith("Here's a plan to research **Capsid research**") + assert ev["meta"][0]["research"] == ev["research"][0]["id"] and "done" in ev + msgs = chat.history(db, cid) + assert msgs[0]["content"] == f"/research {GOAL}" and msgs[-1]["kind"] == "research_plan" + # research: true does the same with the question as it is + ev = sse(c.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": "Capsid shipments", "research": True}).text) + assert ev["research"][0]["goal"] == "Capsid shipments" + # an ordinary question is answered as before + ev = sse(c.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": "What did Dyno say?"}).text) + assert "research" not in ev and ev["token"] + # with no model set up, research isn't offered + live(app)["llm"]["base_url"] = None + assert c.post(f"/api/v1/chats/{cid}/research", headers=h, json={"goal": GOAL}).status_code == 400 + + +def test_an_approved_plan_runs_and_its_report_comes_back_checked(app, db, cfg, folder, new_client, llm, now): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + res = plan(c, h, cid) + llm.tool_script = archive() + r = approve(c, h, res) + assert r.status_code == 200, r.text + assert r.json()["status"] == "done" and r.json()["result"] == {"research": res["id"], "status": "queued"} + got = status(c, h, res) + assert (got["status"], got["stage"], got["ended"], got["error"]) == ("done", "done", None, None) + assert got["progress"]["done"] == 2 and got["progress"]["tool_calls"] == 2 and got["progress"]["sources"] >= 2 + assert got["findings"] == [ + {"question": "What did the capsid model do?", "tool_calls": 1}, + {"question": "When do the samples ship?", "tool_calls": 1}, + ] + assert got["started_at"] and got["finished_at"] and llm.tool_script == [] + # each sub-question was looked up with the archive's tools alone, within its share of the plan's tool calls + rounds = [b for b in llm.seen if b.get("tools")] + assert len(rounds) == 4 + assert {t["function"]["name"] for b in rounds for t in b["tools"]} <= set(research.SOURCE_TOOLS["archive"]) + assert "up to 3 tool calls" in rounds[0]["messages"][0]["content"] + assert "Answer this part of it: What did the capsid model do?" in rounds[0]["messages"][-1]["content"] + # the report was written from both findings, their citations numbered as one list + writer = next(b for b in llm.seen if b["messages"][0]["content"].startswith("You write research reports")) + told = writer["messages"][-1]["content"] + assert "The capsid model beat the human designs [1]." in told and "The exploit benchmark measures working exploits [2]." in told + assert "- Findings\n- Open questions" in told + # checked: the first unsupported claim rewritten, the one the excerpts don't support at all marked + view = c.get(f"/api/v1/chats/{cid}", headers=h).json() + report = view["messages"][-1] + assert report["id"] == got["answer"] and report["kind"] == "research" and report["parent"] == res["plan_message"] + assert report["content"] == REPORT + research.UNSUPPORTED + assert [p["n"] for p in report["passages"]] == [1, 2] + assert {p["recording_id"] for p in report["passages"]} <= {s.a, s.b} + assert [x["tool"] for x in report["steps"]] == ["search_transcripts", "search_transcripts"] + meta = report["meta"] + assert meta["research"] == res["id"] and meta["outcome"] == "answered" and meta["tools"]["picked"] == ["archive"] + assert meta["used"] == {"rounds": 2, "tool_calls": 2} and meta["limits"]["tool_calls"] == 6 + # every call it made counts for research:: planning, four rounds, writing, checking, rewriting + assert activity.totals(db, f"research:{res['id']}")["by_kind"]["out"]["calls"] == 8 + assert db.rows("SELECT * FROM research_run") == [] # its claim is let go + # an agent follows it: the report, and its sources with links + st = mcp(c, token(c, h), "research_status", id=res["id"]) + assert st["status"] == "done" and st["report"] == report["content"] and st["url"].endswith(f"/chat/{cid}") + assert [x["n"] for x in st["sources"]] == [1, 2] and all(x["url"] for x in st["sources"]) + # the next plan's estimate is drawn from what this run cost per tool call + nxt = plan(c, h, cid, "More about capsids") + assert nxt["estimate"]["basis"] == "what each tool call of 1 past research run cost" and nxt["estimate"]["tool_calls"] == 6 + + +def test_a_plan_that_looks_on_the_web_searches_without_asking_again(site, app, db, cfg, folder, new_client, llm, now): # noqa: F811 - the site first: it widens the networks the app reads + s = Assist(app, db, cfg, folder, new_client) + set_up(s, "web", search="searxng", searxng_url=f"{site}/searx") + cfg["web"].update(search="searxng", searxng_url=f"{site}/searx") + llm.research_plan = { + "title": "Harbour research", + "goal": "When do boats come in?", + "sub_questions": ["When do boats come in?"], + "sources": ["archive", "web"], + "max_tool_calls": 4, + "report_outline": [], + } + llm.research_report = "# Harbour research\n\nBoats come in at dawn [1]." + c, h = s.cl["editor"] + cid = new_chat(c, h) + res = plan(c, h, cid, "When do boats come in at the harbour?") + assert ( + res["plan"]["sources"] == ["archive", "web"] + and "the web" in c.get(f"/api/v1/approvals/{res['approval']}", headers=h).json()["summary"] + ) + # the second search's words are the model's own, after it read the first's results: the plan approved the web, + # so it searches without asking again + llm.tool_script = [ + {"content": "", "tool_calls": [call(1, "web_search", {"query": "harbour boats"})]}, + {"content": "", "tool_calls": [call(2, "web_search", {"query": "secret budget figures"})]}, + {"content": "Boats come in at dawn [1]."}, + ] + assert approve(c, h, res).status_code == 200 + assert status(c, h, res)["status"] == "done" + assert [g["query"]["q"] for g in _Site.got if g["path"] == "/searx/search"] == ["harbour boats", "secret budget figures"] + assert db.rows("SELECT id FROM approval WHERE tool != 'research'") == [] + report = c.get(f"/api/v1/chats/{cid}", headers=h).json()["messages"][-1] + assert report["kind"] == "research" and report["passages"][0]["source"] == "web" + assert report["passages"][0]["url"] == "https://www.harbour.example/news" + # the web tools allowed for the run are the plan's alone + box = research.toolbox(db, cfg, research.get(db, res["id"]), research.account_of(db, me(db)["id"])) + assert box.allowed_out == {"web_search", "web_read"} and box.only == set( + research.SOURCE_TOOLS["archive"] + research.SOURCE_TOOLS["web"] + ) + + +def test_who_may_approve_a_plan(app, db, cfg, folder, new_client, llm, launched): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + ht = token(c, h) + cid = new_chat(c, h) + res = plan(c, h, cid) + # an API key can only decline it, and can't start research without review + assert c.get(f"/api/v1/approvals/{res['approval']}", headers=ht).json()["can"] == { + "approve": False, + "decline": True, + "edit": False, + "always": False, + } + r = approve(c, ht, res) + assert r.status_code == 403 and "the person's own pick" in r.json()["detail"] + assert c.post(f"/api/v1/chats/{cid}/research", headers=ht, json={"goal": GOAL, "review": False}).status_code == 403 + assert status(c, h, res)["status"] == "planned" and launched == [] + r = c.post(f"/api/v1/approvals/{res['approval']}", headers=ht, json={"decision": "decline"}) + assert r.status_code == 200 and r.json()["status"] == "declined" + assert status(c, h, res)["status"] == "declined" and approve(c, h, res).status_code == 400 # decided once + # the person approves it edited, within the sources they have + res = plan(c, h, cid) + r = approve(c, h, res, params={"sources": ["web"]}) + assert r.status_code == 400 and r.json()["detail"] == "a plan looks in archive, graph, notes" + assert status(c, h, res)["status"] == "planned" # a bad edit doesn't decide it + r = approve(c, h, res, params={"sub_questions": ["What did the capsid model do?"], "max_tool_calls": 2}) + assert r.status_code == 200, r.text + got = status(c, h, res) + assert got["status"] == "queued" and got["plan"]["sub_questions"] == ["What did the capsid model do?"] + assert got["plan"]["max_tool_calls"] == 2 and launched == [res["id"]] + # signed in, the person may skip the review; so may an agent once research.agents_skip_review lets it + r = c.post(f"/api/v1/chats/{cid}/research", headers=h, json={"goal": GOAL, "review": False}) + assert r.status_code == 200 and r.json()["research"]["status"] == "queued" + set_up(s, "research", agents_skip_review=True) + res = plan(c, ht, cid) + assert c.get(f"/api/v1/approvals/{res['approval']}", headers=ht).json()["can"]["approve"] is True + assert approve(c, ht, res).status_code == 200 and status(c, h, res)["status"] == "queued" + # a person has one run going at a time: the others wait their turn (none was running here, so each launched) + assert len(launched) == 3 + + +def test_a_plan_is_approved_with_a_yes_in_a_chat_room(app, db, cfg, folder, new_client, llm, launched): + Assist(app, db, cfg, folder, new_client) + account = me(db) + cfg["bridge"] = {**cfg["bridge"], "users": ["alice"], "approve": "low_risk"} + cid = chat.create(db, account["id"], kind="bridge") + text, waiting = bridge.answer(db, cfg, cfg, account, cid, f"/research {GOAL}") + assert text.startswith("Here's a plan to research **Capsid research**") and [a["tool"] for a in waiting] == ["research"] + rid = db.one("SELECT record::id(id) AS id FROM research WHERE approval = $a", a=waiting[0]["id"])["id"] + q = chat.history(db, cid)[0] + assert q["content"] == f"/research {GOAL}" and q["meta"] == {"via": "room", "options": {"research": True}} + # someone not let in to approve isn't heard; alice's yes approves it as it is + assert bridge.decide_in_room(db, cfg, cfg, account, cid, {"username": "eve"}, "yes") is None + assert bridge.decide_in_room(db, cfg, cfg, account, cid, {"username": "alice"}, "yes") == f"Done: {waiting[0]['summary']}" + assert research.get(db, rid)["status"] == "queued" and launched == [rid] + + +def test_agents_start_research_over_mcp(app, db, cfg, folder, new_client, llm, launched): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + ht = token(c, h) + got = mcp(c, ht, "start_research", goal=GOAL) + assert got["status"] == "planned" and got["approval"] and got["next"].startswith("waiting for the person to approve") + assert got["url"].endswith(f"/chat/{got['chat_id']}") and got["questions"][0]["question"] == "Which period?" + assert chat.get(db, got["chat_id"], me(db)["id"])["title"] == GOAL + st = mcp(c, ht, "research_status", id=got["research"]) + assert st["status"] == "planned" and "report" not in st + # declining is an agent's to do; approving isn't + with pytest.raises(PermissionError): + ai_tools.approve(db, cfg, got["approval"], me(db), {s.pods}, via="agent") + # with research.agents_skip_review, it starts at once + set_up(s, "research", agents_skip_review=True) + again = mcp(c, ht, "start_research", goal="Capsid shipments", chat_id=got["chat_id"], tools=["archive"], max_tool_calls=3) + assert again["status"] == "queued" and again["next"].startswith("running") and launched == [again["research"]] + r = research.get(db, again["research"]) + assert r["plan"]["max_tool_calls"] == 3 and r["tools"] == ["archive"] + + +def test_the_assistant_plans_research_with_its_own_tool(app, db, cfg, folder, new_client, llm): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + groups = {g["name"]: g for g in c.get("/api/v1/chats/tools", headers=h).json()["groups"]} + assert groups["research"]["available"] and [t["name"] for t in groups["research"]["tools"]] == ["start_research"] + llm.tool_script = [ + {"content": "", "tool_calls": [call(1, "start_research", {"goal": GOAL})]}, + {"content": "I planned it: approve it to start."}, + ] + ev = sse(c.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": "Research the capsid work for me"}).text) + assert ev["step"][0]["summary"].startswith("Planned Research: Capsid research (2 sub-questions") + assert ev["approval"][0]["tool"] == "research" + r = db.one("SELECT record::id(id) AS id, status, chat FROM research WHERE approval = $a", a=ev["approval"][0]["id"]) + assert r["status"] == "planned" and r["chat"] == cid + assert activity.totals(db, f"research:{r['id']}")["by_kind"]["out"]["calls"] == 1 # its planning call + # outside a conversation there's nothing to research in + box = ai_tools.Toolbox(db, cfg, me(db), {s.pods}, {s.pods}, {}, None) + assert "start_research" not in {t["function"]["name"] for t in box.specs()} + # without a model that uses tools, the group says why + live(app)["ai"]["tools"] = False + groups = {g["name"]: g for g in c.get("/api/v1/chats/tools", headers=h).json()["groups"]} + assert not groups["research"]["available"] and groups["research"]["setting"] == "ai.tools" + cfg["llm"]["base_url"] = None + assert research.unavailable(cfg) == ("no language model is set up", "llm.model") + + +def test_a_budget_holds_a_run_until_it_allows_it(app, db, cfg, folder, new_client, llm, now): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + res = plan(c, h, cid) + budgets.put(db, "all", usd=0.5, on_over="ask") + activity.record("out", "test.spend", db=db, cost_usd=1.0) + llm.tool_script = archive() + assert approve(c, h, res).status_code == 200 + got = status(c, h, res) + assert got["status"] == "held" and got["held"]["budget"] == "all" and "spent" in got["held"]["why"] + assert got["progress"]["done"] == 0 and len(llm.tool_script) == 4 # it asked nothing + # the sweep leaves it while the budget is over, and starts it once the budget allows it + assert research.sweep(db, cfg) == [] + budgets.put(db, "all", usd=100, on_over="ask") + assert research.sweep(db, cfg) == [res["id"]] + assert status(c, h, res)["status"] == "done" and status(c, h, res)["held"] is None + # the person can let a held run go on anyway; only a held one + budgets.put(db, "all", usd=0.5, on_over="ask") + res = plan(c, h, cid) + llm.tool_script = archive() + approve(c, h, res) + assert status(c, h, res)["status"] == "held" + r = c.post(f"/api/v1/research/{res['id']}/resume", headers=h) + assert r.status_code == 200, r.text + assert status(c, h, res)["status"] == "done" + assert c.post(f"/api/v1/research/{res['id']}/resume", headers=h).status_code == 409 + # a budget that skips stops the run with what it has + budgets.put(db, "all", usd=0.5, on_over="skip") + res = plan(c, h, cid) + approve(c, h, res) + got = status(c, h, res) + assert (got["status"], got["ended"]) == ("stopped", "budget") + last = c.get(f"/api/v1/chats/{cid}", headers=h).json()["messages"][-1] + assert last["id"] == got["answer"] and last["content"] == "# Capsid research\n\nStopped before it found anything." + + +def test_stop_ends_a_run_and_keeps_what_it_found(app, db, cfg, folder, new_client, llm, monkeypatch): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + # one waiting its turn stops at once; a plan that isn't approved is declined, not stopped + monkeypatch.setattr(research, "launch", lambda db, cfg, rid: None) + res = plan(c, h, cid) + approve(c, h, res) + assert c.post(f"/api/v1/research/{res['id']}/stop", headers=h).json() == {"ok": True, "stopping": True} + got = status(c, h, res) + assert (got["status"], got["ended"], got["answer"]) == ("stopped", "stopped", None) + assert c.post(f"/api/v1/research/{plan(c, h, cid)['id']}/stop", headers=h).json()["stopping"] is False + # a running one stops after the step it's on: here, as the second sub-question's search comes back + monkeypatch.setattr(research, "launch", lambda db, cfg, rid: research.run(db, cfg, rid)) + res = plan(c, h, cid) + first, found, second, _ = archive() + + def stop_now(body): + research.request_stop(db, cfg, res["id"]) + return second + + llm.tool_script = [first, found, stop_now, {"content": "never asked"}] + approve(c, h, res) + got = status(c, h, res) + assert (got["status"], got["ended"], got["progress"]["done"]) == ("stopped", "stopped", 1) + assert llm.tool_script == [{"content": "never asked"}] + last = c.get(f"/api/v1/chats/{cid}", headers=h).json()["messages"][-1] + assert last["id"] == got["answer"] and last["kind"] == "research" and last["stopped"] is True + assert last["content"] == ( + "# Capsid research\n\nStopped. What it found so far:\n\n## What did the capsid model do?\nThe capsid model beat the human designs [1]." + ) + assert last["meta"]["outcome"] == "stopped" and [p["n"] for p in last["passages"]] == [1] + assert c.post(f"/api/v1/research/{res['id']}/stop", headers=h).json()["stopping"] is False # it's over + + +def test_a_run_whose_process_went_away_is_taken_over_where_it_left_off(app, db, cfg, folder, new_client, llm, monkeypatch): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + monkeypatch.setattr(research, "launch", lambda db, cfg, rid: None) + res = plan(c, h, cid) + approve(c, h, res) + rid = res["id"] + # one process holds a run at a time, kept by its heartbeat + assert research.claim(db, rid, "a") and not research.claim(db, rid, "b") + assert research.beat(db, rid, "a") and not research.beat(db, rid, "b") + # process a answered the first sub-question, then went away + box = ai_tools.Toolbox(db, cfg, me(db), {s.pods}, {s.pods}, {}, cid) + box.call("search_transcripts", {"query": "capsid model"}) + finding = { + "question": "What did the capsid model do?", + "text": "The capsid model beat the human designs [1].", + "passages": box.cited("[1]"), + "tool_calls": 1, + "steps": [], + } + db.q("UPDATE $r SET status = 'running', findings = [$f], used = {tool_calls: 1}", r=store.R("research", rid), f=finding) + assert research.sweep(db, cfg) == [] # its claim is fresh: a process is on it + db.q("UPDATE $r SET heartbeat_at = $t", r=store.R("research_run", rid), t=research._ago(research.STALE + 5)) + monkeypatch.setattr(research, "launch", lambda db, cfg, rid: research.run(db, cfg, rid)) + llm.tool_script = archive()[2:] + assert research.sweep(db, cfg) == [rid] + got = status(c, h, res) + assert got["status"] == "done" and got["progress"]["done"] == 2 and got["progress"]["tool_calls"] == 2 + # only the second sub-question was looked up again + rounds = [b for b in llm.seen if b.get("tools")] + assert len(rounds) == 2 and "When do the samples ship?" in rounds[0]["messages"][-1]["content"] + assert not research.beat(db, rid, "a") and research.run(db, cfg, rid) is False # done: nothing to take + + +def test_a_report_becomes_a_note_and_a_podcast(app, db, cfg, folder, new_client, llm, now, monkeypatch): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + res = plan(c, h, cid) + assert c.post(f"/api/v1/research/{res['id']}/note", headers=h).status_code == 400 # no report yet + llm.tool_script = archive() + approve(c, h, res) + r = c.post(f"/api/v1/research/{res['id']}/note", headers=h) + assert r.status_code == 200, r.text + made = r.json() + assert made["namespace"] == "pods" + page = c.get(f"/api/v1/notes/{made['page']}", headers=h).json() + passages = research.get(db, res["id"])["passages"] + first = passages[0] + assert page["title"] == "Capsid research" and page["author"] == "assistant" + assert research.mention(first) == f"@[{first.get('title') or 'Asset'}](recording:{first['recording_id']})" + assert f"The capsid model beat the human designs ({research.mention(first)})." in page["body"] + assert page["body"].rstrip().endswith(f"@[Research: Capsid research](chat:{cid})") + links = {x["target"]: x for x in page["links"]} + assert links[f"chat:{cid}"]["name"] == GOAL and links[f"recording:{first['recording_id']}"]["name"] + assert status(c, h, res)["notes"] == [made["page"]] + # someone else who reads the note sees the conversation's link, but not its name + cv, hv = s.cl["viewer"] + seen = {x["target"]: x for x in cv.get(f"/api/v1/notes/{made['page']}", headers=hv).json()["links"]} + assert seen[f"chat:{cid}"]["name"] is None and seen[f"recording:{first['recording_id']}"]["name"] + # a podcast episode from the assets the report cites, about the goal, titled after the report + asked = [] + + def start(acl, user, db, cfg, body): + asked.append(body) + return {"episode": 99, "job": 1, "namespace": "podcasts", "placed": "podcasts"} + + monkeypatch.setattr(podcast_routes, "start", start) + r = c.post(f"/api/v1/research/{res['id']}/podcast", headers=h, json={"length": 5}) + assert r.status_code == 202, r.text + assert asked[0].selection.recordings == list(dict.fromkeys(p["recording_id"] for p in passages)) + assert (asked[0].prompt, asked[0].title, asked[0].length) == (GOAL, "Capsid research", 5) + + +def test_research_settings(app, db, cfg, folder, new_client, llm): + s = Assist(app, db, cfg, folder, new_client) + ca, ha = s.cl["admin"] + for bad in ({"max_tool_calls": 0}, {"max_minutes": 241}, {"agents_skip_review": "yes"}, {"model": 3}, {"max_tool_calls": True}): + assert ca.put("/api/v1/settings/research", headers=ha, json=bad).status_code == 400, bad + r = ca.put("/api/v1/settings/research", headers=ha, json={"max_tool_calls": 12, "max_minutes": 5, "model": " fake-large "}) + assert r.status_code == 200, r.text + got = ca.get("/api/v1/settings", headers=ha).json()["research"]["values"] + assert got == {"max_tool_calls": 12, "max_minutes": 5, "model": "fake-large", "agents_skip_review": False} + # a plan's tool calls stay within research.max_tool_calls + c, h = s.cl["editor"] + cid = new_chat(c, h) + assert c.post(f"/api/v1/chats/{cid}/research", headers=h, json={"goal": GOAL, "max_tool_calls": 13}).status_code == 400 + assert research.clean_plan(cfg | {"research": got}, {"sub_questions": ["a"], "max_tool_calls": 99}, ["archive"])["max_tool_calls"] == 12 + assert ca.put("/api/v1/settings/research", headers=s.cl["editor"][1], json={"max_minutes": 9}).status_code == 403 + + +def test_a_run_out_of_time_writes_its_report_with_what_it_found(app, db, cfg, folder, new_client, llm, now, monkeypatch): + s = Assist(app, db, cfg, folder, new_client) + c, h = s.cl["editor"] + cid = new_chat(c, h) + res = plan(c, h, cid) + # out of time once the first sub-question is answered + monkeypatch.setattr(research.Run, "out_of_time", lambda self: bool(research.get(self.db, self.rid).get("findings") or [])) + llm.tool_script = archive() + approve(c, h, res) + got = status(c, h, res) + assert (got["status"], got["ended"], got["progress"]["done"]) == ("done", "timeout", 1) + writer = next(b for b in llm.seen if b["messages"][0]["content"].startswith("You write research reports")) + assert "ran out of time before every sub-question was looked into" in writer["messages"][-1]["content"] + assert c.get(f"/api/v1/chats/{cid}", headers=h).json()["messages"][-1]["meta"]["outcome"] == "limit" diff --git a/fastapi_backend/tests/fake_llm.py b/fastapi_backend/tests/fake_llm.py index 615e5f8ec..4d3be7a79 100644 --- a/fastapi_backend/tests/fake_llm.py +++ b/fastapi_backend/tests/fake_llm.py @@ -19,6 +19,8 @@ class Handler(http.server.BaseHTTPRequestHandler): podcast_script = None # the lines a podcast script comes back with (podcasts.write), else a built-in one spec = None # a spec's answer (specs.write), else one quoting the transcript's lines in turn (see _spec) usage = None # token counts to report with each answer (and as a streamed answer's last chunk), like OpenAI + research_plan = None # a research plan (research.plan) to come back with, else a built-in one about the capsid + research_report = None # a research report (research._write) to come back with, else one citing [1] and [2] title = None # a conversation's title (chat.suggest_title), else the first question, labelled and quoted as models do def _json(self, obj): @@ -173,6 +175,30 @@ def do_POST(self): else: checks.append({"line": int(k), "verdict": "supported"}) content = json.dumps({"checks": checks}) + elif {"questions", "plan"} <= set(schema.get("properties", {})): + # a research plan (research.plan): Handler.research_plan, else two sub-questions about the capsid in the + # archive, and a follow-up question unless it's a revision (answers given) + text = body["messages"][-1]["content"] + goal = re.search(r"^Goal: (.*)$", text, re.M).group(1) + plan = Handler.research_plan or { + "title": "Capsid research", + "goal": goal, + "sub_questions": ["What did the capsid model do?", "When do the samples ship?"], + "sources": ["archive"], + "max_tool_calls": 6, + "report_outline": ["Findings", "Open questions"], + } + revised = "Their answers:" in text or "The earlier plan" in text + questions = [] if revised else [{"question": "Which period?", "answers": ["This year", "All time"]}] + if revised: + plan = {**plan, "title": plan["title"] + " (revised)"} + content = json.dumps({"questions": questions, "plan": plan}) + elif "rewrites" in schema.get("properties", {}): + # a research report's unsupported claims (research._check): the first one rewritten, the rest given up on + claims = [l[2:] for l in body["messages"][-1]["content"].splitlines() if l.startswith("- ")] + content = json.dumps( + {"rewrites": [{"claim": c, "text": "Samples ship on Friday [2]." if i == 0 else ""} for i, c in enumerate(claims)]} + ) elif "cypher" in schema.get("properties", {}): # graph questions: the scripted queries in order, else one that finds organisations q = Handler.cypher_script.pop(0) if Handler.cypher_script else "MATCH (e:Organisation) RETURN e, e.name LIMIT 50" @@ -225,6 +251,11 @@ def do_POST(self): content = json.dumps({"title": "Capsid plan" if title == "Untitled" else title, "summary": f"About {first}"}) elif set(schema.get("properties", {})) == {"choice", "confidence"}: content = json.dumps(Handler.decision or {"choice": schema["properties"]["choice"]["enum"][0], "confidence": 0.9}) + elif str(body["messages"][0].get("content") or "").startswith("You write research reports"): + content = Handler.research_report or ( + "# Capsid research\n\nThe capsid model beat the human designs [1]. The shipment leaves on Friday [2]. " + "Mars has capsids too [2]." + ) elif body.get("response_format"): content = json.dumps( { diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index b1fe8e144..98044fd24 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -36,6 +36,7 @@ export { Podcasts, Public, Rdf, + Research as Research2, Resources, Routines, Search, @@ -1399,6 +1400,11 @@ export type { GetRecordingWordcloudErrors, GetRecordingWordcloudResponse, GetRecordingWordcloudResponses, + GetResearchData, + GetResearchError, + GetResearchErrors, + GetResearchResponse, + GetResearchResponses, GetResourceFieldsData, GetResourceFieldsError, GetResourceFieldsErrors, @@ -2006,6 +2012,11 @@ export type { ListRecordingsErrors, ListRecordingsResponse, ListRecordingsResponses, + ListResearchData, + ListResearchError, + ListResearchErrors, + ListResearchResponse, + ListResearchResponses, ListRolesData, ListRolesResponse, ListRolesResponses, @@ -2744,6 +2755,30 @@ export type { RequestAccessErrors, RequestAccessResponse, RequestAccessResponses, + Research, + ResearchEstimate, + ResearchFinding, + ResearchHeld, + ResearchNote, + ResearchNoteData, + ResearchNoteError, + ResearchNoteErrors, + ResearchNoteMade, + ResearchNoteResponse, + ResearchNoteResponses, + ResearchPlan, + ResearchPodcast, + ResearchPodcastData, + ResearchPodcastError, + ResearchPodcastErrors, + ResearchPodcastResponse, + ResearchPodcastResponses, + ResearchProgress, + ResearchQuestion, + ResearchRevise, + ResearchStart, + ResearchStarted, + ResearchStop, ResetPasswordData, ResetPasswordError, ResetPasswordErrors, @@ -2791,6 +2826,11 @@ export type { RestoreVersionResponse, RestoreVersionResponses, RestoreWork, + ResumeResearchData, + ResumeResearchError, + ResumeResearchErrors, + ResumeResearchResponse, + ResumeResearchResponses, RetagRecordingsData, RetagRecordingsError, RetagRecordingsErrors, @@ -2815,6 +2855,11 @@ export type { ReviewNewData, ReviewNewResponse, ReviewNewResponses, + ReviseResearchData, + ReviseResearchError, + ReviseResearchErrors, + ReviseResearchResponse, + ReviseResearchResponses, RevokeAnyTokenData, RevokeAnyTokenError, RevokeAnyTokenErrors, @@ -3198,6 +3243,11 @@ export type { StartRcloneSigninErrors, StartRcloneSigninResponse, StartRcloneSigninResponses, + StartResearchData, + StartResearchError, + StartResearchErrors, + StartResearchResponse, + StartResearchResponses, StartUpgradeData, StartUpgradeError, StartUpgradeErrors, @@ -3219,6 +3269,11 @@ export type { StopAnswerErrors, StopAnswerResponse, StopAnswerResponses, + StopResearchData, + StopResearchError, + StopResearchErrors, + StopResearchResponse, + StopResearchResponses, StopResult, Stream, StreamActivity, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 331eea1ff..a721ced0b 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -677,6 +677,9 @@ import type { GetRecordingWordcloudData, GetRecordingWordcloudErrors, GetRecordingWordcloudResponses, + GetResearchData, + GetResearchErrors, + GetResearchResponses, GetResourceFieldsData, GetResourceFieldsErrors, GetResourceFieldsResponses, @@ -1003,6 +1006,9 @@ import type { ListRecordingsData, ListRecordingsErrors, ListRecordingsResponses, + ListResearchData, + ListResearchErrors, + ListResearchResponses, ListRolesData, ListRolesResponses, ListRoutinesData, @@ -1318,6 +1324,12 @@ import type { RequestAccessData, RequestAccessErrors, RequestAccessResponses, + ResearchNoteData, + ResearchNoteErrors, + ResearchNoteResponses, + ResearchPodcastData, + ResearchPodcastErrors, + ResearchPodcastResponses, ResetPasswordData, ResetPasswordErrors, ResetPasswordResponses, @@ -1342,6 +1354,9 @@ import type { RestoreVersionData, RestoreVersionErrors, RestoreVersionResponses, + ResumeResearchData, + ResumeResearchErrors, + ResumeResearchResponses, RetagRecordingsData, RetagRecordingsErrors, RetagRecordingsResponses, @@ -1356,6 +1371,9 @@ import type { RevertMetadataEditResponses, ReviewNewData, ReviewNewResponses, + ReviseResearchData, + ReviseResearchErrors, + ReviseResearchResponses, RevokeAnyTokenData, RevokeAnyTokenErrors, RevokeAnyTokenResponses, @@ -1520,6 +1538,9 @@ import type { StartRcloneSigninData, StartRcloneSigninErrors, StartRcloneSigninResponses, + StartResearchData, + StartResearchErrors, + StartResearchResponses, StartUpgradeData, StartUpgradeErrors, StartUpgradeResponses, @@ -1531,6 +1552,9 @@ import type { StopAnswerData, StopAnswerErrors, StopAnswerResponses, + StopResearchData, + StopResearchErrors, + StopResearchResponses, StreamEventsData, StreamEventsErrors, StreamEventsResponses, @@ -10448,6 +10472,10 @@ export class Chats { * `/clear` makes the model start afresh (the messages stay to read) and `/compact [what to keep]` folds the * conversation so far into its summary now; each streams cleared or compacted, then its reply as an answer. * + * With `research` (or /research ) the content is a research goal: the answer is a plan to approve (streamed as + * `research`, then `approval`), which runs in the background once approved (POST /chats/{cid}/research, GET + * /research/{id}). + * * With `browser` (the session of a browser's Lens side panel, POST /browser/sessions), the assistant can use that * browser while the side panel is connected: open pages, read them, click, type and download files into the * conversation (docs/browser-assistant.md). @@ -10560,6 +10588,148 @@ export class Chats { } } +export class Research { + /** + * Start Research + * + * Start research on a goal in a conversation: the goal is saved as a question and one model call plans it (up to + * three follow-up questions with answers to pick, the sub-questions, where it looks, the tool calls it may make, the + * report's outline, and an estimate of the cost). The plan waits as an approval (POST /approvals/{id}: approve it as + * it is, or with `params` its edits; or decline), or revise it (POST /research/{id}/revise). With `review` false it + * starts at once: your own pick when signed in; an API key or an app needs research.agents_skip_review. The same as a + * question with `research: true`, or /research . + */ + public static startResearch( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/chats/{cid}/research", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } + + /** + * List Research + * + * Your research, newest first; with `chat_id`, that conversation's (to show each plan's or run's card). + */ + public static listResearch( + options?: Options, + ): RequestResult { + return (options?.client ?? client).get({ + url: "/api/v1/research", + ...options, + }); + } + + /** + * Get Research + * + * A research run: its plan, status and stage (gather, write, check, done), progress (sub-questions done, sources + * found, tool calls and cost so far, from the activity ledger) and, once it's done, the report's message + * (`answer`, in the conversation). What it waits for when a budget holds it (`held`), and why it ended early. + */ + public static getResearch( + options: Options, + ): RequestResult { + return (options.client ?? client).get({ + url: "/api/v1/research/{rid}", + ...options, + }); + } + + /** + * Revise Research + * + * Send your answers to the follow-up questions and your edits to the plan back for one more planning call; the + * plan and its approval change in place. Only while the plan waits for approval. + */ + public static reviseResearch( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/research/{rid}/revise", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } + + /** + * Stop Research + * + * Stop a run: it stops after the step it's on and keeps what it found, saved in the conversation as its answer, + * marked stopped (one that hasn't started, or waits on a budget, stops now). Works whichever process runs it. + */ + public static stopResearch( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/research/{rid}/stop", + ...options, + }); + } + + /** + * Resume Research + * + * Let a run that a budget holds go on anyway (your pick, as "run it once" is for a held routine run). + */ + public static resumeResearch( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/research/{rid}/resume", + ...options, + }); + } + + /** + * Research Note + * + * Save the report as a note: in the namespace it looked in (`namespace` when there are several), its archive + * citations as links to the assets they cite, web citations as links, and a link back to the conversation (shown to + * you alone). Needs editor access there. + */ + public static researchNote( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/research/{rid}/note", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } + + /** + * Research Podcast + * + * Make a podcast episode from the assets the report cites (POST /podcasts with them picked): about the research + * goal unless `prompt` says otherwise, titled after the report. + */ + public static researchPodcast( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/research/{rid}/podcast", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } +} + export class Browser { /** * List Sessions diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 017e33798..a5b152252 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -3690,6 +3690,12 @@ export type ChatMessage = { * what produced it; none on messages from before it was recorded */ meta?: MessageRecord | null; + /** + * Kind + * + * research: a deep research report; research_plan: the plan it waits on (GET /research/{meta.research}) + */ + kind?: "research" | "research_plan" | null; [key: string]: unknown; }; @@ -11740,6 +11746,12 @@ export type MessageCreate = { * what this answer draws on, instead of the conversation's scope (which stays as it is); {} for everything you can read */ scope?: ChatScope | null; + /** + * Research + * + * the content is a research goal: the reply is a plan to approve (streams `research`, then `approval`), as /research and POST /chats/{cid}/research do + */ + research?: boolean | null; }; /** @@ -11824,6 +11836,12 @@ export type MessageDetail = { * what produced it; none on messages from before it was recorded */ meta?: MessageRecord | null; + /** + * Kind + * + * research: a deep research report; research_plan: the plan it waits on (GET /research/{meta.research}) + */ + kind?: "research" | "research_plan" | null; /** * Ledger * @@ -17758,6 +17776,442 @@ export type ReprocessRequest = { pipeline?: number | null; }; +/** + * Research + */ +export type Research = { + /** + * Id + */ + id: number; + /** + * Chat + */ + chat: number; + /** + * Message + * + * the question that asked for it + */ + message?: number | null; + /** + * Plan Message + * + * the answer that shows the plan + */ + plan_message?: number | null; + /** + * Answer + * + * the report, once saved in the conversation + */ + answer?: number | null; + /** + * Approval + * + * the plan's approval (POST /approvals/{id}: approve, with `params` its edits, or decline) + */ + approval?: number | null; + /** + * Status + * + * planned (waiting for approval), declined, queued (approved, waiting its turn), running, held (by a budget), done, stopped or failed + */ + status: "planned" | "declined" | "queued" | "running" | "held" | "done" | "stopped" | "failed"; + /** + * Stage + */ + stage?: "gather" | "write" | "check" | "done" | null; + /** + * Goal + */ + goal?: string | null; + plan: ResearchPlan; + /** + * Questions + * + * follow-up questions; answer them with revise + */ + questions?: Array; + /** + * Answers + * + * the answers given to them + */ + answers?: { + [key: string]: string; + }; + /** + * Scope + */ + scope?: { + [key: string]: unknown; + }; + estimate?: ResearchEstimate | null; + progress: ResearchProgress; + /** + * Findings + * + * the sub-questions answered so far + */ + findings?: Array; + held?: ResearchHeld | null; + /** + * Error + */ + error?: string | null; + /** + * Ended + * + * why it ended early: stopped, out of time (research.max_minutes), or a budget + */ + ended?: "stopped" | "timeout" | "budget" | null; + /** + * Revisions + */ + revisions?: number; + /** + * Notes + * + * notes made from the report + */ + notes?: Array; + /** + * Created At + */ + created_at?: string | null; + /** + * Started At + */ + started_at?: string | null; + /** + * Finished At + */ + finished_at?: string | null; + [key: string]: unknown; +}; + +/** + * ResearchEstimate + */ +export type ResearchEstimate = { + /** + * Usd + * + * what it will likely cost, from the prices in Settings; null without a price + */ + usd?: number | null; + /** + * Tokens + */ + tokens?: number | null; + /** + * Tool Calls + */ + tool_calls?: number | null; + /** + * Basis + * + * what the estimate is drawn from + */ + basis?: string | null; + /** + * Estimate + */ + estimate?: boolean; + [key: string]: unknown; +}; + +/** + * ResearchFinding + */ +export type ResearchFinding = { + /** + * Question + */ + question: string; + /** + * Tool Calls + */ + tool_calls?: number; + [key: string]: unknown; +}; + +/** + * ResearchHeld + */ +export type ResearchHeld = { + /** + * Why + */ + why: string; + /** + * Budget + * + * the budget it waits for (all, space:) + */ + budget?: string | null; + [key: string]: unknown; +}; + +/** + * ResearchNote + */ +export type ResearchNote = { + /** + * Namespace + * + * where the note goes; needed when the research looked in several + */ + namespace?: string | null; +}; + +/** + * ResearchNoteMade + */ +export type ResearchNoteMade = { + /** + * Page + */ + page: number; + /** + * Namespace + */ + namespace: string; + [key: string]: unknown; +}; + +/** + * ResearchPlan + */ +export type ResearchPlan = { + /** + * Title + */ + title: string; + /** + * Goal + */ + goal: string; + /** + * Sub Questions + * + * what it answers, one look-up at a time, in order + */ + sub_questions: Array; + /** + * Sources + * + * where it looks: the archive, the graph, notes, the web, the code sandbox + */ + sources: Array<"archive" | "graph" | "notes" | "web" | "code">; + /** + * Max Tool Calls + * + * the most tool calls the run makes in all (research.max_tool_calls at most) + */ + max_tool_calls: number; + /** + * Report Outline + * + * the report's sections + */ + report_outline?: Array; + [key: string]: unknown; +}; + +/** + * ResearchPodcast + */ +export type ResearchPodcast = { + /** + * Prompt + * + * an angle; else the research goal + */ + prompt?: string | null; + /** + * Length + * + * minutes (up to podcasts.max_minutes) + */ + length?: number; + /** + * Style + */ + style?: "deep-dive" | "recap" | "debate" | "beginner"; + /** + * Title + * + * else the report's title + */ + title?: string | null; +}; + +/** + * ResearchProgress + */ +export type ResearchProgress = { + /** + * Sub Questions + */ + sub_questions?: number; + /** + * Done + * + * sub-questions answered + */ + done?: number; + /** + * Sources + * + * sources found so far + */ + sources?: number; + /** + * Tool Calls + */ + tool_calls?: number; + /** + * Cost Usd + * + * spent so far (the activity ledger rows of research:) + */ + cost_usd?: number; + /** + * Tokens + */ + tokens?: number; + /** + * Cost Estimated + */ + cost_estimated?: boolean; + /** + * Minutes + * + * how long it has run + */ + minutes?: number; + [key: string]: unknown; +}; + +/** + * ResearchQuestion + */ +export type ResearchQuestion = { + /** + * Question + */ + question: string; + /** + * Answers + * + * suggested answers to pick from + */ + answers?: Array; + [key: string]: unknown; +}; + +/** + * ResearchRevise + */ +export type ResearchRevise = { + /** + * Answers + * + * the follow-up questions' answers, by question + */ + answers?: { + [key: string]: string; + }; + /** + * Plan + * + * the plan as edited: any of title, sub_questions, sources, max_tool_calls, report_outline + */ + plan?: { + [key: string]: unknown; + } | null; +}; + +/** + * ResearchStart + */ +export type ResearchStart = { + /** + * Goal + * + * what to research + */ + goal: string; + /** + * Tools + * + * what it may use (GET /chats/tools); null: the conversation's choice + */ + tools?: Array | null; + /** + * Max Tool Calls + * + * the most tool calls the plan may make (research.max_tool_calls at most) + */ + max_tool_calls?: number | null; + /** + * what it draws on, instead of the conversation's scope + */ + scope?: ChatScope | null; + /** + * Review + * + * false: start the run at once, without the plan waiting for review (signed in; an API key or an app only with research.agents_skip_review) + */ + review?: boolean; +}; + +/** + * ResearchStarted + */ +export type ResearchStarted = { + research: Research; + /** + * Question + * + * the question saved in the conversation + */ + question: number; + /** + * Message + * + * the answer that shows the plan + */ + message: number; + /** + * Text + * + * the plan as the conversation shows it + */ + text: string; + meta?: MessageRecord | null; + [key: string]: unknown; +}; + +/** + * ResearchStop + */ +export type ResearchStop = { + /** + * Ok + */ + ok?: boolean; + /** + * Stopping + * + * it stops after the step it's on (or now, when it hadn't started or waited on a budget) + */ + stopping: boolean; + [key: string]: unknown; +}; + /** * ResetPasswordRequest */ @@ -41159,6 +41613,258 @@ export type DecideApprovalResponses = { export type DecideApprovalResponse = DecideApprovalResponses[keyof DecideApprovalResponses]; +export type StartResearchData = { + body: ResearchStart; + path: { + /** + * Cid + */ + cid: number; + }; + query?: never; + url: "/api/v1/chats/{cid}/research"; +}; + +export type StartResearchErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type StartResearchError = StartResearchErrors[keyof StartResearchErrors]; + +export type StartResearchResponses = { + /** + * Successful Response + */ + 200: ResearchStarted; +}; + +export type StartResearchResponse = StartResearchResponses[keyof StartResearchResponses]; + +export type ListResearchData = { + body?: never; + path?: never; + query?: { + /** + * Chat Id + */ + chat_id?: number | null; + /** + * Limit + */ + limit?: number; + }; + url: "/api/v1/research"; +}; + +export type ListResearchErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type ListResearchError = ListResearchErrors[keyof ListResearchErrors]; + +export type ListResearchResponses = { + /** + * Response Research-List Research + * + * Successful Response + */ + 200: Array; +}; + +export type ListResearchResponse = ListResearchResponses[keyof ListResearchResponses]; + +export type GetResearchData = { + body?: never; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/research/{rid}"; +}; + +export type GetResearchErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type GetResearchError = GetResearchErrors[keyof GetResearchErrors]; + +export type GetResearchResponses = { + /** + * Successful Response + */ + 200: Research; +}; + +export type GetResearchResponse = GetResearchResponses[keyof GetResearchResponses]; + +export type ReviseResearchData = { + body: ResearchRevise; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/research/{rid}/revise"; +}; + +export type ReviseResearchErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type ReviseResearchError = ReviseResearchErrors[keyof ReviseResearchErrors]; + +export type ReviseResearchResponses = { + /** + * Successful Response + */ + 200: Research; +}; + +export type ReviseResearchResponse = ReviseResearchResponses[keyof ReviseResearchResponses]; + +export type StopResearchData = { + body?: never; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/research/{rid}/stop"; +}; + +export type StopResearchErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type StopResearchError = StopResearchErrors[keyof StopResearchErrors]; + +export type StopResearchResponses = { + /** + * Successful Response + */ + 200: ResearchStop; +}; + +export type StopResearchResponse = StopResearchResponses[keyof StopResearchResponses]; + +export type ResumeResearchData = { + body?: never; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/research/{rid}/resume"; +}; + +export type ResumeResearchErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type ResumeResearchError = ResumeResearchErrors[keyof ResumeResearchErrors]; + +export type ResumeResearchResponses = { + /** + * Successful Response + */ + 200: Research; +}; + +export type ResumeResearchResponse = ResumeResearchResponses[keyof ResumeResearchResponses]; + +export type ResearchNoteData = { + /** + * Body + */ + body?: ResearchNote | null; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/research/{rid}/note"; +}; + +export type ResearchNoteErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type ResearchNoteError = ResearchNoteErrors[keyof ResearchNoteErrors]; + +export type ResearchNoteResponses = { + /** + * Successful Response + */ + 200: ResearchNoteMade; +}; + +export type ResearchNoteResponse = ResearchNoteResponses[keyof ResearchNoteResponses]; + +export type ResearchPodcastData = { + /** + * Body + */ + body?: ResearchPodcast | null; + path: { + /** + * Rid + */ + rid: number; + }; + query?: never; + url: "/api/v1/research/{rid}/podcast"; +}; + +export type ResearchPodcastErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type ResearchPodcastError = ResearchPodcastErrors[keyof ResearchPodcastErrors]; + +export type ResearchPodcastResponses = { + /** + * Successful Response + */ + 202: PodcastStarted; +}; + +export type ResearchPodcastResponse = ResearchPodcastResponses[keyof ResearchPodcastResponses]; + export type ListSessionsData = { body?: never; path?: never; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index e33a161c2..418576b6f 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -26101,7 +26101,7 @@ "chats" ], "summary": "Send Message", - "description": "Ask a question, optionally from a page (`context`: the page, its text and any highlighted part, which the model\nreads with the question). Streams events: step, approval, notice, passages, token (answer text), error, stopped (POST\n/chats/{cid}/stop: what came before is saved, marked stopped), meta (the answer's record, kept with it: what produced\nit), done (the saved message id).\n\nA conversation is a tree of versions (GET /chats/{cid}: `tree`, and each message's `versions`); the question goes\non from `parent` when given (any message of the conversation), else from the version showing (`head`). With\n`regenerate` (an answer's id) its question is answered again, as a new version beside that answer (`content` isn't\nneeded; `model` may pick another). With `edit`, one of your earlier questions is edited: this question becomes a new\nversion beside it and is answered, and the old one stays with what followed it (404 if it isn't a question in this\nconversation). Either keeps the page the question was asked from (where, and the highlighted text; not the page's\ntext, which isn't kept) unless `context` is given, and streams `branch` first ({message, parent, versions}: the\nnew version, the message it follows and its versions); so does a question beside others under `parent`.\n\nWhat the answer may use: `tools` (tool groups and tools, GET /chats/tools; [] for none, when it answers from a\nsearch) and `max_tool_calls` (at most that many tool calls, in as many rounds as it takes), else the conversation's\nchoice (PATCH /chats/{cid}); and `scope`, what this answer alone draws on (the conversation keeps its own). Its\nrecord keeps what it was given and used.\n\n`/clear` makes the model start afresh (the messages stay to read) and `/compact [what to keep]` folds the\nconversation so far into its summary now; each streams cleared or compacted, then its reply as an answer.\n\nWith `browser` (the session of a browser's Lens side panel, POST /browser/sessions), the assistant can use that\nbrowser while the side panel is connected: open pages, read them, click, type and download files into the\nconversation (docs/browser-assistant.md).", + "description": "Ask a question, optionally from a page (`context`: the page, its text and any highlighted part, which the model\nreads with the question). Streams events: step, approval, notice, passages, token (answer text), error, stopped (POST\n/chats/{cid}/stop: what came before is saved, marked stopped), meta (the answer's record, kept with it: what produced\nit), done (the saved message id).\n\nA conversation is a tree of versions (GET /chats/{cid}: `tree`, and each message's `versions`); the question goes\non from `parent` when given (any message of the conversation), else from the version showing (`head`). With\n`regenerate` (an answer's id) its question is answered again, as a new version beside that answer (`content` isn't\nneeded; `model` may pick another). With `edit`, one of your earlier questions is edited: this question becomes a new\nversion beside it and is answered, and the old one stays with what followed it (404 if it isn't a question in this\nconversation). Either keeps the page the question was asked from (where, and the highlighted text; not the page's\ntext, which isn't kept) unless `context` is given, and streams `branch` first ({message, parent, versions}: the\nnew version, the message it follows and its versions); so does a question beside others under `parent`.\n\nWhat the answer may use: `tools` (tool groups and tools, GET /chats/tools; [] for none, when it answers from a\nsearch) and `max_tool_calls` (at most that many tool calls, in as many rounds as it takes), else the conversation's\nchoice (PATCH /chats/{cid}); and `scope`, what this answer alone draws on (the conversation keeps its own). Its\nrecord keeps what it was given and used.\n\n`/clear` makes the model start afresh (the messages stay to read) and `/compact [what to keep]` folds the\nconversation so far into its summary now; each streams cleared or compacted, then its reply as an answer.\n\nWith `research` (or /research ) the content is a research goal: the answer is a plan to approve (streamed as\n`research`, then `approval`), which runs in the background once approved (POST /chats/{cid}/research, GET\n/research/{id}).\n\nWith `browser` (the session of a browser's Lens side panel, POST /browser/sessions), the assistant can use that\nbrowser while the side panel is connected: open pages, read them, click, type and download files into the\nconversation (docs/browser-assistant.md).", "operationId": "send_message", "parameters": [ { @@ -26499,6 +26499,425 @@ } } }, + "/api/v1/chats/{cid}/research": { + "post": { + "tags": [ + "research" + ], + "summary": "Start Research", + "description": "Start research on a goal in a conversation: the goal is saved as a question and one model call plans it (up to\nthree follow-up questions with answers to pick, the sub-questions, where it looks, the tool calls it may make, the\nreport's outline, and an estimate of the cost). The plan waits as an approval (POST /approvals/{id}: approve it as\nit is, or with `params` its edits; or decline), or revise it (POST /research/{id}/revise). With `review` false it\nstarts at once: your own pick when signed in; an API key or an app needs research.agents_skip_review. The same as a\nquestion with `research: true`, or /research .", + "operationId": "start_research", + "parameters": [ + { + "name": "cid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Cid" + } + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ResearchStart" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ResearchStarted" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research": { + "get": { + "tags": [ + "research" + ], + "summary": "List Research", + "description": "Your research, newest first; with `chat_id`, that conversation's (to show each plan's or run's card).", + "operationId": "list_research", + "parameters": [ + { + "name": "chat_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Chat Id" + } + }, + { + "name": "limit", + "in": "query", + "required": false, + "schema": { + "type": "integer", + "default": 50, + "title": "Limit" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "$ref": "#/components/schemas/Research" + }, + "title": "Response Research-List Research" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research/{rid}": { + "get": { + "tags": [ + "research" + ], + "summary": "Get Research", + "description": "A research run: its plan, status and stage (gather, write, check, done), progress (sub-questions done, sources\nfound, tool calls and cost so far, from the activity ledger) and, once it's done, the report's message\n(`answer`, in the conversation). What it waits for when a budget holds it (`held`), and why it ended early.", + "operationId": "get_research", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Research" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research/{rid}/revise": { + "post": { + "tags": [ + "research" + ], + "summary": "Revise Research", + "description": "Send your answers to the follow-up questions and your edits to the plan back for one more planning call; the\nplan and its approval change in place. Only while the plan waits for approval.", + "operationId": "revise_research", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ResearchRevise" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Research" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research/{rid}/stop": { + "post": { + "tags": [ + "research" + ], + "summary": "Stop Research", + "description": "Stop a run: it stops after the step it's on and keeps what it found, saved in the conversation as its answer,\nmarked stopped (one that hasn't started, or waits on a budget, stops now). Works whichever process runs it.", + "operationId": "stop_research", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ResearchStop" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research/{rid}/resume": { + "post": { + "tags": [ + "research" + ], + "summary": "Resume Research", + "description": "Let a run that a budget holds go on anyway (your pick, as \"run it once\" is for a held routine run).", + "operationId": "resume_research", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Research" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research/{rid}/note": { + "post": { + "tags": [ + "research" + ], + "summary": "Research Note", + "description": "Save the report as a note: in the namespace it looked in (`namespace` when there are several), its archive\ncitations as links to the assets they cite, web citations as links, and a link back to the conversation (shown to\nyou alone). Needs editor access there.", + "operationId": "research_note", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "requestBody": { + "content": { + "application/json": { + "schema": { + "anyOf": [ + { + "$ref": "#/components/schemas/ResearchNote" + }, + { + "type": "null" + } + ], + "title": "Body" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ResearchNoteMade" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/research/{rid}/podcast": { + "post": { + "tags": [ + "research" + ], + "summary": "Research Podcast", + "description": "Make a podcast episode from the assets the report cites (POST /podcasts with them picked): about the research\ngoal unless `prompt` says otherwise, titled after the report.", + "operationId": "research_podcast", + "parameters": [ + { + "name": "rid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Rid" + } + } + ], + "requestBody": { + "content": { + "application/json": { + "schema": { + "anyOf": [ + { + "$ref": "#/components/schemas/ResearchPodcast" + }, + { + "type": "null" + } + ], + "title": "Body" + } + } + } + }, + "responses": { + "202": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/PodcastStarted" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/browser/sessions": { "get": { "tags": [ @@ -37811,6 +38230,22 @@ } ], "description": "what produced it; none on messages from before it was recorded" + }, + "kind": { + "anyOf": [ + { + "type": "string", + "enum": [ + "research", + "research_plan" + ] + }, + { + "type": "null" + } + ], + "title": "Kind", + "description": "research: a deep research report; research_plan: the plan it waits on (GET /research/{meta.research})" } }, "additionalProperties": true, @@ -51686,6 +52121,18 @@ } ], "description": "what this answer draws on, instead of the conversation's scope (which stays as it is); {} for everything you can read" + }, + "research": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Research", + "description": "the content is a research goal: the reply is a plan to approve (streams `research`, then `approval`), as /research and POST /chats/{cid}/research do" } }, "additionalProperties": false, @@ -51849,6 +52296,22 @@ ], "description": "what produced it; none on messages from before it was recorded" }, + "kind": { + "anyOf": [ + { + "type": "string", + "enum": [ + "research", + "research_plan" + ] + }, + { + "type": "null" + } + ], + "title": "Kind", + "description": "research: a deep research report; research_plan: the plan it waits on (GET /research/{meta.research})" + }, "ledger": { "items": { "$ref": "#/components/schemas/ActivityEntry" @@ -62011,6 +62474,724 @@ "type": "object", "title": "ReprocessRequest" }, + "Research": { + "properties": { + "id": { + "type": "integer", + "title": "Id" + }, + "chat": { + "type": "integer", + "title": "Chat" + }, + "message": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Message", + "description": "the question that asked for it" + }, + "plan_message": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Plan Message", + "description": "the answer that shows the plan" + }, + "answer": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Answer", + "description": "the report, once saved in the conversation" + }, + "approval": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Approval", + "description": "the plan's approval (POST /approvals/{id}: approve, with `params` its edits, or decline)" + }, + "status": { + "type": "string", + "enum": [ + "planned", + "declined", + "queued", + "running", + "held", + "done", + "stopped", + "failed" + ], + "title": "Status", + "description": "planned (waiting for approval), declined, queued (approved, waiting its turn), running, held (by a budget), done, stopped or failed" + }, + "stage": { + "anyOf": [ + { + "type": "string", + "enum": [ + "gather", + "write", + "check", + "done" + ] + }, + { + "type": "null" + } + ], + "title": "Stage" + }, + "goal": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Goal" + }, + "plan": { + "$ref": "#/components/schemas/ResearchPlan" + }, + "questions": { + "items": { + "$ref": "#/components/schemas/ResearchQuestion" + }, + "type": "array", + "title": "Questions", + "description": "follow-up questions; answer them with revise" + }, + "answers": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Answers", + "description": "the answers given to them" + }, + "scope": { + "additionalProperties": true, + "type": "object", + "title": "Scope" + }, + "estimate": { + "anyOf": [ + { + "$ref": "#/components/schemas/ResearchEstimate" + }, + { + "type": "null" + } + ] + }, + "progress": { + "$ref": "#/components/schemas/ResearchProgress" + }, + "findings": { + "items": { + "$ref": "#/components/schemas/ResearchFinding" + }, + "type": "array", + "title": "Findings", + "description": "the sub-questions answered so far" + }, + "held": { + "anyOf": [ + { + "$ref": "#/components/schemas/ResearchHeld" + }, + { + "type": "null" + } + ] + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + }, + "ended": { + "anyOf": [ + { + "type": "string", + "enum": [ + "stopped", + "timeout", + "budget" + ] + }, + { + "type": "null" + } + ], + "title": "Ended", + "description": "why it ended early: stopped, out of time (research.max_minutes), or a budget" + }, + "revisions": { + "type": "integer", + "title": "Revisions", + "default": 0 + }, + "notes": { + "items": { + "type": "integer" + }, + "type": "array", + "title": "Notes", + "description": "notes made from the report" + }, + "created_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Created At" + }, + "started_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Started At" + }, + "finished_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Finished At" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "chat", + "status", + "plan", + "progress" + ], + "title": "Research" + }, + "ResearchEstimate": { + "properties": { + "usd": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Usd", + "description": "what it will likely cost, from the prices in Settings; null without a price" + }, + "tokens": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Tokens" + }, + "tool_calls": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Tool Calls" + }, + "basis": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Basis", + "description": "what the estimate is drawn from" + }, + "estimate": { + "type": "boolean", + "title": "Estimate", + "default": true + } + }, + "additionalProperties": true, + "type": "object", + "title": "ResearchEstimate" + }, + "ResearchFinding": { + "properties": { + "question": { + "type": "string", + "title": "Question" + }, + "tool_calls": { + "type": "integer", + "title": "Tool Calls", + "default": 0 + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "question" + ], + "title": "ResearchFinding" + }, + "ResearchHeld": { + "properties": { + "why": { + "type": "string", + "title": "Why" + }, + "budget": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Budget", + "description": "the budget it waits for (all, space:)" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "why" + ], + "title": "ResearchHeld" + }, + "ResearchNote": { + "properties": { + "namespace": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Namespace", + "description": "where the note goes; needed when the research looked in several" + } + }, + "additionalProperties": false, + "type": "object", + "title": "ResearchNote" + }, + "ResearchNoteMade": { + "properties": { + "page": { + "type": "integer", + "title": "Page" + }, + "namespace": { + "type": "string", + "title": "Namespace" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "page", + "namespace" + ], + "title": "ResearchNoteMade" + }, + "ResearchPlan": { + "properties": { + "title": { + "type": "string", + "title": "Title" + }, + "goal": { + "type": "string", + "title": "Goal" + }, + "sub_questions": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Sub Questions", + "description": "what it answers, one look-up at a time, in order" + }, + "sources": { + "items": { + "type": "string", + "enum": [ + "archive", + "graph", + "notes", + "web", + "code" + ] + }, + "type": "array", + "title": "Sources", + "description": "where it looks: the archive, the graph, notes, the web, the code sandbox" + }, + "max_tool_calls": { + "type": "integer", + "title": "Max Tool Calls", + "description": "the most tool calls the run makes in all (research.max_tool_calls at most)" + }, + "report_outline": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Report Outline", + "description": "the report's sections" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "title", + "goal", + "sub_questions", + "sources", + "max_tool_calls" + ], + "title": "ResearchPlan" + }, + "ResearchPodcast": { + "properties": { + "prompt": { + "anyOf": [ + { + "type": "string", + "maxLength": 2000 + }, + { + "type": "null" + } + ], + "title": "Prompt", + "description": "an angle; else the research goal" + }, + "length": { + "type": "integer", + "maximum": 120.0, + "minimum": 1.0, + "title": "Length", + "description": "minutes (up to podcasts.max_minutes)", + "default": 10 + }, + "style": { + "type": "string", + "enum": [ + "deep-dive", + "recap", + "debate", + "beginner" + ], + "title": "Style", + "default": "deep-dive" + }, + "title": { + "anyOf": [ + { + "type": "string", + "maxLength": 200 + }, + { + "type": "null" + } + ], + "title": "Title", + "description": "else the report's title" + } + }, + "additionalProperties": false, + "type": "object", + "title": "ResearchPodcast" + }, + "ResearchProgress": { + "properties": { + "sub_questions": { + "type": "integer", + "title": "Sub Questions", + "default": 0 + }, + "done": { + "type": "integer", + "title": "Done", + "description": "sub-questions answered", + "default": 0 + }, + "sources": { + "type": "integer", + "title": "Sources", + "description": "sources found so far", + "default": 0 + }, + "tool_calls": { + "type": "integer", + "title": "Tool Calls", + "default": 0 + }, + "cost_usd": { + "type": "number", + "title": "Cost Usd", + "description": "spent so far (the activity ledger rows of research:)", + "default": 0.0 + }, + "tokens": { + "type": "integer", + "title": "Tokens", + "default": 0 + }, + "cost_estimated": { + "type": "boolean", + "title": "Cost Estimated", + "default": false + }, + "minutes": { + "type": "number", + "title": "Minutes", + "description": "how long it has run", + "default": 0.0 + } + }, + "additionalProperties": true, + "type": "object", + "title": "ResearchProgress" + }, + "ResearchQuestion": { + "properties": { + "question": { + "type": "string", + "title": "Question" + }, + "answers": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Answers", + "description": "suggested answers to pick from" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "question" + ], + "title": "ResearchQuestion" + }, + "ResearchRevise": { + "properties": { + "answers": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Answers", + "description": "the follow-up questions' answers, by question" + }, + "plan": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Plan", + "description": "the plan as edited: any of title, sub_questions, sources, max_tool_calls, report_outline" + } + }, + "additionalProperties": false, + "type": "object", + "title": "ResearchRevise" + }, + "ResearchStart": { + "properties": { + "goal": { + "type": "string", + "maxLength": 4000, + "minLength": 1, + "title": "Goal", + "description": "what to research" + }, + "tools": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 100 + }, + { + "type": "null" + } + ], + "title": "Tools", + "description": "what it may use (GET /chats/tools); null: the conversation's choice" + }, + "max_tool_calls": { + "anyOf": [ + { + "type": "integer", + "minimum": 1.0 + }, + { + "type": "null" + } + ], + "title": "Max Tool Calls", + "description": "the most tool calls the plan may make (research.max_tool_calls at most)" + }, + "scope": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatScope" + }, + { + "type": "null" + } + ], + "description": "what it draws on, instead of the conversation's scope" + }, + "review": { + "type": "boolean", + "title": "Review", + "description": "false: start the run at once, without the plan waiting for review (signed in; an API key or an app only with research.agents_skip_review)", + "default": true + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "goal" + ], + "title": "ResearchStart" + }, + "ResearchStarted": { + "properties": { + "research": { + "$ref": "#/components/schemas/Research" + }, + "question": { + "type": "integer", + "title": "Question", + "description": "the question saved in the conversation" + }, + "message": { + "type": "integer", + "title": "Message", + "description": "the answer that shows the plan" + }, + "text": { + "type": "string", + "title": "Text", + "description": "the plan as the conversation shows it" + }, + "meta": { + "anyOf": [ + { + "$ref": "#/components/schemas/MessageRecord" + }, + { + "type": "null" + } + ] + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "research", + "question", + "message", + "text" + ], + "title": "ResearchStarted" + }, + "ResearchStop": { + "properties": { + "ok": { + "type": "boolean", + "title": "Ok", + "default": true + }, + "stopping": { + "type": "boolean", + "title": "Stopping", + "description": "it stops after the step it's on (or now, when it hadn't started or waited on a budget)" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "stopping" + ], + "title": "ResearchStop" + }, "ResetPasswordRequest": { "properties": { "token": { diff --git a/threat-model.json b/threat-model.json index a2fd3838b..4a3ad1edc 100644 --- a/threat-model.json +++ b/threat-model.json @@ -683,6 +683,15 @@ "provenance": "documented", "source": "docs/assistant.md \"Code sandbox\"" }, + { + "entry_point": "deep research", + "parameter": "/research , research: true, POST /chats/{cid}/research, /research/{id} (revise, stop, resume, note, podcast), start_research (assistant and MCP), research_status (MCP): a goal, answers to follow-up questions, plan edits; research.* settings (admins)", + "attacker_controllable": "yes", + "component": "assistant", + "caller_must_enforce": "none — the plan waits for the person's own approval (P-RESEARCH); the run uses only its sources' read tools, as the person, within its tool calls, minutes and budgets; its web searches and page reads don't ask again, bounded by the plan and the ledger; one run per person at a time; the report's claims are checked against what they cite", + "provenance": "documented", + "source": "docs/assistant.md \"Deep research\"" + }, { "entry_point": "microsandbox program requests", "parameter": "ask/NAME.want in /data/.lens/programs: a program's name", @@ -1169,7 +1178,7 @@ "property": "P-INJECTION-EGRESS", "violation_symptom": "info-leak", "severity_tier": "correctness", - "conditions": "Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no bridge.users, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own read web tools, a web search or page read neither the person nor a tool gave it, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is VALID-HARDENING (Q26). An automation's model and agent actions (ask_model, call_tool, ask_agent) run with nobody reading along: the activity and earlier steps reach the model marked as data, every change they would make (notes included) is an approval of the automation's person, and anything that could carry data out of Lens asks first whatever the run read (docs/automations.md \"The model and the agent\"). A saved variable a run set from outside text stays marked as data in later runs' prompts, and an action whose params read it counts as reading outside text (docs/automations.md \"Saved variables\").", + "conditions": "Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no bridge.users, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own read web tools, a web search or page read neither the person nor a tool gave it, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is VALID-HARDENING (Q26). An automation's model and agent actions (ask_model, call_tool, ask_agent) run with nobody reading along: the activity and earlier steps reach the model marked as data, every change they would make (notes included) is an approval of the automation's person, and anything that could carry data out of Lens asks first whatever the run read (docs/automations.md \"The model and the agent\"). A saved variable a run set from outside text stays marked as data in later runs' prompts, and an action whose params read it counts as reading outside text (docs/automations.md \"Saved variables\"). Within a research run the person approved, the plan's web searches and page reads, and code with the network when the plan names the code sandbox, don't ask again (P-RESEARCH).", "provenance": "documented", "source": "docs/assistant.md \"it read out of Lens waits for your approval\"" }, @@ -1373,6 +1382,14 @@ "provenance": "documented", "source": "docs/assistant.md \"Code sandbox\"" }, + { + "property": "P-RESEARCH", + "violation_symptom": "integrity-bypass", + "severity_tier": "correctness", + "conditions": "a research plan runs only once its person approves it, signed in or with a yes in their chat room from someone in bridge.users; an API key, an app or an agent only declines it unless research.agents_skip_review; the run acts as the person with their access at run time, offered only its sources' read tools, within the plan's tool calls, research.max_minutes and the budgets of its namespaces and all, checked before each stage; its web searches and page reads (and code with the network, when the plan names the sandbox) don't ask again; one run per person at a time; one process at a time; every call in the ledger under research:", + "provenance": "documented", + "source": "docs/assistant.md \"Deep research\"" + }, { "property": "MX-AUTHZ-LIST-CAPS", "violation_symptom": "x-cap-exceeded", diff --git a/threat-model.md b/threat-model.md index 12d47ce27..7e194247d 100644 --- a/threat-model.md +++ b/threat-model.md @@ -319,6 +319,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `/api/v1/automations` (save, try, run now, decide, draft from a sentence), the MCP and assistant automation tools; the activities an automation runs on | triggers, `where` and If match trees, the event workflow graph (action params are Jinja templates), `policy`, `limits`, an activity id; the text of the activities themselves | yes: any editor, for automations in namespaces they edit; indirectly through activity text (mail, titles, chat) that reaches action params | callback/code (declarative), data, x-prompt-injection | none; it runs as its maker with their permissions at run time (P-AUTOMATION-ACTOR), actions reaching outside Lens ask first, outside text marks intents untrusted and breaks up links (P-INJECTION-EGRESS), and chains and bursts are bounded (P-AUTOMATION-LOOP) | *(documented, docs/automations.md "Running automations")* | | Assistant code sandbox tools (`run_python`, `run_bash`, `sandbox_files`, `share_sandbox_file`, `import_sandbox_file`) and the `sandbox.*` settings | code and scripts the model writes, attachment ids, a path in the sandbox; the gateway address, image and hosts it may reach (admins) | yes: the people `sandbox.who` names, and indirectly through content the model reads | callback/code, data, x-prompt-injection | none; off by default; the code runs in the conversation's own OpenShell sandbox, which holds none of Lens's data or keys and reaches no network unless `sandbox.network` allows it (P-SANDBOX); with the network on, a run after outside text asks first (P-INJECTION-EGRESS); files go in only from the person's own attachments to that conversation, and come out only as an import the person approves or as a download for that person (`share_sandbox_file`) | *(documented, docs/assistant.md "Code sandbox")* | | `GET /api/v1/chats/{cid}/files/{fid}`: a file the code sandbox shared (`share_sandbox_file`) | a file id; the file's bytes, which code the model wrote produced | yes: the person whose conversation it is; indirectly through content the model reads | data, x-prompt-injection | none; served only to the conversation's person (404 for anyone else, or for an id of another conversation) as an `attachment` with `nosniff` and a `sandbox` content policy, never rendered by Lens; the id is random (96 bits) and checked against the conversation; kept encrypted (AES-256-GCM with a key derived from the server's secret, bound to the file and its conversation), at most 20 a conversation and 25 MB each, and deleted with the conversation or the person's account | *(documented, docs/assistant.md "Code sandbox")* | +| Deep research: `/research `, `research: true` on a question, `POST /chats/{cid}/research`, `/research/{id}` (revise, stop, resume, note, podcast), the assistant's and MCP's `start_research`, MCP's `research_status`, and the `research.*` settings | a goal, answers to follow-up questions, plan edits (sub-questions, sources, tool calls, outline); settings (admins) | yes: anyone who may use the assistant, in their own conversations; indirectly through what the run reads, through the model | data, x-prompt-injection, size | none; the plan waits for the person's own approval (P-RESEARCH); the run uses only the read tools of its plan's sources, as the person with their current access, within its tool calls, minutes and budgets; its web searches and page reads don't ask again, bounded by the plan and kept in the ledger; one run per person at a time; the report's claims are checked against what they cite and marked when unsupported | *(documented, docs/assistant.md "Deep research")* | | `ask/NAME.want` in a microsandbox's `/data/.lens/programs` (LibreOffice and Chromium on first use) | a program's name | writable by the Lens user inside the sandbox | x-env-config | the start script, as root, installs only `libreoffice` or `chromium`, each a fixed list of signed Debian packages, and answers in `got/`, which only root writes; it reads nothing from the file but its name, and only deletes any other name or link | *(documented, docs/microsandbox.md "LibreOffice and Chromium")* | **Stateful postconditions.** A refused or failed one-time secret check consumes nothing; a used one is deleted before @@ -540,7 +541,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-RETRIEVAL | Chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at `ai.max_steps` and `ai.max_transcript_reads`. A namespace's chat completions answer from that namespace alone, with only the read tools its owners offered. | info-leak (scope); hang (caps) | security-critical (scope); correctness-only (caps) | *(documented, docs/assistant.md "limited to what the person can read")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | | P-APPROVAL | Assistant tools that change data wait for the person's approval, except notes and an admin's setup conversation. Inviting people and changing their access always wait, even there, and the assistant cannot remove an account. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Approvals")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | | P-AUTOMATION-APPROVAL | An automation's action that asks first (effect `out`, or `lens` beyond the event's own object) waits as an approval of the automation's person, and nothing runs until they pick. Approving, editing or allowing a node from now on (`always`) needs them signed in to Lens, or (approving an action that stays inside Lens, effect `lens`) a yes in the private chat Lens told them in, from someone on that channel's `users` list with approving on, while it is the only thing waiting there or named by its number; an action that reaches outside Lens is approved on its card. API keys, apps, MCP and the assistant can only list and decline. A decision is claimed once (two at once can't both carry it out). At most 100 wait per automation and each tells the person once, so a flood of activities can't bury the inbox or their chat; undecided ones expire after `automations.approval_days` and nothing runs. | integrity-bypass | security-critical | *(documented, docs/automations.md "Approvals")* | Voided by: the person setting a node to `auto` (`always`, or the builder's toggle) — that node then runs without asking; an admin raising `automations.approval_days`. The summary and the message the person is told hold text the run read from outside Lens: links in it are broken up (P-INJECTION-EGRESS), and what runs is the intent the card shows. | -| P-INJECTION-EGRESS | Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no `bridge.users`, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own `read` web tools, a web search or page read neither the person nor a tool gave it, other servers' MCP tools, Python tools with the network, code in the code sandbox with its network on) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is `VALID-HARDENING` (Q26). An automation's model and agent actions (`ask_model`, `call_tool`, `ask_agent`) run with nobody reading along: the activity and earlier steps reach the model marked as data, every change they would make (notes included) is an approval of the automation's person, and anything that could carry data out of Lens asks first whatever the run read (docs/automations.md "The model and the agent"). A saved variable a run set from outside text stays marked as data in later runs' prompts, and an action whose params read it counts as reading outside text (docs/automations.md "Saved variables"). | info-leak | correctness-only | *(documented, docs/assistant.md "it read out of Lens waits for your approval")* | Voided by: `ai.untrusted_asks: false` — `domain/ai_tools.py:519`; `bridge.safe_links: false` — `domain/bridge.py:298`. Notes and memories are written at once: they stay in Lens. Hooks run as their owners set them (P-EXT-SCOPE). An address the read text wrote itself stays linkable in a room. The person's browser asks first as P-BROWSER-GUARD says. | +| P-INJECTION-EGRESS | Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no `bridge.users`, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own `read` web tools, a web search or page read neither the person nor a tool gave it, other servers' MCP tools, Python tools with the network, code in the code sandbox with its network on) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is `VALID-HARDENING` (Q26). An automation's model and agent actions (`ask_model`, `call_tool`, `ask_agent`) run with nobody reading along: the activity and earlier steps reach the model marked as data, every change they would make (notes included) is an approval of the automation's person, and anything that could carry data out of Lens asks first whatever the run read (docs/automations.md "The model and the agent"). A saved variable a run set from outside text stays marked as data in later runs' prompts, and an action whose params read it counts as reading outside text (docs/automations.md "Saved variables"). Within a research run the person approved, the plan's web searches and page reads, and code runs with the network when the plan names the code sandbox, don't ask again: approving the plan approved them (P-RESEARCH). | info-leak | correctness-only | *(documented, docs/assistant.md "it read out of Lens waits for your approval")* | Voided by: `ai.untrusted_asks: false` — `domain/ai_tools.py:519`; `bridge.safe_links: false` — `domain/bridge.py:298`. Notes and memories are written at once: they stay in Lens. Hooks run as their owners set them (P-EXT-SCOPE). An address the read text wrote itself stays linkable in a room. The person's browser asks first as P-BROWSER-GUARD says. | | P-EXT-SCOPE | Your own hooks run in your conversations; another person's hooks run only when an admin shared them. | integrity-bypass | security-critical | *(documented, docs/assistant.md "other people's run only when an admin shared them")* | Voided by: An admin sharing the extension — `domain/extensions.py:595-598`. | | P-NO-CODE | No non-admin can run code on the host. This covers owners, editors, viewers, API tokens, MCP agents, OAuth apps and model tool calls acting for one. Python tools are written by admins only and run only while an admin owns them. | integrity-bypass | security-critical | *(maintainer, 2026-10)* | Voided by: Making someone admin; authoring gate — `domain/extensions.py:129-130`; run gate — `domain/extensions.py:600-601`. `grep -rnE "eval\(\|exec\(\|pickle\|yaml\.load\(\|__import__" fastapi_backend/app` — one `exec`, inside the code-tool child. | | P-UPLOAD | Uploads accept only the types in `uploads.extensions` and are size-capped. Nothing uploaded is executed. Files are served as audio, video or a download, and names carry no folders. | bad-data-accepted | security-critical | *(documented, docs/configuration.md "Uploads accept only the types")* | Voided by: Admin-added types; check — `domain/uploads.py:157-158`. The `attachments` role takes any type (never executed). Bytes are not checked against the extension. | @@ -566,6 +567,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-BROWSER | The assistant uses a person's browser only from that person's own conversations and API calls, while their own Lens side panel is connected (seen in the last 20 seconds, not closed). Only that person sees the session's actions and answers them, once each. `ai.browser` off refuses new sessions and offers no browser tools. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Only you")* | Voided by: The side panel carries out what it's asked: which sites it acts on, and what it asks the person first, is the extension's (docs/browser-assistant.md); an agent holding the person's write token can ask too — `api/v1/routes/browser.py`. | | P-BROWSER-GUARD | The Lens extension's side panel reads and acts only on sites the person allowed in it (and the browser granted), and opens an address only on such a site. It asks the person before a click, submit or Enter that the model marks consequential or whose control's words or posting form look so, and, once the answer has read text from outside Lens (`ai.untrusted_asks`), before opening or downloading an address or typing text the person's own message doesn't contain. It never types into password, card or file fields, never reports their values, and shows each action as it happens. | integrity-bypass | security-critical | *(documented, docs/browser-assistant.md "Letting it use your browser")* | Voided by: The consequential check is the model's flag plus a word list (`extensions/browser/src/core/act.js`): a control with no telling words that a misled model doesn't mark runs without asking; synthetic events reach only the page's own handlers. Removing a site from the options stops the side panel acting there, but leaves the browser's access to it for the clipper. Lens sends the ask (`confirm`) and the extension honours it: an extension older than this doesn't ask. | | P-SANDBOX | Code the assistant runs (`run_python`, `run_bash`; `sandbox.enabled`, off by default) runs only for people `sandbox.who` allows (editors and admins, or admins; never viewers), in a sandbox of that conversation's own on the OpenShell gateway, never in Lens's processes. The sandbox gets none of Lens's data, keys or database address: only the code and the files a run copies in, which must be attached to that conversation by that person (25 MB each, 100 MB a run). Its policy allows no egress with `sandbox.network: off`, only pypi.org, files.pythonhosted.org and registry.npmjs.org on 443 with `packages`, and those and the names in `sandbox.hosts` on 80 and 443 with `hosts`; every endpoint is held to public addresses (no private, loopback, link-local or reserved range), so a listed name that resolves to one is refused. A file leaves it only as one of the person's held uploads (25 MB) placed in a namespace by an `import_files` approval, or as a download (`share_sandbox_file`, 25 MB, 20 a conversation) kept encrypted with the conversation, served only to its person as an attachment with `nosniff`, and deleted with the conversation or their account. A sandbox stops after `sandbox.idle_minutes` idle and is deleted after `sandbox.keep_hours` unused or with its conversation. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Code sandbox")* | Voided by: isolation inside the sandbox is OpenShell's (§1.9); `landlock.compatibility: best_effort` runs without Landlock on a kernel that lacks it — `domain/sandbox.py`; whoever the gateway lets in can run sandboxes of their own (in the `sandbox` profile, a holder of its client certificate); a file shared to download is readable by whoever holds `secret.key` or `ARCHIVE_SECRET_KEY` (Q14); what the code prints reaches the model as third-party text (N-PROMPT). | +| P-RESEARCH | A research plan (`domain/research.py`) runs only once the person who asked for it approves it: signed in, or with a "yes" in their chat room from someone in `bridge.users`; an API key, an app or an agent (MCP, the assistant) can only decline it, unless an admin turns on `research.agents_skip_review`. The run acts as that person with their access at run time and stops if their account is disabled. It is offered only the read tools of the sources its plan names (never a tool that changes anything), makes at most the plan's tool calls (`research.max_tool_calls`, 40 by default), stops looking things up after `research.max_minutes` and writes up what it found, and checks the budgets of its namespaces and `all` before each stage, waiting or stopping as they say. Its web searches and page reads, and code runs with the network when the plan names the code sandbox, don't ask again after outside text, since the plan approved them. A person has one run going at a time. One process carries a run out at a time (a claim on `research_run:` with a heartbeat every 10 seconds, taken over only after two minutes of silence). Every call it makes is in the activity ledger for the conversation and `research:`. | integrity-bypass | correctness-only | *(documented, docs/assistant.md "Deep research")* | Voided by: `research.agents_skip_review: true` lets an agent's plan run without review — `domain/research.py` `agents_may_approve`; a run's web queries can carry archive text it read, within its plan's tool calls (the misled model of N-PROMPT); a "yes" in a room is the room's people's (N-BRIDGE). | ### Worked routing examples diff --git a/threat-model.yaml b/threat-model.yaml index 334f0cb35..92c845e46 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -628,6 +628,15 @@ entry_points: control_kinds: [data, x-prompt-injection] caller_must_enforce: "none — served only to the conversation's person (404 otherwise, or for another conversation's id) as an attachment with nosniff and a sandbox content policy, never rendered; random 96-bit id checked against the conversation; kept AES-256-GCM encrypted with a key derived from the server's secret, bound to the file and conversation; at most 20 a conversation, 25 MB each; deleted with the conversation or the account" provenance: {kind: documented, source: 'docs/assistant.md "Code sandbox"'} + - id: "deep research" + component: assistant + parameters: + - name: "/research , research: true, POST /chats/{cid}/research, /research/{id} (revise, stop, resume, note, podcast), start_research (assistant and MCP), research_status (MCP): a goal, answers to follow-up questions, plan edits; research.* settings (admins)" + attacker_controllable: true + x-condition: "anyone who may use the assistant, in their own conversations; indirectly through what the run reads, through the model" + control_kinds: [data, x-prompt-injection, size] + caller_must_enforce: "none — the plan waits for the person's own approval (P-RESEARCH); the run uses only its sources' read tools, as the person, within its tool calls, minutes and budgets; its web searches and page reads don't ask again, bounded by the plan and the ledger; one run per person at a time; the report's claims are checked against what they cite" + provenance: {kind: documented, source: 'docs/assistant.md "Deep research"'} - id: "microsandbox program requests" component: authz parameters: @@ -1618,7 +1627,7 @@ properties_claimed: kind: confidentiality components: [assistant] tier: correctness-only - conditions: "Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no bridge.users, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own read web tools, a web search or page read neither the person nor a tool gave it, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is VALID-HARDENING (Q26). An automation's model and agent actions (ask_model, call_tool, ask_agent) run with nobody reading along: the activity and earlier steps reach the model marked as data, every change they would make (notes included) is an approval of the automation's person, and anything that could carry data out of Lens asks first whatever the run read (docs/automations.md \"The model and the agent\"). A saved variable a run set from outside text stays marked as data in later runs' prompts, and an action whose params read it counts as reading outside text (docs/automations.md \"Saved variables\")." + conditions: "Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no bridge.users, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own read web tools, a web search or page read neither the person nor a tool gave it, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is VALID-HARDENING (Q26). An automation's model and agent actions (ask_model, call_tool, ask_agent) run with nobody reading along: the activity and earlier steps reach the model marked as data, every change they would make (notes included) is an approval of the automation's person, and anything that could carry data out of Lens asks first whatever the run read (docs/automations.md \"The model and the agent\"). A saved variable a run set from outside text stays marked as data in later runs' prompts, and an action whose params read it counts as reading outside text (docs/automations.md \"Saved variables\"). Within a research run the person approved, the plan's web searches and page reads, and code with the network when the plan names the code sandbox, don't ask again (P-RESEARCH)." violation_symptoms: [info-leak] x-voided-by: "ai.untrusted_asks: false (domain/ai_tools.py:519); bridge.safe_links: false (domain/bridge.py:298); notes and memories are written at once; hooks run as their owners set them (P-EXT-SCOPE); an address the read text wrote itself stays linkable in a room; the person's browser asks first as P-BROWSER-GUARD says" provenance: {kind: documented, source: 'docs/assistant.md "it read out of Lens waits for your approval"'} @@ -1825,6 +1834,14 @@ properties_claimed: violation_symptoms: [integrity-bypass] x-voided-by: "isolation inside the sandbox is OpenShell's; landlock best_effort runs without Landlock where the kernel lacks it (domain/sandbox.py); whoever the gateway lets in (in the sandbox profile, a holder of its client certificate) can run sandboxes of their own; what the code prints is third-party text (N-PROMPT)" provenance: {kind: documented, source: 'docs/assistant.md "Code sandbox"'} + - id: P-RESEARCH + kind: integrity + components: [assistant] + tier: correctness-only + conditions: "a research plan runs only once its person approves it, signed in or with a yes in their chat room from someone in bridge.users; an API key, an app or an agent only declines it unless research.agents_skip_review; the run acts as the person with their access at run time, offered only its sources' read tools, within the plan's tool calls (research.max_tool_calls), research.max_minutes and the budgets of its namespaces and all, checked before each stage; its web searches and page reads (and code with the network, when the plan names the sandbox) don't ask again; one run per person at a time; one process at a time (research_run:, a heartbeat every 10 s, taken over after 2 minutes of silence); every call in the ledger under research:" + violation_symptoms: [integrity-bypass] + x-voided-by: "research.agents_skip_review lets an agent's plan run without review (domain/research.py agents_may_approve); a run's web queries can carry archive text it read, within its tool calls (N-PROMPT); a yes in a room is the room's people's (N-BRIDGE)" + provenance: {kind: documented, source: 'docs/assistant.md "Deep research"'} - id: MX-AUTHZ-LIST-CAPS kind: resource-bound components: [authz]