diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b22a9a5a..e4757f9fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version the namespace → Connect apps, which copies its address (`/mcp/`). An agent connected there has the same tools but sees that namespace alone, whatever else the person who signs it in can read, and a token an app gets for it opens nothing else in Lens. Off by default ([One namespace's MCP server](docs/mcp.md#one-namespaces-mcp-server)). +- **Chat completions for any app, from one namespace.** A namespace can answer OpenAI's chat completions API at + `/api/v1/namespaces/`, so any app that speaks it (with a Lens API key and the model `lens`) gets answers + from the configured model, grounded in that namespace and citing it: `[n]` with a Sources list, `citations` and + `lens.sources`, streamed or not. Its owners pick the read tools the model may use there (none by default: one search, + one model call). Calls count for the namespace's budget. Off by default; turned on under Admin → Namespaces → the + namespace → Connect apps ([Chat completions for any app](docs/api.md#chat-completions-for-any-app)). ## 0.11.0 October 15, 2026 {id="0.11.0"} diff --git a/docs/api.md b/docs/api.md index facdaaff5..151bbe219 100644 --- a/docs/api.md +++ b/docs/api.md @@ -271,6 +271,10 @@ PATCH /api/v1/namespaces/{name}/collections/{cid} DELETE /api/v1/namespaces/{name}/collections/{cid} GET /api/v1/namespaces/{name}/collections/{cid}/members PUT /api/v1/namespaces/{name}/collections/{cid}/members +GET /api/v1/namespaces/{name}/integrations +PATCH /api/v1/namespaces/{name}/integrations +GET /api/v1/namespaces/{name}/models +POST /api/v1/namespaces/{name}/chat/completions ``` `GET /namespaces` lists the namespaces you have a role in, with their counts and your `role`, and those you see only @@ -312,6 +316,61 @@ also queues the `nsfw` step for its videos, documents and images not classified `queued`. Audited as `namespace.nsfw`. `POST …/nsfw/classify` queues them on its own (editors) and answers the `jobs`. Both are a 409 where classification is off in the namespace. +`/namespaces/{name}/integrations` says whether the namespace's [own MCP server](mcp.md#one-namespaces-mcp-server) +(`mcp`, at `mcp_url`) and its [chat completions](#chat-completions-for-any-app) (`chat`, at `chat_url`) are on, the +read tools the model may use there (`tools`) and the ones it may be offered (`tool_choices`), for anyone who can read +the namespace. `PATCH` with any of `{mcp, chat, tools}` changes them (owners); what isn't sent stays. A tool that +isn't one of `tool_choices` is a 400. Audited as `namespace.integrations`. + +### Chat completions for any app + +A namespace can answer OpenAI's chat completions API, so any app that speaks it (a chat app, an automation tool, a +library's OpenAI client) gets answers from the configured model, grounded in that namespace and citing it. Give the app: + +- **Base URL**: `https://lens.example.org/api/v1/namespaces/` (the app adds `/chat/completions`); +- **API key**: a Lens [API token](authentication.md) (`la_…`; read-only is enough) or an app's OAuth token; +- **Model**: `lens`, the configured model. One of the models people may pick in Lens's chat works too + (`GET …/models` lists them); any other name answers with the configured model. + +```bash +curl https://lens.example.org/api/v1/namespaces/family/chat/completions \ + -H "Authorization: Bearer la_…" -H "Content-Type: application/json" \ + -d '{"model": "lens", "messages": [{"role": "user", "content": "When is the dentist?"}]}' +``` + +Each request is answered like a question in Lens's chat, from that namespace alone, with what the token's person can +read there: + +- the last message is the question (a `user` message), the `user` and `assistant` messages before it the + conversation, and `system` (or `developer`) messages the app's own instructions, which the model reads after Lens's + rules. Pictures and files in a message are left out; +- the namespace's excerpts that best answer it are found and the model answers from them, citing each claim as `[n]`. + With no model set up, the best excerpts come back on their own; +- when the namespace's owners picked tools, the model may first look things up with them, in the namespace. They are + the assistant's tools that only read (`tool_choices`). A model that can't call tools answers from the excerpts and + says so in `lens.notice`; +- when the namespace's [own assistant](assistant.md#a-namespaces-own-assistant) is on, its name and instructions answer. + +What it cites comes back three ways, so any app can show it: `[n]` in the text with a **Sources** list of links after +it (left out when the request has a `response_format`), `citations` (the link for `[n]` is the n-th), and +`lens.sources` (`n`, `recording_id`, `title`, `time`, `speaker`, `url`, `cited`). `lens.steps` lists the tools the +model used. With `"stream": true` the answer comes as OpenAI's `chat.completion.chunk` events, ending with +`data: [DONE]`; the last chunk before it carries `finish_reason: "stop"`, `citations` and `lens`. While the tools run, +the stream sends SSE comments, so nothing in between closes it. + +It is off until an owner of the namespace turns it on: **Admin → Namespaces →** the namespace **→ Connect apps**, +which also copies the base URL and picks the tools, or `PATCH /namespaces/{name}/integrations` with `{"chat": true}`, +or an agent's `set_namespace_integrations`. Errors come in OpenAI's shape (`{"error": {"message", "type", "code"}}`): +401 `invalid_api_key` without a token, 422 for a body that isn't a request (FastAPI's shape), 404 `not_found` for a +namespace the token's person can't read (or that doesn't exist), 403 `endpoint_off` while it's off, 400 +`invalid_messages` with nothing to answer, 429 `insufficient_quota` when the namespace's [budget](budgets.md) or +Lens's overall one is used up, 502 `upstream_error` when the model server fails. + +It only reads. Model calls go through the activity ledger and count for the namespace (`space:`), so its budget +covers them. Not yet: the app's own `tools` (they are ignored), `max_tokens`, enforcing a `response_format`, token +counts in `usage`, streaming the tools' answer piece by piece (it comes in one chunk), and what the namespace's +assistant remembers. + ### Collections of a namespace Every asset lives in exactly one collection of its namespace, and collections nest (at most 8 deep, 5000 per diff --git a/docs/mcp.md b/docs/mcp.md index 21186465e..60a77ba3b 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -57,6 +57,9 @@ app was given for a namespace's server opens that server and nothing else: not a the API. A token for Lens or for `/mcp` (and an API token) works on every namespace's own server, seeing only that namespace there. +For an app that speaks OpenAI's API rather than MCP, the namespace can also answer +[chat completions](api.md#chat-completions-for-any-app), turned on in the same place (Connect apps). + ## Tools | Tool | | @@ -90,8 +93,8 @@ namespace there. | `nsfw_settings` | how [NSFW classification](nsfw.md) works in a namespace: on or off, the threshold, how flagged resources show, and how many are classified, flagged and not classified yet | | `set_nsfw` | turn NSFW classification on or off, set its threshold and display, for a namespace (owners) or the archive (admins), and optionally classify what's there. Needs a write-scope token | | `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access | -| `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) on, and its address | -| `set_namespace_integrations` | turn a namespace's own MCP server on or off. Owners, with a write-scope token | +| `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there | +| `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token | | `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files | | `read_file` | a text file of that tree (an asset's `.txt` text or `.json` metadata), a page at a time; other files' download link | | `file_storage` | where Lens keeps its own files (notes' images and attachments) and the connections it could use; `check` tests it. Admins only | @@ -117,7 +120,7 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed `merge_topics`, `delete_topic` and `undo_topic_change`, which curate entities and topics as an editor does in the web app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which queues an episode, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and -editors), `set_namespace_integrations`, which turns a namespace's own MCP server on or off (owners), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring` and +editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_file_storage`, which changes where new files are kept (admins), and `set_monitoring` and `set_install_settings`, which have the machine change how Lens runs (admins). Importing stays in the web app and the API. diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index c7d8b21e4..3d22db0db 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -1761,15 +1761,19 @@ def _integrations(ctx: Context, namespace: str, sid: int) -> dict[str, Any]: "namespace": namespace, **integrations.profile(ctx.db, sid), "mcp_url": f"{ctx.web}/mcp/{namespace}", + "chat_url": f"{ctx.web}/api/v1/namespaces/{namespace}", + "tool_choices": [{"name": n, "description": d} for n, d in integrations.tool_choices()], "settings": f"{ctx.web}/admin/namespaces/{namespace}#connect-apps", } @tool( "namespace_integrations", - "A namespace's own MCP server", + "A namespace's own MCP server and chat completions", "Whether a namespace has its own MCP server on (mcp), seeing that namespace alone, and its address (mcp_url) to " - "give an agent that should see nothing else. Off unless the namespace's owner turns it on.", + "give an agent that should see nothing else; whether its OpenAI-style chat completions are on (chat), answered " + "from that namespace alone and citing it, with the base URL to give an app (chat_url), and the read tools the " + "model may use there (tools, from tool_choices). Each is off unless the namespace's owner turns it on.", Arg("namespace", "string", "the namespace (see list_namespaces)", required=True), ) def namespace_integrations(ctx: Context, namespace: str) -> dict[str, Any]: @@ -1778,20 +1782,34 @@ def namespace_integrations(ctx: Context, namespace: str) -> dict[str, Any]: @tool( "set_namespace_integrations", - "Turn a namespace's own MCP server on or off", - "Turn on (mcp true) or off a namespace's own MCP server, which sees that namespace alone: agents connected to it " - "read only that namespace, with the roles of the person who signs them in. Owners of the namespace, with a token " - "that has the write scope.", + "Turn a namespace's own MCP server or chat completions on or off", + "Turn on or off a namespace's own MCP server (mcp) or its OpenAI-style chat completions (chat), and pick the read " + 'tools the model may use there (tools, from namespace_integrations\' tool_choices; ["none"] for none). Each sees that ' + "namespace alone, with the roles of the person who signs the app in. What isn't given stays as it is. Owners of " + "the namespace, with a token that has the write scope.", Arg("namespace", "string", "the namespace", required=True), - Arg("mcp", "boolean", "on or off", required=True), + Arg("mcp", "boolean", "its own MCP server on or off"), + Arg("chat", "boolean", "its chat completions on or off"), + Arg( + "tools", + "string[]", + 'the read tools its chat completions may use, or ["none"]', + enum=(*(n for n, _ in integrations.tool_choices()), "none"), + ), writes=True, ) -def set_namespace_integrations(ctx: Context, namespace: str, mcp: bool) -> dict[str, Any]: +def set_namespace_integrations( + ctx: Context, namespace: str, mcp: bool | None, chat: bool | None, tools: list[str] | None +) -> dict[str, Any]: if not ctx.user.can_write: raise ToolError("this token is read-only: ask for a token with the write scope to change this") + if mcp is None and chat is None and tools is None: + raise ToolError("give mcp, chat or tools to change") sid = ctx.acl.namespace(namespace, "manage") - integrations.save(ctx.db, sid, mcp, ctx.user.email) - auth.audit(ctx.db, ctx.user.as_audit(), "namespace.integrations", namespace, {"mcp": mcp, "via": "mcp"}) + tools = None if tools is None else [t for t in tools if t != "none"] + integrations.save(ctx.db, sid, mcp=mcp, chat=chat, tools=tools, user=ctx.user.email) + changed = store.clean({"mcp": mcp, "chat": chat, "tools": tools}) + auth.audit(ctx.db, ctx.user.as_audit(), "namespace.integrations", namespace, {**changed, "via": "mcp"}) return _integrations(ctx, namespace, sid) diff --git a/fastapi_backend/app/api/v1/router.py b/fastapi_backend/app/api/v1/router.py index 0daf45592..d72dfee96 100644 --- a/fastapi_backend/app/api/v1/router.py +++ b/fastapi_backend/app/api/v1/router.py @@ -13,6 +13,7 @@ chats, collections, comments, + completions, components, content_types, custom_nodes, @@ -86,6 +87,7 @@ namespaces, ns_assistant, integrations, + completions, hierarchy, recordings, notes, diff --git a/fastapi_backend/app/api/v1/routes/completions.py b/fastapi_backend/app/api/v1/routes/completions.py new file mode 100644 index 000000000..b88d2fceb --- /dev/null +++ b/fastapi_backend/app/api/v1/routes/completions.py @@ -0,0 +1,204 @@ +"""A namespace's own chat completions, OpenAI's way (completions.py): any app that speaks OpenAI's API gets answers +grounded in one namespace, citing it, with the base URL /api/v1/namespaces/ and a Lens token as its key. + +Errors come in OpenAI's shape ({"error": {"message", "type", "code"}}), so apps show them as they would OpenAI's. +""" + +from __future__ import annotations + +import contextvars +import json +import logging +import queue +import secrets +import threading +import time +from collections.abc import Iterator +from typing import Any + +from fastapi import APIRouter, Request +from fastapi.responses import JSONResponse, StreamingResponse + +from app.api.deps import Db, Principal, optional_user, within +from app.api.v1.routes.oauth import web_app_base +from app.domain import activity, auth, budgets, completions, integrations, llm, store +from app.schemas.completions import ChatCompletion, ChatCompletionRequest, CompletionModels + +log = logging.getLogger("lens") +router = APIRouter(prefix="/namespaces", tags=["chat completions"]) +KEEPALIVE = 15 # seconds between SSE comments while the tools are looked up, so proxies keep the stream open +OPENAI_ERRORS: dict[int | str, dict[str, Any]] = { + 400: {"description": "OpenAI's error shape: {error: {message, type, code}}"}, + 401: {"description": "no token, or one that doesn't open this namespace"}, + 403: {"description": "the namespace's chat completions are off"}, + 404: {"description": "no such namespace, or not one you can read"}, + 429: {"description": "a budget is used up"}, + 502: {"description": "the model server failed"}, +} + + +class Refused(Exception): + def __init__(self, status: int, message: str, code: str, kind: str = "invalid_request_error") -> None: + super().__init__(message) + self.status, self.message, self.code, self.kind = status, message, code, kind + + def response(self) -> JSONResponse: + headers = {"WWW-Authenticate": "Bearer"} if self.status == 401 else None + body = {"error": {"message": self.message, "type": self.kind, "param": None, "code": self.code}} + return JSONResponse(body, status_code=self.status, headers=headers) + + +def _open(request: Request, db: Db, name: str) -> tuple[Principal, int]: + """Who asks, and the namespace they ask about, when its chat completions are on and theirs to read.""" + user = optional_user(request, db) + if not user: + raise Refused(401, "give a Lens token as the API key (Authorization: Bearer …)", "invalid_api_key") + try: + sid = store.ns_id(db, name, create=False) + except KeyError: + raise Refused(404, f"there's no namespace {name} you can read", "not_found") from None + if not auth.allows(user.roles, sid): + raise Refused(404, f"there's no namespace {name} you can read", "not_found") + if not integrations.profile(db, sid)["chat"]: + raise Refused( + 403, + f"{name}'s chat completions are off: an owner of {name} turns them on under Admin → Namespaces → {name} → Connect apps", + "endpoint_off", + "permission_error", + ) + return within(user, sid), sid + + +@router.get("/{name}/models", responses=OPENAI_ERRORS) +def list_completion_models(name: str, request: Request, db: Db) -> CompletionModels: + """OpenAI's model list for the namespace's chat completions: lens (the configured model) and the models people may + pick in Lens's chat.""" + try: + _open(request, db, name) + except Refused as e: + return e.response() # type: ignore[return-value] + cfg = request.app.state.settings.current() + return CompletionModels(data=[{"id": m} for m in completions.models(cfg)]) + + +def _chunk(cid: str, created: int, model: str, delta: dict[str, Any], finish: str | None = None, **extra: Any) -> str: + body = { + "id": cid, + "object": "chat.completion.chunk", + "created": created, + "model": model, + "choices": [{"index": 0, "delta": delta, "finish_reason": finish}], + **extra, + } + return f"data: {json.dumps(body, ensure_ascii=False)}\n\n" + + +@router.post("/{name}/chat/completions", response_model=ChatCompletion, responses=OPENAI_ERRORS) +def create_chat_completion(name: str, body: ChatCompletionRequest, request: Request, db: Db) -> Any: + """OpenAI's chat completions, answered from this namespace alone: the last user message is the question, the + messages before it the conversation and system messages the app's instructions. The answer cites what it read as + [n], with a Sources list after it, `citations` (the n-th link for [n]) and `lens.sources`. With `stream`, it comes + as OpenAI's chunks (server-sent events) ending in `data: [DONE]`. Off until an owner of the namespace turns it on; + read-only tokens are enough. 429 when the namespace's budget or Lens's overall one is used up.""" + cfg = request.app.state.settings.current() + try: + user, sid = _open(request, db, name) + try: + question, history, app = completions.read(body.messages) + except ValueError as e: + raise Refused(400, str(e), "invalid_messages") from None + if budgets.status(db, f"space:{sid}")["state"] == "over": + raise Refused( + 429, f"{name}'s budget is used up; raise it in Settings › Budgets to go on", "insufficient_quota", "insufficient_quota" + ) + except Refused as e: + return e.response() + model = completions.pick(cfg, body.model) + tools = integrations.profile(db, sid)["tools"] + web = web_app_base(request) + args = (db, cfg, sid, name, user.as_audit(), question, history, app, model, tools, request.app.state.archive.base, body.temperature) + cid, created = "chatcmpl-" + secrets.token_hex(12), int(time.time()) + with_sources = body.response_format is None + + def extras(done: dict[str, Any], steps: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], dict[str, Any]]: + found = completions.sources(web, done["text"], done["refs"]) + lens = {"namespace": name, "sources": found, "steps": steps, **({"notice": done["notice"]} if done.get("notice") else {})} + return found, {"citations": [s["url"] for s in found], "lens": lens} + + if not body.stream: + steps: list[dict[str, Any]] = [] + text, done = "", {} + try: + with activity.scope(db, f"space:{sid}", cfg=cfg): + for kind, data in completions.answer(*args): + if kind == "step": + steps.append(data) + elif kind == "token": + text += data + else: + done = data + except activity.OverBudget as e: + return Refused(429, str(e), "insufficient_quota", "insufficient_quota").response() + except llm.LLMError as e: + return Refused(502, str(e), "upstream_error", "api_error").response() + found, more = extras(done, steps) + content = text + (completions.footer(found) if with_sources else "") + return { + "id": cid, + "object": "chat.completion", + "created": created, + "model": done["model"], + "choices": [{"index": 0, "message": {"role": "assistant", "content": content}, "finish_reason": "stop"}], + **more, + } + + # Streamed: the answer is worked out in a thread of its own (the ledger's scope holds for all of it) and handed + # over piece by piece; a quiet stretch (the tools) sends an SSE comment so nothing in between closes the stream. + events: queue.Queue[tuple[str, Any]] = queue.Queue() + + def work() -> None: + try: + with activity.scope(db, f"space:{sid}", cfg=cfg): + for ev in completions.answer(*args): + events.put(ev) + except activity.OverBudget as e: + events.put(("error", Refused(429, str(e), "insufficient_quota", "insufficient_quota"))) + except llm.LLMError as e: + events.put(("error", Refused(502, str(e), "upstream_error", "api_error"))) + except Exception: # noqa: BLE001 - the app hears it broke off rather than waiting + log.exception("chat completions for %s failed", name) + events.put(("error", Refused(500, "something went wrong while answering; try again", "server_error", "api_error"))) + events.put(("end", None)) + + threading.Thread(target=contextvars.copy_context().run, args=(work,), daemon=True, name="lens-completion").start() + shown = completions.MODEL if model is None else model + + def gen() -> Iterator[str]: + yield _chunk(cid, created, shown, {"role": "assistant", "content": ""}) + steps: list[dict[str, Any]] = [] + while True: + try: + kind, data = events.get(timeout=KEEPALIVE) + except queue.Empty: + yield ": still working\n\n" + continue + if kind == "step": + steps.append(data) + yield f": looked up {data['tool']}\n\n" + elif kind == "token": + yield _chunk(cid, created, shown, {"content": data}) + elif kind == "done": + found, more = extras(data, steps) + tail = completions.footer(found) if with_sources else "" + if tail: + yield _chunk(cid, created, shown, {"content": tail}) + usage = {"usage": None} if (body.stream_options or {}).get("include_usage") else {} + yield _chunk(cid, created, shown, {}, "stop", **more, **usage) + elif kind == "error": + err: Refused = data + yield f"data: {json.dumps({'error': {'message': err.message, 'type': err.kind, 'param': None, 'code': err.code}})}\n\n" + else: + yield "data: [DONE]\n\n" + return + + return StreamingResponse(gen(), media_type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}) diff --git a/fastapi_backend/app/api/v1/routes/integrations.py b/fastapi_backend/app/api/v1/routes/integrations.py index 774487fbb..bafdbbf86 100644 --- a/fastapi_backend/app/api/v1/routes/integrations.py +++ b/fastapi_backend/app/api/v1/routes/integrations.py @@ -1,5 +1,6 @@ -"""A namespace's own ways in for other apps (integrations.py): owners turn on its MCP server, which sees that -namespace alone; anyone who can read the namespace sees whether it's on and where it is.""" +"""A namespace's own ways in for other apps (integrations.py): owners turn on its MCP server and its chat completions, +which see that namespace alone, and pick the tools the model may use there; anyone who can read the namespace sees +what is on and where it is.""" from __future__ import annotations @@ -8,18 +9,24 @@ from app.api.deps import Acl, Db, Writer from app.api.v1.routes.oauth import MCP, web_app_base from app.domain import auth, integrations -from app.schemas.integrations import NamespaceIntegrations, NamespaceIntegrationsUpdate +from app.schemas.integrations import IntegrationTool, NamespaceIntegrations, NamespaceIntegrationsUpdate router = APIRouter(prefix="/namespaces", tags=["namespaces"]) def _view(request: Request, db: Db, sid: int, name: str) -> NamespaceIntegrations: - return NamespaceIntegrations(**integrations.profile(db, sid), mcp_url=f"{web_app_base(request)}{MCP}/{name}") + web = web_app_base(request) + return NamespaceIntegrations( + **integrations.profile(db, sid), + mcp_url=f"{web}{MCP}/{name}", + chat_url=f"{web}/api/v1/namespaces/{name}", + tool_choices=[IntegrationTool(name=n, description=d) for n, d in integrations.tool_choices()], + ) @router.get("/{name}/integrations") def get_namespace_integrations(name: str, request: Request, acl: Acl, db: Db) -> NamespaceIntegrations: - """Whether the namespace's own MCP server is on, and its address.""" + """Whether the namespace's own MCP server and chat completions are on, their addresses, and the tools offered.""" return _view(request, db, acl.namespace(name), name) @@ -27,12 +34,14 @@ def get_namespace_integrations(name: str, request: Request, acl: Acl, db: Db) -> def update_namespace_integrations( name: str, body: NamespaceIntegrationsUpdate, request: Request, acl: Acl, user: Writer, db: Db ) -> NamespaceIntegrations: - """Owners: turn the namespace's own MCP server on or off. Off, it answers 403; apps keep their access to the rest of - Lens as before.""" + """Owners: turn the namespace's own MCP server or chat completions on or off, or pick the read tools the model may + use in its chat completions. Off, each answers 403; apps keep their access to the rest of Lens as before.""" sid = acl.namespace(name, "manage") try: - integrations.save(db, sid, body.mcp, user.email) + integrations.save(db, sid, body.mcp, body.chat, body.tools, user.email) except KeyError: raise HTTPException(404, "not found") from None + except ValueError as e: + raise HTTPException(400, str(e)) from None auth.audit(db, user.as_audit(), "namespace.integrations", name, body.model_dump(exclude_unset=True)) return _view(request, db, sid, name) diff --git a/fastapi_backend/app/domain/completions.py b/fastapi_backend/app/domain/completions.py new file mode 100644 index 000000000..44c44f45d --- /dev/null +++ b/fastapi_backend/app/domain/completions.py @@ -0,0 +1,187 @@ +"""Grounded answers for any app, OpenAI's way: a namespace's own chat completions (docs/api.md, "Chat completions for +any app"). + +An app that speaks OpenAI's chat completions API (a chat app, an automation tool, a library's OpenAI client) points +its base URL at ``/api/v1/namespaces/`` and its key at a Lens token. Each request is answered like a +question in Lens's chat, by the configured model, from that namespace alone: + +* the last user message is the question, the user and assistant messages before it the conversation, and system (or + developer) messages the app's own instructions, read after Lens's rules; +* the namespace's excerpts that best answer it are found (chat.retrieve) and the model answers from them, citing + each claim as [n]; with no model set up, the best excerpts come back on their own; +* when the namespace's owners offered tools (integrations.tool_choices: the assistant's tools that only read), the + model may look things up with them first (chat.tool_answer), within the namespace; a model that can't call tools + answers from the excerpts instead; +* when the namespace's own assistant is on (ns_assistant.py), the answer comes from it: its name and instructions. + +What it cites comes back three ways, so any app can show it: [n] in the text with a "Sources" list after it (left +out when the app asked for a response_format), ``citations`` (the n-th link for [n]) and ``lens.sources``. + +It only reads. Model calls go through the activity ledger and count for the namespace (space:), and the +overall budget's hard limit stops them as it does any call. + +Refine later: the app's own tools (tools in the request, handed back as tool_calls), streaming the tool loop's +answer token by token, max_tokens, what the namespace's assistant remembers, a model per namespace. +""" + +from __future__ import annotations + +import re + +from . import ai_tools, chat, llm, ns_assistant, store + +MODEL = "lens" # the model id apps pick: the configured model, answering from the namespace +ROLES = ("system", "developer", "user", "assistant", "tool") +QUESTION_CHARS = 4000 # as a question in Lens's chat +APP_CHARS = 4000 # of the app's own instructions + + +def _text(content, i): + """A message's text: a string, or the text parts of a list of parts (pictures and files are left out).""" + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + return "\n".join( + p["text"] for p in content if isinstance(p, dict) and p.get("type") in ("text", "input_text") and isinstance(p.get("text"), str) + ) + raise ValueError(f"messages[{i}].content is text, or a list of parts") + + +def read(messages): + """(question, the conversation before it, the app's instructions) from OpenAI's `messages`. ValueError when there + is nothing to answer.""" + if not isinstance(messages, list) or not messages: + raise ValueError("messages is a list with at least one message") + app, turns = [], [] + for i, m in enumerate(messages): + if not isinstance(m, dict) or m.get("role") not in ROLES: + raise ValueError(f"messages[{i}] needs a role: {', '.join(ROLES)}") + text = _text(m.get("content"), i).strip() + if not text: + continue + if m["role"] in ("system", "developer"): + app.append(text) + elif m["role"] in ("user", "assistant"): + turns.append({"id": i, "role": m["role"], "content": text}) + if not turns or turns[-1]["role"] != "user" or messages[-1].get("role") != "user": + raise ValueError("the last message is the question: a user message with some text") + return turns[-1]["content"][:QUESTION_CHARS], turns[:-1], "\n\n".join(app)[:APP_CHARS] + + +def models(cfg): + """The model ids apps may pick: lens (the configured model) and the models people may pick in Lens's chat.""" + return [MODEL, *chat.model_choices(cfg)] + + +def pick(cfg, asked): + """The model to answer with: one people may pick in Lens's chat when that's asked for, else the configured one + (None), whatever an app sends by default.""" + if asked and asked != MODEL and asked != cfg["llm"].get("model") and asked in chat.model_choices(cfg): + return asked + return None + + +def note(db, sid, ns, app): + """What the model is told besides Lens's own rules: where it answers from, the namespace's assistant (when it is + on), and the app's own instructions.""" + out = f"\n\nYou answer for another app, through Lens, from the {ns} namespace alone." + p = ns_assistant.profile(db, sid) + if p["enabled"]: + out += f" You are {p['name']}, the assistant of the {ns} namespace." + if p["instructions"]: + out += f"\n\nInstructions for this namespace's assistant, from its owners:\n{p['instructions']}" + if app: + out += f"\n\nInstructions from the app you answer through (Lens's rules above still hold):\n{app}" + return out + + +class Box(ai_tools.Toolbox): + """The assistant's tools for an app: only the read tools the namespace's owners offered, in that namespace, with + none of the person's extensions, skills or hooks, and nothing that changes data.""" + + def __init__(self, db, cfg, user, sid, ns, tools, said, base, note_text): + super().__init__(db, cfg, user, {sid}, set(), {"namespaces": [ns]}, None, base, said=said) + self.only, self.note_text = set(tools), note_text + self.ext, self.home, self.browser = None, None, None + + def system_note(self): + return self.note_text + + def specs(self): + return [s for s in super().specs() if s["function"]["name"] in self.only] + + +def answer(db, cfg, sid, ns, user, question, history, app="", model=None, tools=(), base=None, temperature=None): + """Answer one request. Yields ("step", {tool, args, summary}) for each tool call, ("token", text) for the + answer's pieces, then ("done", {refs, text, model, notice}): what it can cite (numbered, as [n]) and the whole + answer. Raises llm.LLMError when the model server fails, and activity.OverBudget at the hard limit.""" + scope = {"namespaces": [ns]} + passages = chat.retrieve(db, question, {sid}, scope, cfg=cfg) + extra, notice = note(db, sid, ns, app), None + used = model or cfg["llm"].get("model") + if not llm.configured(cfg): + text = chat.fallback(passages) + yield "token", text + yield "done", {"refs": passages, "text": text, "model": MODEL, "notice": None} + return + if tools and cfg["ai"].get("tools", True): + box = Box(db, cfg, user, sid, ns, tools, question, base, extra) + try: + text = "" + for kind, data in chat.tool_answer(cfg, box, question, history, cfg["ai"].get("max_steps") or 6, model): + if kind == "step": + yield "step", data + # a model that skips the tools hasn't seen the namespace: it answers from the excerpts instead + elif kind == "direct" and passages and not box.cited(data): + break + else: + text = data + else: + yield "token", text + yield "done", {"refs": box.refs, "text": text, "model": used, "notice": None} + return + except llm.ToolsUnsupported: + notice = "This model can't use tools, so the answer comes from a search instead." + msgs = chat.messages_for(question, passages, history, None, cfg) + msgs[0]["content"] += extra + text = "" + for piece in llm.stream_chat(cfg, msgs, model=model, temperature=0.2 if temperature is None else temperature): + text += piece + yield "token", piece + yield "done", {"refs": passages, "text": text, "model": used, "notice": notice} + + +def link(web, ref): + """A reference's page in the web app, at its moment (whole seconds, as the page reads `?t=`).""" + t0 = ref.get("t0") or 0 + return f"{web}/resources/{ref['recording_id']}" + (f"?t={int(t0) // 1000}" if t0 >= 1000 else "") + + +def sources(web, text, refs): + """Every numbered reference, with whether the answer cites it ([n]).""" + cited = {int(n) for n in re.findall(r"\[(\d+)\]", text or "")} + return [ + store.clean( + { + "n": r["n"], + "recording_id": r["recording_id"], + "title": r.get("title"), + "time": r.get("time"), + "speaker": r.get("speaker"), + "url": link(web, r), + "cited": r["n"] in cited, + } + ) + for r in refs + ] + + +def footer(found): + """The sources the answer cites, as Markdown after it, for apps that show only the text.""" + cited = [s for s in found if s["cited"]] + if not cited: + return "" + lines = [f"[{s['n']}] [{' · '.join(x for x in (s.get('title') or 'Untitled', s.get('time')) if x)}]({s['url']})" for s in cited] + return "\n\nSources:\n" + "\n".join(lines) diff --git a/fastapi_backend/app/domain/integrations.py b/fastapi_backend/app/domain/integrations.py index fefe43eca..19e96a923 100644 --- a/fastapi_backend/app/domain/integrations.py +++ b/fastapi_backend/app/domain/integrations.py @@ -1,19 +1,31 @@ -"""A namespace's own ways in for other apps (docs/mcp.md, "One namespace's MCP server"). +"""A namespace's own ways in for other apps (docs/mcp.md, "One namespace's MCP server"; docs/api.md, "Chat +completions for any app"). * ``mcp``: an MCP server that sees this namespace alone, at ``/mcp/`` (app/api/mcp.py). An agent connected to it searches, reads and cites this namespace and nothing else, whatever else the person who signed it in can read; a token an app was given for it (its RFC 8707 resource) opens nothing but it. +* ``chat``: OpenAI's chat completions API at ``/api/v1/namespaces/`` (``/chat/completions`` and + ``/models``, completions.py), so any app that speaks it gets answers from the configured model, grounded in this + namespace and citing it. ``tools``: the assistant's read tools the model may also use there (none by default: one + search, one model call). -Off until an owner of the namespace turns it on (Admin → Namespaces → the namespace → Connect apps, or PATCH +Each is off until an owner of the namespace turns it on (Admin → Namespaces → the namespace → Connect apps, or PATCH /namespaces/{name}/integrations). Turned on, it opens nothing new: apps still sign in as a person (OAuth, or an API token) and see what that person sees in this namespace, never more. """ from __future__ import annotations -from . import store +from . import ai_tools, store R = store.R +NOT_OFFERED = {"entity_setup"} # reads, but is about setting the namespace up, not about what it holds + + +def tool_choices(): + """The tools a namespace's chat completions may offer the model: the assistant's own tools that only read, as + (name, what it does).""" + return [(t[0], t[1]) for t in ai_tools.TOOLS if not t[4] and t[0] not in NOT_OFFERED] def profile(db, sid): @@ -22,14 +34,29 @@ def profile(db, sid): if row is None: raise KeyError(sid) i = row.get("integrations") or {} - return {"mcp": bool(i.get("mcp")), "updated_at": i.get("updated_at"), "updated_by": i.get("updated_by")} + known = {n for n, _ in tool_choices()} + return { + "mcp": bool(i.get("mcp")), + "chat": bool(i.get("chat")), + "tools": [t for t in i.get("tools") or [] if t in known], + "updated_at": i.get("updated_at"), + "updated_by": i.get("updated_by"), + } -def save(db, sid, mcp=None, user=None): - """Change what is given; the rest stays.""" +def save(db, sid, mcp=None, chat=None, tools=None, user=None): + """Change what is given; the rest stays. ValueError for a tool that isn't one of tool_choices.""" i = profile(db, sid) if mcp is not None: i["mcp"] = bool(mcp) + if chat is not None: + i["chat"] = bool(chat) + if tools is not None: + known = [n for n, _ in tool_choices()] + bad = [t for t in tools if t not in known] + if bad: + raise ValueError(f"{', '.join(bad)} isn't one of the tools offered here: {', '.join(known)}") + i["tools"] = [t for t in known if t in set(tools)] i.update(updated_at=store.now(), updated_by=user) db.q("UPDATE $r SET integrations = $i", r=R("space", int(sid)), i=store.clean(i)) return profile(db, sid) diff --git a/fastapi_backend/app/schemas/completions.py b/fastapi_backend/app/schemas/completions.py new file mode 100644 index 000000000..d777d5899 --- /dev/null +++ b/fastapi_backend/app/schemas/completions.py @@ -0,0 +1,77 @@ +"""OpenAI's chat completions shapes, as a namespace's own endpoint takes and gives them (completions.py).""" + +from __future__ import annotations + +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field + +from app.schemas.common import ResponseModel + + +class ChatCompletionRequest(BaseModel): + """OpenAI's request. Fields it has that aren't listed here (n, top_p, user, max_tokens, tools, …) are taken and + left alone.""" + + model_config = ConfigDict(extra="allow") + + messages: list[dict[str, Any]] = Field( + description="the conversation: system, user, assistant (and tool) messages; the last is the question", max_length=1000 + ) + model: str | None = Field(None, description="lens (the configured model), or one people may pick in Lens's chat; anything else is lens") + stream: bool = Field(False, description="send the answer as server-sent events, OpenAI's chunks, ending in data: [DONE]") + temperature: float | None = Field(None, ge=0, le=2) + stream_options: dict[str, Any] | None = None + response_format: dict[str, Any] | None = Field( + None, description="any format asked for: the answer then has no Sources list after it (the format itself isn't enforced yet)" + ) + + +class CompletionSource(ResponseModel): + n: int = Field(description="what the answer cites it as: [n]") + recording_id: int = Field(description="the asset it comes from") + title: str | None = None + time: str | None = Field(None, description='the moment, as "12:34", or a document\'s page, as "p. 2"') + speaker: str | None = Field(None, description="who spoke, in audio or video") + url: str = Field(description="the moment in the web app") + cited: bool = Field(description="whether the answer cites it") + + +class CompletionMessage(ResponseModel): + role: str = "assistant" + content: str + + +class CompletionChoice(ResponseModel): + index: int = 0 + message: CompletionMessage + finish_reason: str = "stop" + + +class CompletionLens(ResponseModel): + namespace: str + sources: list[CompletionSource] + steps: list[dict[str, Any]] = Field(default_factory=list, description="the tools it looked things up with") + notice: str | None = None + + +class ChatCompletion(ResponseModel): + id: str + object: str = "chat.completion" + created: int + model: str + choices: list[CompletionChoice] + citations: list[str] = Field(description="the links of what it can cite, in order: [n] is the n-th") + lens: CompletionLens + + +class CompletionModel(ResponseModel): + id: str + object: str = "model" + created: int = 0 + owned_by: str = "lens" + + +class CompletionModels(ResponseModel): + object: str = "list" + data: list[CompletionModel] diff --git a/fastapi_backend/app/schemas/integrations.py b/fastapi_backend/app/schemas/integrations.py index cc537cd86..9387eaa3a 100644 --- a/fastapi_backend/app/schemas/integrations.py +++ b/fastapi_backend/app/schemas/integrations.py @@ -5,9 +5,18 @@ from app.schemas.common import RequestModel, ResponseModel +class IntegrationTool(ResponseModel): + name: str + description: str + + class NamespaceIntegrations(ResponseModel): mcp: bool = Field(description="the namespace's own MCP server answers, seeing this namespace alone") mcp_url: str = Field(description="where MCP clients connect to it") + chat: bool = Field(False, description="the namespace's own chat completions answer apps, from this namespace alone") + chat_url: str = Field("", description="the base URL OpenAI-compatible apps are given (they add /chat/completions)") + tools: list[str] = Field(default_factory=list, description="the read tools the model may use there (none: one search)") + tool_choices: list[IntegrationTool] = Field(default_factory=list, description="the tools that can be offered there") updated_at: str | None = None updated_by: str | None = None @@ -16,3 +25,5 @@ class NamespaceIntegrationsUpdate(RequestModel): """Only the fields you send change.""" mcp: bool | None = None + chat: bool | None = None + tools: list[str] | None = Field(None, max_length=50) diff --git a/fastapi_backend/tests/api/test_completions.py b/fastapi_backend/tests/api/test_completions.py new file mode 100644 index 000000000..dfea013cc --- /dev/null +++ b/fastapi_backend/tests/api/test_completions.py @@ -0,0 +1,241 @@ +"""A namespace's own chat completions (/api/v1/namespaces//chat/completions, docs/api.md): OpenAI's API, +answered by the configured model from that namespace alone, citing it; off until an owner turns it on.""" + +from __future__ import annotations + +import json +from typing import Any + +import pytest + +from app.domain import budgets, store +from tests import fake_llm +from tests.api._assist import start_llm +from tests.helpers import login, make_user, seed + +Q = "When does the Dyno Therapeutics shipment leave?" + + +@pytest.fixture +def llm(cfg): + srv = start_llm(cfg) + yield fake_llm.Handler + srv.shutdown() + fake_llm.Handler.usage = None + + +@pytest.fixture +def app(cfg, db, llm): + from app.main import create_app + + return create_app(cfg, db, background=False) + + +@pytest.fixture +def env(client, db, cfg, folder): + ids = seed(db, cfg, folder) + make_user(db, "own@x.io", "owner password 1", roles={"pods": "owner", "calls": "viewer"}) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + make_user(db, "root@x.io", "admin password 1", admin=True) + return { + "ids": ids, + "pods": store.ns_id(db, "pods"), + "own": login(client, "own@x.io", "owner password 1"), + "vi": login(client, "vi@x.io", "viewer password 1"), + "root": login(client, "root@x.io", "admin password 1"), + } + + +def turn_on(client, h, ns="pods", **body: Any) -> dict[str, Any]: + r = client.patch(f"/api/v1/namespaces/{ns}/integrations", headers=h, json={"chat": True, **body}) + assert r.status_code == 200, r.text + return r.json() + + +def ask(client, h, content=Q, ns="pods", status=200, **body: Any) -> dict[str, Any]: + msgs = body.pop("messages", None) or [{"role": "user", "content": content}] + r = client.post(f"/api/v1/namespaces/{ns}/chat/completions", headers=h, json={"model": "lens", "messages": msgs, **body}) + assert r.status_code == status, r.text + return r.json() + + +def chunks(text: str) -> list[Any]: + out = [] + for block in text.strip().split("\n\n"): + for line in block.splitlines(): + if line.startswith("data: "): + out.append(line[6:] if line[6:] == "[DONE]" else json.loads(line[6:])) + return out + + +def _system(llm) -> str: + return next(m["content"] for m in llm.seen[-1]["messages"] if m["role"] == "system") + + +def test_off_until_an_owner_turns_it_on(client, env, llm): + got = client.get("/api/v1/namespaces/pods/integrations", headers=env["vi"]).json() + assert (got["chat"], got["tools"], got["chat_url"]) == (False, [], "http://localhost:3000/api/v1/namespaces/pods") + assert "search_transcripts" in [t["name"] for t in got["tool_choices"]] + assert not {"remember", "entity_setup", "propose_entity_change"} & {t["name"] for t in got["tool_choices"]} # reads only + err = ask(client, env["own"], status=403)["error"] + assert err["code"] == "endpoint_off" and "Connect apps" in err["message"] # OpenAI's error shape + r = client.post("/api/v1/namespaces/pods/chat/completions", json={"messages": [{"role": "user", "content": Q}]}) + assert r.status_code == 401 and r.json()["error"]["code"] == "invalid_api_key" and r.headers["www-authenticate"] == "Bearer" + # owners turn it on, and only with tools that read + assert client.patch("/api/v1/namespaces/pods/integrations", headers=env["vi"], json={"chat": True}).status_code == 403 + r = client.patch("/api/v1/namespaces/pods/integrations", headers=env["own"], json={"chat": True, "tools": ["remember"]}) + assert r.status_code == 400 and "remember" in r.json()["detail"] + got = turn_on(client, env["own"], tools=["search_transcripts", "list_recordings", "search_transcripts"]) + assert (got["chat"], got["mcp"], got["tools"], got["updated_by"]) == ( + True, + False, + ["search_transcripts", "list_recordings"], + "own@x.io", + ) + assert turn_on(client, env["own"], mcp=True)["tools"] == ["search_transcripts", "list_recordings"] # the rest stays + # someone who can't read a namespace doesn't learn it's there + assert ask(client, env["vi"], ns="calls", status=404)["error"]["code"] == "not_found" + assert ask(client, env["vi"], ns="nowhere", status=404)["error"]["code"] == "not_found" + assert ask(client, env["own"], ns="calls", status=403)["error"]["code"] == "endpoint_off" + audit = [a["action"] for a in client.get("/api/v1/audit", headers=env["root"]).json()] + assert "namespace.integrations" in audit + + +def test_answers_from_the_namespace_alone_with_citations(client, env, cfg, llm): + ep1, ep2, call_id = env["ids"] + turn_on(client, env["own"]) + msgs = [ + {"role": "system", "content": "Answer in one line."}, + {"role": "user", "content": "Who makes capsids?"}, + {"role": "assistant", "content": "Dyno Therapeutics [1]."}, + {"role": "user", "content": [{"type": "text", "text": Q}, {"type": "image_url", "image_url": {"url": "x"}}]}, + ] + got = ask(client, env["own"], messages=msgs) # reads pods and calls; here, only pods + assert (got["object"], got["model"], got["choices"][0]["finish_reason"]) == ("chat.completion", "fake", "stop") + answer, sources = got["choices"][0]["message"]["content"].split("\n\nSources:\n") + assert answer == "The shipment leaves on Friday [1]." + found = got["lens"]["sources"] + assert found and {s["recording_id"] for s in found} <= {ep1, ep2} and found[0]["cited"] + assert got["citations"] == [s["url"] for s in found] and got["citations"][0].startswith( + f"http://localhost:3000/resources/{found[0]['recording_id']}" + ) + assert sources.startswith("[1] [") and got["citations"][0] in sources and got["lens"]["namespace"] == "pods" + sent = llm.seen[-1]["messages"] + assert "shipment leaves on Friday" not in json.dumps(sent) # the calls namespace's line never reached the model + assert [m["role"] for m in sent] == ["system", "user", "assistant", "user"] and sent[1]["content"] == "Who makes capsids?" + system = _system(llm) + assert "from the pods namespace alone" in system and system.endswith("Answer in one line.") + assert "Lens's rules above still hold" in system # the app's instructions come after Lens's + # an app asking for a format gets the answer alone + got = ask(client, env["own"], response_format={"type": "text"}) + assert got["choices"][0]["message"]["content"] == "The shipment leaves on Friday [1]." and got["citations"] + # the namespace's own assistant answers when it's on + client.patch( + "/api/v1/namespaces/pods/assistant", headers=env["own"], json={"enabled": True, "name": "Pod pal", "instructions": "Be brief."} + ) + ask(client, env["vi"]) + assert "You are Pod pal, the assistant of the pods namespace" in _system(llm) and "Be brief." in _system(llm) + # nothing to answer + for bad in ([], [{"role": "assistant", "content": "hi"}], [{"role": "user", "content": " "}], [{"role": "robot", "content": "x"}]): + r = client.post("/api/v1/namespaces/pods/chat/completions", headers=env["own"], json={"messages": bad}) + assert r.status_code in (400, 422), bad + if r.status_code == 400: + assert r.json()["error"]["code"] == "invalid_messages" + + +def test_streams_as_openai_does(client, env, cfg, llm): + turn_on(client, env["own"]) + r = client.post( + "/api/v1/namespaces/pods/chat/completions", + headers=env["vi"], + json={"messages": [{"role": "user", "content": Q}], "stream": True, "stream_options": {"include_usage": True}}, + ) + assert r.status_code == 200 and r.headers["content-type"].startswith("text/event-stream") + got = chunks(r.text) + assert got[-1] == "[DONE]" + first, *rest, last, done = got + assert first["choices"][0]["delta"] == {"role": "assistant", "content": ""} and first["object"] == "chat.completion.chunk" + text = "".join(c["choices"][0]["delta"].get("content") or "" for c in rest) + assert text.startswith("The shipment leaves on Friday [1].\n\nSources:\n[1] [") + assert last["choices"][0]["finish_reason"] == "stop" and last["citations"] and last["lens"]["sources"][0]["cited"] + assert "usage" in last and len({c["id"] for c in got[:-1]}) == 1 + + +def test_tools_the_owners_offer(client, env, cfg, llm): + ep1, ep2, call_id = env["ids"] + turn_on(client, env["own"], tools=["search_transcripts"]) + + def tool(i, name, args): + return {"id": f"c{i}", "type": "function", "function": {"name": name, "arguments": json.dumps(args)}} + + llm.tool_script = [ + {"content": "", "tool_calls": [tool(1, "search_transcripts", {"query": "Dyno Therapeutics"}), tool(2, "remember", {"fact": "x"})]}, + {"content": "Dyno Therapeutics comes up on the pods [1]."}, + ] + got = ask(client, env["own"]) + offered = [t["function"]["name"] for t in llm.seen[0]["tools"]] + assert offered == ["search_transcripts"] # what the owners offered, and nothing that changes data + steps = got["lens"]["steps"] + assert [s["tool"] for s in steps] == ["search_transcripts", "remember"] and "unknown tool" in steps[1]["summary"], steps + assert got["choices"][0]["message"]["content"].startswith("Dyno Therapeutics comes up on the pods [1].\n\nSources:") + found = got["lens"]["sources"] + assert found and {s["recording_id"] for s in found} <= {ep1, ep2} # the search never left pods + result = next(m["content"] for m in llm.seen[1]["messages"] if m["role"] == "tool") + assert "Dyno Therapeutics" in result and "leaves on Friday" not in result # calls is another namespace + assert result.startswith("<") and 'source="tool search_transcripts"' in result # read as data (untrusted.wrap) + # streamed, each look-up is a comment while the answer is worked out + llm.tool_script = [{"content": "", "tool_calls": [tool(1, "search_transcripts", {"query": "capsid"})]}, {"content": "Capsids [1]."}] + r = client.post( + "/api/v1/namespaces/pods/chat/completions", headers=env["own"], json={"messages": [{"role": "user", "content": Q}], "stream": True} + ) + assert ": looked up search_transcripts" in r.text and chunks(r.text)[-2]["lens"]["steps"][0]["tool"] == "search_transcripts" + assert "".join(c["choices"][0]["delta"].get("content") or "" for c in chunks(r.text)[:-1]).startswith("Capsids [1].\n\nSources:") + # a model that answers without looking gets the search's excerpts instead + llm.tool_script = [{"content": "I don't know."}] + got = ask(client, env["own"]) + assert got["choices"][0]["message"]["content"].startswith("The shipment leaves on Friday [1].") + # one that can't call tools answers from a search, and says so + llm.reject_tools = True + got = ask(client, env["own"]) + assert got["lens"]["notice"].startswith("This model can't use tools") and got["choices"][0]["message"]["content"].startswith( + "The shipment" + ) + + +def test_models_tokens_and_no_model(client, env, cfg, llm): + turn_on(client, env["own"]) + got = client.get("/api/v1/namespaces/pods/models", headers=env["vi"]).json() + assert [m["id"] for m in got["data"]] == ["lens", "fake", "fake-large"] and got["object"] == "list" + assert client.get("/api/v1/namespaces/calls/models", headers=env["vi"]).status_code == 404 + assert ask(client, env["vi"], model="fake-large")["model"] == "fake-large" and llm.seen[-1]["model"] == "fake-large" + assert ask(client, env["vi"], model="gpt-4o")["model"] == "fake" # what an app sends by default: the configured one + # a read-only API key is enough + key = client.post("/api/v1/tokens", headers=env["vi"], json={"name": "my app"}).json()["token"] + assert ask(client, {"Authorization": f"Bearer {key}"})["choices"][0]["message"]["content"].startswith("The shipment") + assert ask(client, {"Authorization": "Bearer la_nope"}, status=401)["error"]["code"] == "invalid_api_key" + # no model set up: the best excerpts on their own + assert client.put("/api/v1/settings/llm", headers=env["root"], json={"base_url": None}).status_code == 200 + got = ask(client, env["vi"]) + assert got["choices"][0]["message"]["content"].startswith("No language model is configured") and got["model"] == "lens" + + +def test_counts_for_the_namespace_and_its_budget(client, env, db, cfg, llm): + turn_on(client, env["own"]) + llm.usage = {"prompt_tokens": 40, "completion_tokens": 8, "total_tokens": 48} + ask(client, env["vi"]) + ref = f"space:{env['pods']}" + assert budgets.status(db, ref, {"tokens": 1000, "period": "month"})["spent"]["tokens"] >= 48 + budgets.put(db, ref, tokens=10) + err = ask(client, env["vi"], status=429)["error"] + assert err["code"] == "insufficient_quota" and "budget" in err["message"] + + +def test_agents_turn_it_on(client, env): + from tests.api.test_mcp_namespace import call + + is_err, got = call( + client, env["own"], "set_namespace_integrations", server="/mcp", namespace="pods", chat=True, tools=["list_recordings"] + ) + assert not is_err and (got["chat"], got["tools"]) == (True, ["list_recordings"]) and got["chat_url"].endswith("/api/v1/namespaces/pods") + is_err, got = call(client, env["vi"], "namespace_integrations", server="/mcp", namespace="pods") + assert not is_err and got["chat"] is True and "list_recordings" in [t["name"] for t in got["tool_choices"]] diff --git a/nextjs-frontend/__tests__/connect-apps.test.tsx b/nextjs-frontend/__tests__/connect-apps.test.tsx index 9b9c71e77..2bafb7005 100644 --- a/nextjs-frontend/__tests__/connect-apps.test.tsx +++ b/nextjs-frontend/__tests__/connect-apps.test.tsx @@ -13,7 +13,11 @@ jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: const ok = (data: unknown) => Promise.resolve({ data, response: { ok: true, status: 200 } }); const m = (f: unknown) => f as jest.Mock; -const OFF = { mcp: false, mcp_url: "http://localhost:3000/mcp/pods" }; +const CHOICES = [ + { name: "search_transcripts", description: "Search the transcripts" }, + { name: "list_recordings", description: "List recordings" }, +]; +const OFF = { mcp: false, mcp_url: "http://localhost:3000/mcp/pods", chat: false, tools: [], tool_choices: CHOICES }; function show(isOwner = true) { const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } }); @@ -52,6 +56,34 @@ test("people who don't own the namespace see whether it's on, and who can change expect(screen.getByText(`${window.location.origin}/mcp/pods`)).toBeInTheDocument(); }); +test("owners turn on chat completions, copy the base URL and pick the tools the model may use", async () => { + const on = { ...OFF, chat: true }; + m(Namespaces.updateNamespaceIntegrations) + .mockReturnValueOnce(ok(on)) + .mockReturnValueOnce(ok({ ...on, tools: ["list_recordings"] })); + show(); + const sw = await screen.findByRole("switch", { name: "pods’s chat completions" }); + expect(screen.queryByRole("checkbox")).not.toBeInTheDocument(); // off: nothing to pick + fireEvent.click(sw); + await waitFor(() => + expect(Namespaces.updateNamespaceIntegrations).toHaveBeenCalledWith( + expect.objectContaining({ path: { name: "pods" }, body: { chat: true } }), + ), + ); + expect(await screen.findByText(`${window.location.origin}/api/v1/namespaces/pods`)).toBeInTheDocument(); + expect(screen.getByRole("button", { name: "Copy Base URL" })).toBeInTheDocument(); + expect(screen.getByRole("link", { name: "API key" })).toHaveAttribute("href", "/account/tokens"); + const box = screen.getByRole("checkbox", { name: "list recordings" }); + expect(box).not.toBeChecked(); // none by default: one search per question + fireEvent.click(box); + await waitFor(() => + expect(Namespaces.updateNamespaceIntegrations).toHaveBeenLastCalledWith( + expect.objectContaining({ body: { tools: ["list_recordings"] } }), + ), + ); + await waitFor(() => expect(screen.getByRole("checkbox", { name: "list recordings" })).toBeChecked()); +}); + test("an app signing in for one namespace's server is asking for that namespace alone", () => { expect(namespaceOf("https://lens.example.org/mcp/pods")).toBe("pods"); expect(namespaceOf("https://lens.example.org/mcp/pods/")).toBe("pods"); diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index c2f07c530..1dc06c894 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -8,6 +8,7 @@ export { Batches, Browser, Budgets, + ChatCompletions, Chats, Collections, Comments, @@ -264,6 +265,8 @@ export type { ChatCapabilitiesData, ChatCapabilitiesResponse, ChatCapabilitiesResponses, + ChatCompletion, + ChatCompletionRequest, ChatCreate, ChatMessage, ChatScope, @@ -328,6 +331,12 @@ export type { Comment, CommentCreate, CommentUpdate, + CompletionChoice, + CompletionLens, + CompletionMessage, + CompletionModel, + CompletionModels, + CompletionSource, Component, Components, ComponentState, @@ -370,6 +379,11 @@ export type { CreateBatchErrors, CreateBatchResponse, CreateBatchResponses, + CreateChatCompletionData, + CreateChatCompletionError, + CreateChatCompletionErrors, + CreateChatCompletionResponse, + CreateChatCompletionResponses, CreateChatData, CreateChatError, CreateChatErrors, @@ -1485,6 +1499,7 @@ export type { InstallMarketplacePluginResponse, InstallMarketplacePluginResponses, InstallStatus, + IntegrationTool, IpGroup, IpGroupCreate, IpGroups, @@ -1580,6 +1595,11 @@ export type { ListCommentsErrors, ListCommentsResponse, ListCommentsResponses, + ListCompletionModelsData, + ListCompletionModelsError, + ListCompletionModelsErrors, + ListCompletionModelsResponse, + ListCompletionModelsResponses, ListComponentsData, ListComponentsResponse, ListComponentsResponses, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index cfdb7fdb0..f08c7255f 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -139,6 +139,9 @@ import type { CreateBatchData, CreateBatchErrors, CreateBatchResponses, + CreateChatCompletionData, + CreateChatCompletionErrors, + CreateChatCompletionResponses, CreateChatData, CreateChatErrors, CreateChatResponses, @@ -784,6 +787,9 @@ import type { ListCommentsData, ListCommentsErrors, ListCommentsResponses, + ListCompletionModelsData, + ListCompletionModelsErrors, + ListCompletionModelsResponses, ListComponentsData, ListComponentsResponses, ListContentTypesData, @@ -4257,7 +4263,7 @@ export class Namespaces { /** * Get Namespace Integrations * - * Whether the namespace's own MCP server is on, and its address. + * Whether the namespace's own MCP server and chat completions are on, their addresses, and the tools offered. */ public static getNamespaceIntegrations( options: Options, @@ -4272,8 +4278,8 @@ export class Namespaces { /** * Update Namespace Integrations * - * Owners: turn the namespace's own MCP server on or off. Off, it answers 403; apps keep their access to the rest of - * Lens as before. + * Owners: turn the namespace's own MCP server or chat completions on or off, or pick the read tools the model may + * use in its chat completions. Off, each answers 403; apps keep their access to the rest of Lens as before. */ public static updateNamespaceIntegrations( options: Options, @@ -4423,6 +4429,45 @@ export class Namespaces { } } +export class ChatCompletions { + /** + * List Completion Models + * + * OpenAI's model list for the namespace's chat completions: lens (the configured model) and the models people may + * pick in Lens's chat. + */ + public static listCompletionModels( + options: Options, + ): RequestResult { + return (options.client ?? client).get({ + url: "/api/v1/namespaces/{name}/models", + ...options, + }); + } + + /** + * Create Chat Completion + * + * OpenAI's chat completions, answered from this namespace alone: the last user message is the question, the + * messages before it the conversation and system messages the app's instructions. The answer cites what it read as + * [n], with a Sources list after it, `citations` (the n-th link for [n]) and `lens.sources`. With `stream`, it comes + * as OpenAI's chunks (server-sent events) ending in `data: [DONE]`. Off until an owner of the namespace turns it on; + * read-only tokens are enough. 429 when the namespace's budget or Lens's overall one is used up. + */ + public static createChatCompletion( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/namespaces/{name}/chat/completions", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } +} + export class Resources { /** * List Recordings diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index cd6ec1799..bf641f001 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -2003,6 +2003,88 @@ export type ChatCapabilities = { [key: string]: unknown; }; +/** + * ChatCompletion + */ +export type ChatCompletion = { + /** + * Id + */ + id: string; + /** + * Object + */ + object?: string; + /** + * Created + */ + created: number; + /** + * Model + */ + model: string; + /** + * Choices + */ + choices: Array; + /** + * Citations + * + * the links of what it can cite, in order: [n] is the n-th + */ + citations: Array; + lens: CompletionLens; + [key: string]: unknown; +}; + +/** + * ChatCompletionRequest + * + * OpenAI's request. Fields it has that aren't listed here (n, top_p, user, max_tokens, tools, …) are taken and + * left alone. + */ +export type ChatCompletionRequest = { + /** + * Messages + * + * the conversation: system, user, assistant (and tool) messages; the last is the question + */ + messages: Array<{ + [key: string]: unknown; + }>; + /** + * Model + * + * lens (the configured model), or one people may pick in Lens's chat; anything else is lens + */ + model?: string | null; + /** + * Stream + * + * send the answer as server-sent events, OpenAI's chunks, ending in data: [DONE] + */ + stream?: boolean; + /** + * Temperature + */ + temperature?: number | null; + /** + * Stream Options + */ + stream_options?: { + [key: string]: unknown; + } | null; + /** + * Response Format + * + * any format asked for: the answer then has no Sources list after it (the format itself isn't enforced yet) + */ + response_format?: { + [key: string]: unknown; + } | null; + [key: string]: unknown; +}; + /** * ChatCreate */ @@ -2882,6 +2964,149 @@ export type CommentUpdate = { resolved?: boolean | null; }; +/** + * CompletionChoice + */ +export type CompletionChoice = { + /** + * Index + */ + index?: number; + message: CompletionMessage; + /** + * Finish Reason + */ + finish_reason?: string; + [key: string]: unknown; +}; + +/** + * CompletionLens + */ +export type CompletionLens = { + /** + * Namespace + */ + namespace: string; + /** + * Sources + */ + sources: Array; + /** + * Steps + * + * the tools it looked things up with + */ + steps?: Array<{ + [key: string]: unknown; + }>; + /** + * Notice + */ + notice?: string | null; + [key: string]: unknown; +}; + +/** + * CompletionMessage + */ +export type CompletionMessage = { + /** + * Role + */ + role?: string; + /** + * Content + */ + content: string; + [key: string]: unknown; +}; + +/** + * CompletionModel + */ +export type CompletionModel = { + /** + * Id + */ + id: string; + /** + * Object + */ + object?: string; + /** + * Created + */ + created?: number; + /** + * Owned By + */ + owned_by?: string; + [key: string]: unknown; +}; + +/** + * CompletionModels + */ +export type CompletionModels = { + /** + * Object + */ + object?: string; + /** + * Data + */ + data: Array; + [key: string]: unknown; +}; + +/** + * CompletionSource + */ +export type CompletionSource = { + /** + * N + * + * what the answer cites it as: [n] + */ + n: number; + /** + * Recording Id + * + * the asset it comes from + */ + recording_id: number; + /** + * Title + */ + title?: string | null; + /** + * Time + * + * the moment, as "12:34", or a document's page, as "p. 2" + */ + time?: string | null; + /** + * Speaker + * + * who spoke, in audio or video + */ + speaker?: string | null; + /** + * Url + * + * the moment in the web app + */ + url: string; + /** + * Cited + * + * whether the answer cites it + */ + cited: boolean; + [key: string]: unknown; +}; + /** * Component */ @@ -7216,6 +7441,21 @@ export type InstallStatus = { [key: string]: unknown; }; +/** + * IntegrationTool + */ +export type IntegrationTool = { + /** + * Name + */ + name: string; + /** + * Description + */ + description: string; + [key: string]: unknown; +}; + /** * IpGroup * @@ -9236,6 +9476,30 @@ export type NamespaceIntegrations = { * where MCP clients connect to it */ mcp_url: string; + /** + * Chat + * + * the namespace's own chat completions answer apps, from this namespace alone + */ + chat?: boolean; + /** + * Chat Url + * + * the base URL OpenAI-compatible apps are given (they add /chat/completions) + */ + chat_url?: string; + /** + * Tools + * + * the read tools the model may use there (none: one search) + */ + tools?: Array; + /** + * Tool Choices + * + * the tools that can be offered there + */ + tool_choices?: Array; /** * Updated At */ @@ -9257,6 +9521,14 @@ export type NamespaceIntegrationsUpdate = { * Mcp */ mcp?: boolean | null; + /** + * Chat + */ + chat?: boolean | null; + /** + * Tools + */ + tools?: Array | null; }; /** @@ -24988,6 +25260,114 @@ export type UpdateNamespaceIntegrationsResponses = { export type UpdateNamespaceIntegrationsResponse = UpdateNamespaceIntegrationsResponses[keyof UpdateNamespaceIntegrationsResponses]; +export type ListCompletionModelsData = { + body?: never; + path: { + /** + * Name + */ + name: string; + }; + query?: never; + url: "/api/v1/namespaces/{name}/models"; +}; + +export type ListCompletionModelsErrors = { + /** + * OpenAI's error shape: {error: {message, type, code}} + */ + 400: unknown; + /** + * no token, or one that doesn't open this namespace + */ + 401: unknown; + /** + * the namespace's chat completions are off + */ + 403: unknown; + /** + * no such namespace, or not one you can read + */ + 404: unknown; + /** + * Validation Error + */ + 422: HttpValidationError; + /** + * a budget is used up + */ + 429: unknown; + /** + * the model server failed + */ + 502: unknown; +}; + +export type ListCompletionModelsError = ListCompletionModelsErrors[keyof ListCompletionModelsErrors]; + +export type ListCompletionModelsResponses = { + /** + * Successful Response + */ + 200: CompletionModels; +}; + +export type ListCompletionModelsResponse = ListCompletionModelsResponses[keyof ListCompletionModelsResponses]; + +export type CreateChatCompletionData = { + body: ChatCompletionRequest; + path: { + /** + * Name + */ + name: string; + }; + query?: never; + url: "/api/v1/namespaces/{name}/chat/completions"; +}; + +export type CreateChatCompletionErrors = { + /** + * OpenAI's error shape: {error: {message, type, code}} + */ + 400: unknown; + /** + * no token, or one that doesn't open this namespace + */ + 401: unknown; + /** + * the namespace's chat completions are off + */ + 403: unknown; + /** + * no such namespace, or not one you can read + */ + 404: unknown; + /** + * Validation Error + */ + 422: HttpValidationError; + /** + * a budget is used up + */ + 429: unknown; + /** + * the model server failed + */ + 502: unknown; +}; + +export type CreateChatCompletionError = CreateChatCompletionErrors[keyof CreateChatCompletionErrors]; + +export type CreateChatCompletionResponses = { + /** + * Successful Response + */ + 200: ChatCompletion; +}; + +export type CreateChatCompletionResponse = CreateChatCompletionResponses[keyof CreateChatCompletionResponses]; + export type ListNamespaceCollectionsData = { body?: never; path: { diff --git a/nextjs-frontend/components/integrations/connect-apps-section.tsx b/nextjs-frontend/components/integrations/connect-apps-section.tsx index 99abc80e9..788b5f64e 100644 --- a/nextjs-frontend/components/integrations/connect-apps-section.tsx +++ b/nextjs-frontend/components/integrations/connect-apps-section.tsx @@ -2,12 +2,15 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { Plug } from "lucide-react"; +import Link from "next/link"; +import { useEffect, useState } from "react"; import { Namespaces } from "@/app/openapi-client"; +import type { NamespaceIntegrations, NamespaceIntegrationsUpdate } from "@/app/openapi-client"; import { connectHint, useAppSignIn, useMcpAddress } from "@/components/account/connect-assistant"; import { isUnreachable } from "@/components/errors/error-states"; import { Button } from "@/components/ui/button"; -import { Switch } from "@/components/ui/field"; +import { Checkbox, Switch } from "@/components/ui/field"; import { CodeBlock, EmptyState, SkeletonRows } from "@/components/ui/states"; import { useToast } from "@/components/ui/toast"; import { data, useApiClient } from "@/lib/api/browser"; @@ -15,34 +18,97 @@ import { needRole } from "@/lib/hooks/session"; export const integrationsKey = (ns: string) => ["namespace", ns, "integrations"] as const; +/** The base URL OpenAI-compatible apps are given, once the page is in the browser (the server can't know it). */ +function useChatAddress(ns: string): string { + const [url, setUrl] = useState(`/api/v1/namespaces/${ns}`); + useEffect(() => setUrl(`${window.location.origin}/api/v1/namespaces/${ns}`), [ns]); + return url; +} + +function toastFor(ns: string, body: NamespaceIntegrationsUpdate, got: NamespaceIntegrations, mcp: string) { + if (body.mcp != null) + return got.mcp + ? { + title: `${ns}’s MCP server is on`, + body: `Agents connected to ${mcp} see ${ns} alone.`, + tone: "green" as const, + } + : { title: `${ns}’s MCP server is off`, body: "Agents connected to it stop getting answers." }; + if (body.chat != null) + return got.chat + ? { + title: `${ns}’s chat completions are on`, + body: `Apps answer from ${ns} alone, citing it.`, + tone: "green" as const, + } + : { title: `${ns}’s chat completions are off`, body: "Apps using them stop getting answers." }; + const n = got.tools?.length ?? 0; + return { title: n ? `The model may use ${n} tool${n === 1 ? "" : "s"}` : "No tools: one search per question" }; +} + +function Row({ + title, + hint, + checked, + disabled, + onChange, + label, +}: { + title: string; + hint: string; + checked: boolean; + disabled: boolean; + onChange: (v: boolean) => void; + label: string; +}) { + return ( +
+
+ {title} + {hint} +
+ +
+ ); +} + /** - * A namespace's own ways in for other apps (docs/mcp.md, "One namespace's MCP server"): owners turn on an MCP server - * that sees this namespace alone, and its address is a copy away. + * A namespace's own ways in for other apps (docs/mcp.md, "One namespace's MCP server"; docs/api.md, "Chat + * completions for any app"): owners turn on an MCP server and OpenAI-style chat completions that see this namespace + * alone, pick the read tools the model may use there, and the addresses are a copy away. */ export function ConnectAppsSection({ ns, isOwner }: { ns: string; isOwner: boolean }) { const client = useApiClient(); const qc = useQueryClient(); const toast = useToast(); const mcp = `${useMcpAddress()}/${ns}`; + const chatUrl = useChatAddress(ns); const q = useQuery({ queryKey: integrationsKey(ns), queryFn: () => data(Namespaces.getNamespaceIntegrations({ client, path: { name: ns } })), }); const on = q.data?.mcp ?? false; + const chatOn = q.data?.chat ?? false; + const tools = q.data?.tools ?? []; const signIn = useAppSignIn(on); const save = useMutation({ - mutationFn: (mcpOn: boolean) => - data(Namespaces.updateNamespaceIntegrations({ client, path: { name: ns }, body: { mcp: mcpOn } })), - onSuccess: (got) => { + mutationFn: (body: NamespaceIntegrationsUpdate) => + data(Namespaces.updateNamespaceIntegrations({ client, path: { name: ns }, body })), + onSuccess: (got, body) => { qc.setQueryData(integrationsKey(ns), got); - toast( - got.mcp - ? { title: `${ns}’s MCP server is on`, body: `Agents connected to ${mcp} see ${ns} alone.`, tone: "green" } - : { title: `${ns}’s MCP server is off`, body: "Agents connected to it stop getting answers." }, - ); + toast(toastFor(ns, body, got, mcp)); }, onError: (e) => toast({ title: "Couldn’t change it", body: (e as Error).message, tone: "red" }), }); + const owners = needRole("owner", ns); + const pick = (name: string, v: boolean) => + save.mutate({ tools: v ? [...tools, name] : tools.filter((t) => t !== name) }); return (
{q.isPending ? ( - + ) : q.isError ? ( ) : ( -
-
-
- MCP server - - {isOwner + <> +
+ -
- save.mutate(v)} - label={on ? "On" : "Off"} - aria-label={`${ns}’s MCP server`} + onChange={(v) => save.mutate({ mcp: v })} + label={`${ns}’s MCP server`} + /> + {on && ( + <> + +

{connectHint(signIn.data)}

+ + )} +
+
+ save.mutate({ chat: v })} + label={`${ns}’s chat completions`} /> + {chatOn && ( + <> + +

+ Give the app this base URL, the model lens, and an{" "} + + API key + {" "} + as its key. Answers cite their sources as [n], with links after them. +

+
+ Tools the model may use +

+ None: one search, one model call per question. Each tool lets the model look more up in {ns} first, + which takes more model calls. They only read. +

+
+ {(q.data?.tool_choices ?? []).map((t) => ( + + pick(t.name, v)} + label={t.name.replaceAll("_", " ")} + /> + + ))} +
+
+ + )}
- {on && ( - <> - -

{connectHint(signIn.data)}

- - )} -
+ )}
); diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 54b7b741a..480743340 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -6391,7 +6391,7 @@ "namespaces" ], "summary": "Get Namespace Integrations", - "description": "Whether the namespace's own MCP server is on, and its address.", + "description": "Whether the namespace's own MCP server and chat completions are on, their addresses, and the tools offered.", "operationId": "get_namespace_integrations", "parameters": [ { @@ -6432,7 +6432,7 @@ "namespaces" ], "summary": "Update Namespace Integrations", - "description": "Owners: turn the namespace's own MCP server on or off. Off, it answers 403; apps keep their access to the rest of\nLens as before.", + "description": "Owners: turn the namespace's own MCP server or chat completions on or off, or pick the read tools the model may\nuse in its chat completions. Off, each answers 403; apps keep their access to the rest of Lens as before.", "operationId": "update_namespace_integrations", "parameters": [ { @@ -6479,6 +6479,138 @@ } } }, + "/api/v1/namespaces/{name}/models": { + "get": { + "tags": [ + "chat completions" + ], + "summary": "List Completion Models", + "description": "OpenAI's model list for the namespace's chat completions: lens (the configured model) and the models people may\npick in Lens's chat.", + "operationId": "list_completion_models", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string", + "title": "Name" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CompletionModels" + } + } + } + }, + "400": { + "description": "OpenAI's error shape: {error: {message, type, code}}" + }, + "401": { + "description": "no token, or one that doesn't open this namespace" + }, + "403": { + "description": "the namespace's chat completions are off" + }, + "404": { + "description": "no such namespace, or not one you can read" + }, + "429": { + "description": "a budget is used up" + }, + "502": { + "description": "the model server failed" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/namespaces/{name}/chat/completions": { + "post": { + "tags": [ + "chat completions" + ], + "summary": "Create Chat Completion", + "description": "OpenAI's chat completions, answered from this namespace alone: the last user message is the question, the\nmessages before it the conversation and system messages the app's instructions. The answer cites what it read as\n[n], with a Sources list after it, `citations` (the n-th link for [n]) and `lens.sources`. With `stream`, it comes\nas OpenAI's chunks (server-sent events) ending in `data: [DONE]`. Off until an owner of the namespace turns it on;\nread-only tokens are enough. 429 when the namespace's budget or Lens's overall one is used up.", + "operationId": "create_chat_completion", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string", + "title": "Name" + } + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChatCompletionRequest" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ChatCompletion" + } + } + } + }, + "400": { + "description": "OpenAI's error shape: {error: {message, type, code}}" + }, + "401": { + "description": "no token, or one that doesn't open this namespace" + }, + "403": { + "description": "the namespace's chat completions are off" + }, + "404": { + "description": "no such namespace, or not one you can read" + }, + "429": { + "description": "a budget is used up" + }, + "502": { + "description": "the model server failed" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/namespaces/{name}/collections": { "get": { "tags": [ @@ -31408,6 +31540,133 @@ ], "title": "ChatCapabilities" }, + "ChatCompletion": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "object": { + "type": "string", + "title": "Object", + "default": "chat.completion" + }, + "created": { + "type": "integer", + "title": "Created" + }, + "model": { + "type": "string", + "title": "Model" + }, + "choices": { + "items": { + "$ref": "#/components/schemas/CompletionChoice" + }, + "type": "array", + "title": "Choices" + }, + "citations": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Citations", + "description": "the links of what it can cite, in order: [n] is the n-th" + }, + "lens": { + "$ref": "#/components/schemas/CompletionLens" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "created", + "model", + "choices", + "citations", + "lens" + ], + "title": "ChatCompletion" + }, + "ChatCompletionRequest": { + "properties": { + "messages": { + "items": { + "additionalProperties": true, + "type": "object" + }, + "type": "array", + "maxItems": 1000, + "title": "Messages", + "description": "the conversation: system, user, assistant (and tool) messages; the last is the question" + }, + "model": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model", + "description": "lens (the configured model), or one people may pick in Lens's chat; anything else is lens" + }, + "stream": { + "type": "boolean", + "title": "Stream", + "description": "send the answer as server-sent events, OpenAI's chunks, ending in data: [DONE]", + "default": false + }, + "temperature": { + "anyOf": [ + { + "type": "number", + "maximum": 2.0, + "minimum": 0.0 + }, + { + "type": "null" + } + ], + "title": "Temperature" + }, + "stream_options": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Stream Options" + }, + "response_format": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Response Format", + "description": "any format asked for: the answer then has no Sources list after it (the format itself isn't enforced yet)" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "messages" + ], + "title": "ChatCompletionRequest", + "description": "OpenAI's request. Fields it has that aren't listed here (n, top_p, user, max_tokens, tools, \u2026) are taken and\nleft alone." + }, "ChatCreate": { "properties": { "title": { @@ -33040,6 +33299,209 @@ "type": "object", "title": "CommentUpdate" }, + "CompletionChoice": { + "properties": { + "index": { + "type": "integer", + "title": "Index", + "default": 0 + }, + "message": { + "$ref": "#/components/schemas/CompletionMessage" + }, + "finish_reason": { + "type": "string", + "title": "Finish Reason", + "default": "stop" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "message" + ], + "title": "CompletionChoice" + }, + "CompletionLens": { + "properties": { + "namespace": { + "type": "string", + "title": "Namespace" + }, + "sources": { + "items": { + "$ref": "#/components/schemas/CompletionSource" + }, + "type": "array", + "title": "Sources" + }, + "steps": { + "items": { + "additionalProperties": true, + "type": "object" + }, + "type": "array", + "title": "Steps", + "description": "the tools it looked things up with" + }, + "notice": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Notice" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "namespace", + "sources" + ], + "title": "CompletionLens" + }, + "CompletionMessage": { + "properties": { + "role": { + "type": "string", + "title": "Role", + "default": "assistant" + }, + "content": { + "type": "string", + "title": "Content" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "content" + ], + "title": "CompletionMessage" + }, + "CompletionModel": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "object": { + "type": "string", + "title": "Object", + "default": "model" + }, + "created": { + "type": "integer", + "title": "Created", + "default": 0 + }, + "owned_by": { + "type": "string", + "title": "Owned By", + "default": "lens" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id" + ], + "title": "CompletionModel" + }, + "CompletionModels": { + "properties": { + "object": { + "type": "string", + "title": "Object", + "default": "list" + }, + "data": { + "items": { + "$ref": "#/components/schemas/CompletionModel" + }, + "type": "array", + "title": "Data" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "data" + ], + "title": "CompletionModels" + }, + "CompletionSource": { + "properties": { + "n": { + "type": "integer", + "title": "N", + "description": "what the answer cites it as: [n]" + }, + "recording_id": { + "type": "integer", + "title": "Recording Id", + "description": "the asset it comes from" + }, + "title": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Title" + }, + "time": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Time", + "description": "the moment, as \"12:34\", or a document's page, as \"p. 2\"" + }, + "speaker": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Speaker", + "description": "who spoke, in audio or video" + }, + "url": { + "type": "string", + "title": "Url", + "description": "the moment in the web app" + }, + "cited": { + "type": "boolean", + "title": "Cited", + "description": "whether the answer cites it" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "n", + "recording_id", + "url", + "cited" + ], + "title": "CompletionSource" + }, "Component": { "properties": { "id": { @@ -40360,6 +40822,25 @@ ], "title": "InstallStatus" }, + "IntegrationTool": { + "properties": { + "name": { + "type": "string", + "title": "Name" + }, + "description": { + "type": "string", + "title": "Description" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "name", + "description" + ], + "title": "IntegrationTool" + }, "IpGroup": { "properties": { "id": { @@ -43771,6 +44252,34 @@ "title": "Mcp Url", "description": "where MCP clients connect to it" }, + "chat": { + "type": "boolean", + "title": "Chat", + "description": "the namespace's own chat completions answer apps, from this namespace alone", + "default": false + }, + "chat_url": { + "type": "string", + "title": "Chat Url", + "description": "the base URL OpenAI-compatible apps are given (they add /chat/completions)", + "default": "" + }, + "tools": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Tools", + "description": "the read tools the model may use there (none: one search)" + }, + "tool_choices": { + "items": { + "$ref": "#/components/schemas/IntegrationTool" + }, + "type": "array", + "title": "Tool Choices", + "description": "the tools that can be offered there" + }, "updated_at": { "anyOf": [ { @@ -43814,6 +44323,32 @@ } ], "title": "Mcp" + }, + "chat": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Chat" + }, + "tools": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array", + "maxItems": 50 + }, + { + "type": "null" + } + ], + "title": "Tools" } }, "additionalProperties": false, diff --git a/threat-model.json b/threat-model.json index 6c954e818..0244de49b 100644 --- a/threat-model.json +++ b/threat-model.json @@ -1003,7 +1003,7 @@ "property": "P-RETRIEVAL", "violation_symptom": "info-leak, hang", "severity_tier": "security", - "conditions": "chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at ai.max_steps and ai.max_transcript_reads", + "conditions": "chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at ai.max_steps and ai.max_transcript_reads; a namespace's chat completions answer from that namespace alone, with only the read tools its owners offered", "provenance": "documented", "source": "docs/assistant.md \"limited to what the person can read\"" }, diff --git a/threat-model.md b/threat-model.md index 13fcf1842..14535ca8f 100644 --- a/threat-model.md +++ b/threat-model.md @@ -273,6 +273,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /sources/rclone-signin`, `/sources/rclone-signin/{id}` (signing a source in through rclone) | `type`, `params`, `address` (`state`, `code`, `error`), the token rclone prints | yes; admins start it; while it waits (ten minutes, one at a time) rclone listens on 127.0.0.1:53682 on the server, so anything running there can reach it, and the pasted address is passed on only to that port with only its state, code and error, when the state is this sign-in's | data, x-collaborator-data | the person pastes only the address their own sign-in ended on; the host keeps 127.0.0.1:53682 to itself | *(documented, docs/processing.md "Lens passes only that address's state, code and")* | | `/mcp` | bearer, `Origin`, JSON-RPC body | yes | data, size/rate | none | *(documented, docs/mcp.md "A token an app asked for another server")* | | `/mcp/` (a namespace's own MCP server, off until its owner turns it on) | bearer, `Origin`, JSON-RPC body, the namespace in the path | yes | data, size/rate, x-mcp-audience | the person's roles narrowed to that namespace (`api/deps.py` `within`): no administration, no server tools; a token issued for it opens nothing else (`bound_to`) | *(documented, docs/mcp.md "One namespace's MCP server")* | +| `/api/v1/namespaces//chat/completions` and `…/models` (a namespace's chat completions, off until its owner turns it on) | bearer, OpenAI-style JSON body (messages, model, stream), the namespace in the path | yes | data, size/rate | the person's roles narrowed to that namespace (`api/deps.py` `within`); one search there and only the read tools its owners offered (`domain/completions.py` `Box`); the namespace's budget and the overall hard limit stop model calls | *(documented, docs/api.md "Chat completions for any app")* | | namespace, recording and child-object routes | namespace name, `rid`, child ids (sequential integers) | yes | resource-name | none; Lens checks the role in each handler | *(documented, docs/api.md "Namespaces you can't read answer 404")* | | routes not read one by one (about 370) | per route | per gate | per route | follow the gate pattern | *(inferred, Q8)* | | media, report, embed, IIIF content | `exp`, `sig`, `full`, `?s=` share token, `/s/` short code | yes (bearer secrets) | serialized state | holder keeps the link private | *(documented, SECURITY.md "only the server's own signed links should work")* | @@ -511,7 +512,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-ADMIN-GUARD | Guards Lens documents on admin-set operands hold. iCal feeds are fetched like web captures: public addresses only (and `documents.web_networks`), ports 80 and 443, and the password never goes on to a server the calendar redirects to *(documented, docs/processing.md "fetched the way web pages are captured")*. The telemetry endpoint cannot carry a user name or password or be a link-local address *(documented, domain/settings.py validation message "can't be a link-local address")*. | info-leak | security-critical | *(documented, docs/processing.md "fetched the way web pages are captured")* | Voided by: `documents.web_networks` widens the iCal guard — `domain/feeds.py:377`; telemetry checks — `domain/settings.py:751`, `domain/settings.py:757`. | | P-NO-FETCH | SPARQL refuses SERVICE and FROM and never loads a named graph. RDF import fetches nothing, refuses remote JSON-LD contexts and does not read RDF/XML. | info-leak | security-critical | *(documented, rdf_import.py docstring "Nothing is fetched while reading")* | Voided by: Process-global rdflib flag, set per call — `domain/rdf.py:576`; SERVICE refused — `domain/rdf.py:551`; remote contexts — `domain/rdf_import.py:54-56`. | | P-QUERY | Cypher is read-only and runs on the caller's readable namespaces. A query stops within 3 million steps or 8 seconds; paths run up to 8 hops. Threshold: running past the budget is a bug; slow within it is not. | integrity-bypass (writes, scope); hang (budget) | security-critical (scope, writes); correctness-only (budget) | *(documented, docs/graph.md "3 million steps")* | Voided by: Admins see every namespace; write refusals — `domain/cypher.py:39`; budgets — `domain/cypher.py:35-38`. | -| P-RETRIEVAL | Chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at `ai.max_steps` and `ai.max_transcript_reads`. | info-leak (scope); hang (caps) | security-critical (scope); correctness-only (caps) | *(documented, docs/assistant.md "limited to what the person can read")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | +| P-RETRIEVAL | Chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at `ai.max_steps` and `ai.max_transcript_reads`. A namespace's chat completions answer from that namespace alone, with only the read tools its owners offered. | info-leak (scope); hang (caps) | security-critical (scope); correctness-only (caps) | *(documented, docs/assistant.md "limited to what the person can read")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | | P-APPROVAL | Assistant tools that change data wait for the person's approval, except notes and an admin's setup conversation. Inviting people and changing their access always wait, even there, and the assistant cannot remove an account. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Approvals")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | | P-INJECTION-EGRESS | Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no `bridge.users`, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own `read` web tools, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is `VALID-HARDENING` (Q26). | info-leak | correctness-only | *(documented, docs/assistant.md "it read out of Lens waits for your approval")* | Voided by: `ai.untrusted_asks: false` — `domain/ai_tools.py:519`; `bridge.safe_links: false` — `domain/bridge.py:298`. Notes and memories are written at once: they stay in Lens. Hooks run as their owners set them (P-EXT-SCOPE). An address the read text wrote itself stays linkable in a room. The person's browser asks first as P-BROWSER-GUARD says. | | P-EXT-SCOPE | Your own hooks run in your conversations; another person's hooks run only when an admin shared them. | integrity-bypass | security-critical | *(documented, docs/assistant.md "other people's run only when an admin shared them")* | Voided by: An admin sharing the extension — `domain/extensions.py:595-598`. | diff --git a/threat-model.yaml b/threat-model.yaml index d49f7e4c1..f3caccaf0 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:d4b4de082eb743e6dabfa0c671c660189b49a31ac1d394946a9b8ad134876bf1" +prose_version: "threat-model.md@sha256:03600976ceb6965dd3d8312d0efab441e0d524e24ea3abe8ad0fac69a66eafbd" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -1434,7 +1434,7 @@ properties_claimed: components: [assistant, authz] tier: security-critical x-tier-detail: "security-critical (scope); correctness-only (caps)" - conditions: "chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at ai.max_steps and ai.max_transcript_reads" + conditions: "chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at ai.max_steps and ai.max_transcript_reads; a namespace's chat completions answer from that namespace alone, with only the read tools its owners offered" violation_symptoms: [info-leak, hang] x-voided-by: "admins read everything; an admin setup conversation adds server tools and the act flag (domain/ai_tools.py:287-289); ai.extensions: false turns extensions off" provenance: {kind: documented, source: 'docs/assistant.md "limited to what the person can read"'}