diff --git a/docs/assistant.md b/docs/assistant.md index f5cad029a..45ebfc39d 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -49,7 +49,13 @@ then. the chat's model does. A long stretch folded at once (`/compact`, or a conversation from before summaries) is written in as many calls as it takes to fit that model's context, each in the [activity ledger](activity.md) as the conversation's. Settings › AI assistant › Summarise long conversations (`ai.compact`, on) turns it off; off, or with - no model, only the last six messages are read. + no model, only the last six messages are read. When the model server still says a request is too long for the + model (its context was smaller than Lens knew), the answer is tried again rather than failed: what tools returned + in that answer is shortened first (each result keeps its start and end), then all but the last two messages are + folded into the summary now and half of the excerpts are left out; the context the server named is kept as the + model's, so the next conversation is summarised before it fills. In the web app, a line under the last message a + summary covers says how many messages it stands for (or that `/clear` stopped the model reading them), and Show + summary opens the summary; `GET /api/v1/chats/` gives it as `summary` (`text`, `upto`, `covers`). - **Titles:**{#titles} a conversation nobody named is titled with its first question's words as soon as it's asked, then, after the first answer, the model gives it a short title of a few words from the question and the answer (streamed as `title` at the end of the answer's stream). It's one small call, written by **Model for summaries** diff --git a/docs/requirements.md b/docs/requirements.md index a7f3a2b3c..a6f273dd1 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -86,7 +86,7 @@ request, to another check that proves the same thing; loosening a requirement or | Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` | | Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` | | Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` | -| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app` | +| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again`, `test_what_tools_returned_is_cut_to_fit_the_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app`; [chat-summary-mark.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-summary-mark.test.tsx) `says how many messages the summary stands for, and shows it on request` | | A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` | | Telegram and Slack built in, with voice notes and files | [test_telegram.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_telegram.py) `test_it_answers_private_chats_and_what_is_said_to_it_in_groups`, `test_voice_notes_are_heard`, `test_files_are_held_for_the_assistant_to_import`; [test_slack.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_slack.py) `test_it_answers_direct_messages_mentions_and_the_slash_command`, `test_voice_clips_are_heard_and_files_held` | | Other chat rooms through Matterbridge | [test_bridge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_bridge.py) `test_it_answers_what_is_said_to_it_and_nothing_else`; [test_matterbridge_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_matterbridge_run.py) `test_chat_networks_are_set_in_the_app_and_become_matterbridges_config`, `test_the_bridge_talks_to_the_matterbridge_lens_runs`; [matterbridge.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/matterbridge.yml) `Lens writes a config, and Matterbridge starts with it` | diff --git a/fastapi_backend/app/api/v1/routes/chats.py b/fastapi_backend/app/api/v1/routes/chats.py index 8c64eaaa0..20806980c 100644 --- a/fastapi_backend/app/api/v1/routes/chats.py +++ b/fastapi_backend/app/api/v1/routes/chats.py @@ -219,7 +219,8 @@ def _chat_view(db: DB, acl: Access, c: dict[str, Any]) -> Chat: for m in msgs: m["versions"] = versions.get(m["id"]) _readable_citations(db, acl, msgs) - return Chat.model_validate(sign_urls({**c, "messages": msgs, "tree": nodes}, full=True)) + view = {**c, "messages": msgs, "tree": nodes, "summary": chat.covered(db, c["id"], msgs)} + return Chat.model_validate(sign_urls(view, full=True)) def _readable_citations(db: DB, acl: Access, msgs: list[dict[str, Any]]) -> None: @@ -681,6 +682,10 @@ def stopped(text: str, cited: list[dict[str, Any]]) -> Iterator[str]: yield _ev("stopped", {"message": "Stopped"}) yield from save(text or "(stopped)", cited, "stopped", stopped=True) + def shrink() -> tuple[list[dict[str, Any]], dict[str, Any] | None]: + """The model said the conversation doesn't fit its context: fold it into the summary now (chat.shrink).""" + return chat.shrink(db, cfg, cid, past) + def answer(on: chat.Answering) -> Iterator[str]: nonlocal notice if llm.configured(cfg) and cfg["ai"].get("tools") and tools != []: # given no tools, it answers from a search @@ -708,7 +713,17 @@ def answer(on: chat.Answering) -> Iterator[str]: # setting the server up, or working in the person's browser, takes more steps than looking things up steps_max = max(cfg["ai"].get("max_steps") or 6, 12 if setup else 0, browser.steps(cfg) if box.browser else 0) answering = chat.tool_answer( - cfg, box, asked, past, steps_max, model, setup=setup, summary=summary, record=record, max_tool_calls=limit + cfg, + box, + asked, + past, + steps_max, + model, + setup=setup, + summary=summary, + record=record, + max_tool_calls=limit, + shrink=shrink, ) for kind, data in answering: if kind == "step": @@ -745,7 +760,7 @@ def answer(on: chat.Answering) -> Iterator[str]: text, error = "", None try: if llm.configured(cfg): - pieces = llm.stream_chat(cfg, chat.messages_for(asked, passages, past, summary, cfg), model=model) + pieces = chat.stream_fitted(cfg, asked, passages, past, summary, model=model, shrink=shrink) for piece in pieces: text += piece yield _ev("token", {"text": piece}) diff --git a/fastapi_backend/app/domain/bridge.py b/fastapi_backend/app/domain/bridge.py index df6978c2c..093fb0f2f 100644 --- a/fastapi_backend/app/domain/bridge.py +++ b/fastapi_backend/app/domain/bridge.py @@ -375,6 +375,10 @@ def _answer(db, cfg, base, account, cid, q, files, act_on, outside): steps, approvals, notice = [], [], None wrote = cfg["llm"].get("model") if llm.configured(cfg) else None + def shrink(): + """The model said the conversation doesn't fit its context: fold it into the summary now (chat.shrink).""" + return chat.shrink(db, cfg, cid, past) + def said(words, cited, outcome, **kw): """The answer, saved with its record.""" meta = record.meta(db, outcome, spent) @@ -397,7 +401,8 @@ def said(words, cited, outcome, **kw): ) try: text, how = None, "answered" - for kind, data in chat.tool_answer(cfg, box, asked, past, cfg["ai"].get("max_steps") or 6, summary=summary, record=record): + steps_max = cfg["ai"].get("max_steps") or 6 + for kind, data in chat.tool_answer(cfg, box, asked, past, steps_max, summary=summary, record=record, shrink=shrink): if kind == "step": steps.append(data) elif kind == "direct" and passages and not box.cited(data): # unless it cites what it remembers @@ -415,7 +420,7 @@ def said(words, cited, outcome, **kw): return "I couldn't answer that: " + str(e), box.approvals, keep | box.links error = None try: - text = llm.chat(cfg, chat.messages_for(asked, passages, past, summary, cfg)) if llm.configured(cfg) else chat.fallback(passages) + text = chat.reply_fitted(cfg, asked, passages, past, summary, shrink=shrink) if llm.configured(cfg) else chat.fallback(passages) except llm.LLMError as e: text, error = "I couldn't answer that: " + str(e), str(e) how = "error" if error else "fallback" if notice or not llm.configured(cfg) else "answered" diff --git a/fastapi_backend/app/domain/chat.py b/fastapi_backend/app/domain/chat.py index 273edc23b..b0fab6035 100644 --- a/fastapi_backend/app/domain/chat.py +++ b/fastapi_backend/app/domain/chat.py @@ -8,6 +8,7 @@ from __future__ import annotations +import contextlib import contextvars import json import logging @@ -598,6 +599,15 @@ def memory(db, cfg, cid, line=None): return {**had, "window": RECENT} if had.get("upto") else None +def covered(db, cid, line): + """The summary as the app shows it: the one stored for the version `line` is, with "covers", how many of its + messages it stands for; None before the first summary (or /clear).""" + had = _stored(db, cid, line) + if not had.get("upto"): + return None + return {**had, "covers": sum(1 for m in line if m["id"] <= had["upto"])} + + def summary_note(summary): text = ((summary or {}).get("text") or "").strip() return ( @@ -652,11 +662,12 @@ def _batches(cfg, model, had, old): yield run -def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None): +def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None, keep=None): """Fold the messages older than the latest RECENT into the conversation's summary, once COMPACT_AFTER of them have built up or they fill ai.compact_at % of the chat model's context window (window; `full_only`: only then; ai.compact, on by default; needs a model). `force` (/compact) folds every message in now, whatever the setting, - with the person's `focus` (what to keep) if any. The summary is written by ai.compact_model when set, a cheaper + with the person's `focus` (what to keep) if any, but the latest `keep` (shrink). The summary is written by + ai.compact_model when set, a cheaper model than the chat's, in as many calls as it takes to fit that model's window (_batches), each in the ledger as the conversation's. It's of the version `line` is (by default the head's), afresh when the summary was another version's. Returns the new summary (with "folded": how many messages), or None when nothing changed.""" @@ -665,7 +676,8 @@ def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None): line = path(db, cid) if line is None else line had = _stored(db, cid, line) fresh = [m for m in line if m["id"] > (had.get("upto") or 0)] # ids grow along a version - old = fresh if force else fresh[:-RECENT] + keep = (0 if force else RECENT) if keep is None else keep + old = fresh[:-keep] if keep else fresh if not old: return None if not force and (full_only or len(fresh) < RECENT + COMPACT_AFTER): @@ -696,6 +708,81 @@ def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None): return {**out, "folded": folded} +# ---------- fitting an answer's request into the model's context ---------- +FIT_KEEP = 2 # the latest messages still read word for word once a conversation didn't fit the model +MIN_RESULT = 400 # characters of a tool's result kept, however little room is left + + +def shrink(db, cfg, cid, line): + """What to answer with once the model said the conversation doesn't fit its context: (the messages, the summary), + with all but the last FIT_KEEP messages of the version `line` folded into the summary now, whatever the count. With + summaries off (ai.compact), or when one can't be written, the last FIT_KEEP messages alone, for this answer.""" + if cfg["ai"].get("compact", True): + try: + compact(db, cfg, cid, force=True, keep=FIT_KEEP, line=line) + except (llm.LLMError, activity.OverBudget): + log.warning("chat %s: couldn't fold the conversation into its summary to fit the model", cid, exc_info=True) + return list(line)[-FIT_KEEP:], memory(db, cfg, cid, line) + + +def room(cfg, model=None, tools=None): + """About how many tokens an answer's messages may take: the model's context window less room for the answer and + for the tools offered.""" + w = window(cfg, model) + return w - min(w // 4, 8192) - len(json.dumps(tools or [])) // 4 + + +def fit(msgs, budget): + """Shorten what tools returned, oldest first, until the messages take at most `budget` tokens: each loses its middle, + so the mark wrapped round outside text (untrusted.wrap) stays whole, and keeps at least MIN_RESULT characters. + Returns whether it cut anything.""" + over, cut = tokens(msgs) - budget, False + for m in msgs: + if over <= 0: + break + text = m.get("content") or "" + if m["role"] != "tool" or len(text) <= MIN_RESULT: + continue + keep = max(MIN_RESULT, len(text) - over * 4) + if keep >= len(text): + continue + m["content"] = ( + f"{text[: keep - keep // 4]}\n…({len(text) - keep} characters left out, to fit the model's context)…\n{text[-(keep // 4) :]}" + ) + over -= (len(text) - len(m["content"])) // 4 + cut = True + return cut + + +def stream_fitted(cfg, question, passages, history=(), summary=None, model=None, shrink=None): + """llm.stream_chat of messages_for. When the server says they don't fit the model's context before any of the answer + came, it's asked once more with the conversation shrunk (`shrink()`: (history, summary), see shrink) and the first + half of the excerpts.""" + try: + with contextlib.closing(llm.stream_chat(cfg, messages_for(question, passages, history, summary, cfg), model=model)) as pieces: + for piece in pieces: + yield piece + return + except llm.ContextFull: + if shrink is None: + raise + history, summary = shrink() + shorter = passages[: max(1, len(passages) // 2)] + with contextlib.closing(llm.stream_chat(cfg, messages_for(question, shorter, history, summary, cfg), model=model)) as pieces: + yield from pieces + + +def reply_fitted(cfg, question, passages, history=(), summary=None, model=None, shrink=None): + """llm.chat of messages_for, asked once more as stream_fitted is when the server says they don't fit.""" + try: + return llm.chat(cfg, messages_for(question, passages, history, summary, cfg), model=model) + except llm.ContextFull: + if shrink is None: + raise + history, summary = shrink() + return llm.chat(cfg, messages_for(question, passages[: max(1, len(passages) // 2)], history, summary, cfg), model=model) + + # ---------- commands: /clear and /compact, typed in the app or in a chat room ---------- COMMAND = re.compile(r"^/(clear|compact)(?:\s+(.*))?$", re.S | re.I) @@ -1052,37 +1139,65 @@ def _finish(cfg, msgs, tools, model): return text -def tool_answer(cfg, toolbox, question, history=(), max_steps=6, model=None, setup=False, summary=None, record=None, max_tool_calls=None): +def tool_answer( + cfg, toolbox, question, history=(), max_steps=6, model=None, setup=False, summary=None, record=None, max_tool_calls=None, shrink=None +): """The tool loop: yields ("step", {...}) for each tool call, then ("answer", text), or ("direct", text) when the model answered without looking anything up (it never saw the archive, so the caller can answer from a search instead). Every turn ends in an answer: when the steps run out, or the model replies with nothing, it's asked to answer from what it found, with the tools off. Raises llm.ToolsUnsupported. `record` (a Record) learns the tools offered and counts the rounds and tool calls. `max_tool_calls` (tool_calls_allowed): at most that many calls, in as many rounds as it takes (a round makes one call or more), instead of `max_steps` rounds; a round that asks for more makes the - first of them, and the answer is written from what they found (its record's outcome is limit).""" + first of them, and the answer is written from what they found (its record's outcome is limit). Each request is + fitted into the model's context (fit: what tools returned is shortened); when the server still says it doesn't + fit, what tools returned is cut down further, then the conversation is shrunk (`shrink()`: (history, summary), + once).""" limit = tool_calls_allowed(cfg, max_tool_calls) max_steps = limit or max_steps system = TOOL_SYSTEM + untrusted.rules(cfg) + ("\n\n" + SETUP_SYSTEM if setup else "") note = getattr(toolbox, "system_note", None) # skills, and context from hooks (extensions.py) system += note() if note else "" - system += summary_note(summary) if limit: - system += ( + budget_note = ( f"\n\nYou have up to {limit} tool call{'s' if limit > 1 else ''} for this question in all (make calls that don't depend " "on each other in the same round), then you answer. Answer as soon as you have enough." ) else: - system += ( + budget_note = ( f"\n\nYou have up to {max_steps} rounds of tool calls for this question (make calls that don't depend on each other " "in the same round), then you answer. Answer as soon as you have enough." ) - msgs = [{"role": "system", "content": system}] + past_turns(history, summary=summary, cfg=cfg) - msgs.append({"role": "user", "content": question}) + + def opening(history, summary): + msgs = [{"role": "system", "content": system + summary_note(summary) + budget_note}] + past_turns(history, summary=summary, cfg=cfg) + return msgs + [{"role": "user", "content": question}] + + msgs = opening(history, summary) + start = len(msgs) # where this answer's own rounds begin tools, looked, made = toolbox.specs(), False, set() record = record or Record(cfg) record.offer(toolbox, tools, max_steps) + budget, shrunk = room(cfg, model, tools), shrink is None + + def ask(call): + nonlocal start, budget, shrunk + while True: + fit(msgs, budget) + try: + return call() + except llm.ContextFull: + smaller = min(room(cfg, model, tools), tokens(msgs)) * 3 // 4 # the window it said (llm keeps it), or less than was sent + if smaller < budget and fit(msgs, smaller): # what tools returned can still be cut + budget = smaller + elif not shrunk: + shrunk = True + head = opening(*shrink()) + msgs[:start], start = head, len(head) + else: + raise + for _ in range(max_steps): - msg = llm.chat_message(cfg, msgs, tools=tools, model=model) + msg = ask(lambda: llm.chat_message(cfg, msgs, tools=tools, model=model)) if not msg["tool_calls"]: if msg["content"].strip(): yield ("answer" if looked else "direct"), msg["content"] @@ -1125,7 +1240,10 @@ def tool_answer(cfg, toolbox, question, history=(), max_steps=6, model=None, set break else: # the rounds ran out record.limited = True - text = _finish(cfg, msgs, tools, model) or "I couldn't put an answer together from what I found. Try again, or ask something narrower." + text = ( + ask(lambda: _finish(cfg, msgs, tools, model)) + or "I couldn't put an answer together from what I found. Try again, or ask something narrower." + ) yield ("answer" if looked else "direct"), text diff --git a/fastapi_backend/app/domain/llm.py b/fastapi_backend/app/domain/llm.py index 7bfe387a2..a7bda8ffc 100644 --- a/fastapi_backend/app/domain/llm.py +++ b/fastapi_backend/app/domain/llm.py @@ -18,6 +18,22 @@ class LLMError(RuntimeError): pass +class ContextFull(LLMError): + """The server refused a request too long for the model's context. `window`: the context it said the model has, when + it said (kept as the model's, so the next conversation is summarised before it fills that).""" + + def __init__(self, message, window=None): + super().__init__(message) + self.window = window + + +# how servers say a request doesn't fit: OpenAI's, vLLM's and OpenRouter's "maximum context length", llama.cpp's +# "exceeds the available context size", Groq's context_length_exceeded, Mistral's "too large for model with N maximum +# context length", Anthropic's "prompt is too long", Gemini's "exceeds the maximum number of tokens" +FULL_SAYS = re.compile(r"context[ _-]?(?:length|size|window)|maximum context|prompt is too long|exceeds the maximum number of tokens", re.I) +WINDOW_SAYS = re.compile(r"maximum context length is (\d{3,7})|with (\d{3,7}) maximum context length|\"n_ctx\"\s*:\s*(\d{3,7})", re.I) + + def configured(cfg): return bool(cfg["llm"].get("base_url") and cfg["llm"].get("model")) @@ -49,7 +65,14 @@ def _post(cfg, payload): try: return urllib.request.urlopen(req, timeout=l.get("timeout") or 300) except urllib.error.HTTPError as e: - raise LLMError(f"{e.code} from the LLM server: {e.read().decode('utf-8', 'replace')[:300]}") from None + said = e.read().decode("utf-8", "replace") + if e.code in (400, 413) and FULL_SAYS.search(said): + m = WINDOW_SAYS.search(said) + window = _tokens(next(g for g in m.groups() if g)) if m else None + if window: # what the server said, rather than what it lists or nothing + _WINDOWS[(l["base_url"].rstrip("/"), payload.get("model") or l["model"])] = (time.monotonic(), window) + raise ContextFull(f"{e.code} from the LLM server: {said[:300]}", window) from None + raise LLMError(f"{e.code} from the LLM server: {said[:300]}") from None except (urllib.error.URLError, TimeoutError, OSError) as e: raise LLMError(f"can't reach the LLM server at {l['base_url']}: {e}") from None @@ -327,6 +350,8 @@ def chat_message(cfg, messages, tools=None, model=None, temperature=0.2, tool_ch try: with telemetry.model_call(cfg, payload) as call, _post(cfg, payload) as r: j = call.reply(_load(r)) + except ContextFull: # a 400 too, but the tools aren't why + raise except LLMError as e: if tools and str(e)[:3] in ("400", "404", "422", "501"): raise ToolsUnsupported(str(e)) from None diff --git a/fastapi_backend/app/schemas/chats.py b/fastapi_backend/app/schemas/chats.py index 7f9caf38a..d487ea657 100644 --- a/fastapi_backend/app/schemas/chats.py +++ b/fastapi_backend/app/schemas/chats.py @@ -307,11 +307,24 @@ class StopResult(ResponseModel): stopping: bool = Field(description="an answer was being written, and stops after its current piece or step") +class ChatMemory(ResponseModel): + """What the model reads instead of the messages a conversation's summary covers (long conversations are summarised; + /compact and /clear).""" + + text: str = Field(description="the summary; empty after /clear, when the messages it covers aren't read at all") + upto: int = Field(description="the last message it covers, of the version showing") + at: str | None = Field(None, description="when it was written") + covers: int = Field(description="how many messages of the version showing it stands for") + + class Chat(ChatSummary): account: int head: int | None = Field(None, description="the last message of the version showing, which the next question follows") messages: list[ChatMessage] = Field(default_factory=list, description="the version showing, from the first message to `head`") tree: list[ChatNode] = Field(default_factory=list, description="every message of every version, oldest first") + summary: ChatMemory | None = Field( + None, description="what the model reads instead of the messages up to its `upto`; none before the first summary" + ) class HeadUpdate(RequestModel): diff --git a/fastapi_backend/tests/api/_assist.py b/fastapi_backend/tests/api/_assist.py index 7a87d81c1..adf4a78b4 100644 --- a/fastapi_backend/tests/api/_assist.py +++ b/fastapi_backend/tests/api/_assist.py @@ -26,7 +26,7 @@ def sse(text): def start_llm(cfg): fake_llm.Handler.tool_script, fake_llm.Handler.reject_tools, fake_llm.Handler.decision = [], False, None fake_llm.Handler.ignore_tool_choice, fake_llm.Handler.pick = False, None - fake_llm.Handler.about, fake_llm.Handler.props = {}, None + fake_llm.Handler.about, fake_llm.Handler.props, fake_llm.Handler.context = {}, None, None fake_llm.Handler.title = None fake_llm.Handler.names, fake_llm.Handler.cypher_script = None, [] fake_llm.Handler.research_plan = fake_llm.Handler.research_report = None diff --git a/fastapi_backend/tests/api/test_compaction.py b/fastapi_backend/tests/api/test_compaction.py index 32a4bccaf..385ed1b0e 100644 --- a/fastapi_backend/tests/api/test_compaction.py +++ b/fastapi_backend/tests/api/test_compaction.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json from urllib.parse import urlsplit import pytest @@ -184,6 +185,72 @@ def test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model(db, cf assert len(rows) == 3 and all(f"chat:{cid}" in r["resources"] for r in rows) +def test_the_server_saying_the_context_is_full_is_understood(cfg, llm): + """A 400 that says the request is too long is llm.ContextFull, and the window it names is kept as the model's.""" + forget_windows() + llm.context = 500 + with pytest.raises(llm_mod.ContextFull) as e: + llm_mod.chat(cfg, [{"role": "user", "content": "word " * 1000}]) + assert e.value.window == 500 and llm_mod.context_window(cfg) == 500 and chat.window(cfg) == 500 + assert llm_mod.chat(cfg, [{"role": "user", "content": "hi"}]) == "OK" + + +class BigBox: + """A toolbox with one tool, whose result is far too long for a small model.""" + + flagged: list[str] = [] + approvals: list[dict] = [] + + def specs(self): + return [{"type": "function", "function": {"name": "read_big", "parameters": {"type": "object", "properties": {}}}}] + + def call(self, name, args, model=False): + return json.dumps({"text": "line " * 6000}), "Read something big" + + def take_detail(self): + return {} + + +def test_what_tools_returned_is_cut_to_fit_the_model(cfg, llm): + forget_windows() + llm.context = 2000 + llm.tool_script = [ + {"content": "", "tool_calls": [{"id": "1", "type": "function", "function": {"name": "read_big", "arguments": "{}"}}]}, + {"content": "Done [1]."}, + ] + out = list(chat.tool_answer(cfg, BigBox(), "what's in it?", (), 6)) + assert out[-1] == ("answer", "Done [1].") and out[0][0] == "step" + sent = [b for b in llm.seen if b.get("tools")] + results = [m["content"] for b in sent for m in b["messages"] if m["role"] == "tool"] + assert len(results[0]) > 20000 and "left out, to fit the model's context" not in results[0] # as sent first: refused + assert "left out, to fit the model's context" in results[-1] and len(results[-1]) < 2000 * 4 # cut, then answered + assert results[-1].startswith('{"text": "line') and results[-1].endswith('"}') # its ends kept, the middle gone + assert llm_mod.context_window(cfg) == 2000 # what the server said, kept for the next answer + + +def test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again(client, db, cfg, folder, llm): + """The model says the conversation is too long for it (nothing said it had a small context): all but the last two + messages are folded into the summary now, and the question is asked again with that.""" + cfg["ai"]["tools"] = False + seed(db, cfg, folder) + make_user(db, "root@x.io", "root password 1", admin=True) + h = login(client, "root@x.io", "root password 1") + cid = client.post("/api/v1/chats", headers=h, json={}).json()["id"] + long_talk(db, cid, 10) + forget_windows() + llm.context = 900 # ten long messages don't fit; two and a summary do + out = sse(client.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": "shipment Friday?"}).text) + assert "error" not in out and "".join(t["text"] for t in out["token"]) == "The shipment leaves on Friday [1]." + folded = summaries(llm.seen) + assert folded and "long message 7 " in folded[-1]["messages"][1]["content"] and "long message 8 " not in str(folded) + answered = [b for b in llm.seen if b.get("stream")] + assert len(answered) == 2 and "long message 0 " in str(answered[0]["messages"]) # refused, then asked again + assert "Earlier in this conversation" in answered[1]["messages"][0]["content"] + assert [m["content"][:14] for m in answered[1]["messages"][1:]] == ["long message 8", "long message 9", "Excerpts:\n\n Handler.context: + said = f"This model's maximum context length is {Handler.context} tokens. However, you requested {asked} tokens." + data = json.dumps({"error": {"message": said, "type": "invalid_request_error", "code": "context_length_exceeded"}}).encode() + self.send_response(400) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + return if body.get("tools"): if Handler.reject_tools: data = b'{"error": "tools are not supported by this model"}' diff --git a/nextjs-frontend/__tests__/chat-summary-mark.test.tsx b/nextjs-frontend/__tests__/chat-summary-mark.test.tsx new file mode 100644 index 000000000..e13f55625 --- /dev/null +++ b/nextjs-frontend/__tests__/chat-summary-mark.test.tsx @@ -0,0 +1,26 @@ +import { fireEvent, render, screen } from "@testing-library/react"; +import "@testing-library/jest-dom"; + +import { SummaryMark } from "@/components/chat/summary-mark"; + +describe("the mark under the last message a summary covers", () => { + it("says how many messages the summary stands for, and shows it on request", () => { + render( + , + ); + expect(screen.getByRole("note", { name: "Summarised up to here" })).toHaveTextContent( + "the assistant reads a summary of the 14 messages above instead of them", + ); + expect(screen.queryByText("Alice wants the capsid samples by Friday.")).not.toBeInTheDocument(); + fireEvent.click(screen.getByRole("button", { name: "Show summary" })); + expect(screen.getByText("Alice wants the capsid samples by Friday.")).toBeInTheDocument(); + fireEvent.click(screen.getByRole("button", { name: "Hide summary" })); + expect(screen.queryByText("Alice wants the capsid samples by Friday.")).not.toBeInTheDocument(); + }); + + it("says the messages above are no longer read after /clear", () => { + render(); + expect(screen.getByRole("note", { name: "Cleared here" })).toHaveTextContent("no longer reads the 1 message above"); + expect(screen.queryByRole("button")).not.toBeInTheDocument(); + }); +}); diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index eb6bd463e..95d9b740b 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -319,6 +319,7 @@ export type { ChatCompletion, ChatCompletionRequest, ChatCreate, + ChatMemory, ChatMessage, ChatNode, ChatScope, diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 91c73b405..23c99b6de 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -3437,6 +3437,10 @@ export type Chat = { * every message of every version, oldest first */ tree?: Array; + /** + * what the model reads instead of the messages up to its `upto`; none before the first summary + */ + summary?: ChatMemory | null; [key: string]: unknown; }; @@ -3608,6 +3612,40 @@ export type ChatCreate = { max_tool_calls?: number | null; }; +/** + * ChatMemory + * + * What the model reads instead of the messages a conversation's summary covers (long conversations are summarised; + * /compact and /clear). + */ +export type ChatMemory = { + /** + * Text + * + * the summary; empty after /clear, when the messages it covers aren't read at all + */ + text: string; + /** + * Upto + * + * the last message it covers, of the version showing + */ + upto: number; + /** + * At + * + * when it was written + */ + at?: string | null; + /** + * Covers + * + * how many messages of the version showing it stands for + */ + covers: number; + [key: string]: unknown; +}; + /** * ChatMessage */ diff --git a/nextjs-frontend/components/chat/chat-app.tsx b/nextjs-frontend/components/chat/chat-app.tsx index d30806d67..1a9d7be51 100644 --- a/nextjs-frontend/components/chat/chat-app.tsx +++ b/nextjs-frontend/components/chat/chat-app.tsx @@ -31,6 +31,7 @@ import { type TurnState, } from "@/components/chat/stream"; import { onlyFiles, UserBubble } from "@/components/chat/user-bubble"; +import { SummaryMark } from "@/components/chat/summary-mark"; import { switched, VersionSwitch } from "@/components/chat/versions"; import { SharedFrom } from "@/components/page-chat/page-chat"; import { useRecordingIndex, useSpeakerDirectory } from "@/components/search/data"; @@ -558,6 +559,8 @@ export function ChatApp() { }, [chat.data?.id, chat.data]); const msgs = chat.data?.messages ?? []; + // where the model stops reading the conversation word for word: under the last message its summary covers + const memory = chat.data?.summary ?? null; const items = useMemo(() => { const here = live && live.chatId === activeId; // a version being written streams in place of the one beside it (what came before it has smaller ids) @@ -799,6 +802,7 @@ export function ChatApp() { /> )} + {memory && it.q && it.a && memory.upto === it.q.id && } {it.a?.kind === "research_plan" && it.a.meta?.research != null ? ( ) : it.a ? ( @@ -875,6 +879,9 @@ export function ChatApp() {

) )} + {memory && (it.a ? memory.upto === it.a.id : memory.upto === it.q?.id) && ( + + )} ); })} diff --git a/nextjs-frontend/components/chat/summary-mark.tsx b/nextjs-frontend/components/chat/summary-mark.tsx new file mode 100644 index 000000000..9171a43f9 --- /dev/null +++ b/nextjs-frontend/components/chat/summary-mark.tsx @@ -0,0 +1,51 @@ +"use client"; + +import { useState } from "react"; + +import type { ChatMemory } from "@/app/openapi-client/types.gen"; + +/** + * Under the last message a conversation's summary covers: the model reads the summary instead of the messages above + * it (or, after /clear, nothing of them). Show summary opens the text, so what the model remembers is never hidden. + */ +export function SummaryMark({ summary }: { summary: ChatMemory }) { + const [open, setOpen] = useState(false); + const n = summary.covers; + const some = `${n} message${n === 1 ? "" : "s"}`; + const cleared = !summary.text.trim(); + return ( +
+
+ + + {cleared + ? `Cleared here: the assistant no longer reads the ${some} above.` + : `Summarised up to here: the assistant reads a summary of the ${some} above instead of them.`} + {!cleared && ( + <> + {" "} + + + )} + + +
+ {open && !cleared && ( +

+ {summary.text} +

+ )} +
+ ); +} diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 21e0aae43..679a29ecd 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -38900,6 +38900,17 @@ "type": "array", "title": "Tree", "description": "every message of every version, oldest first" + }, + "summary": { + "anyOf": [ + { + "$ref": "#/components/schemas/ChatMemory" + }, + { + "type": "null" + } + ], + "description": "what the model reads instead of the messages up to its `upto`; none before the first summary" } }, "additionalProperties": true, @@ -39177,6 +39188,46 @@ "type": "object", "title": "ChatCreate" }, + "ChatMemory": { + "properties": { + "text": { + "type": "string", + "title": "Text", + "description": "the summary; empty after /clear, when the messages it covers aren't read at all" + }, + "upto": { + "type": "integer", + "title": "Upto", + "description": "the last message it covers, of the version showing" + }, + "at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "At", + "description": "when it was written" + }, + "covers": { + "type": "integer", + "title": "Covers", + "description": "how many messages of the version showing it stands for" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "text", + "upto", + "covers" + ], + "title": "ChatMemory", + "description": "What the model reads instead of the messages a conversation's summary covers (long conversations are summarised;\n/compact and /clear)." + }, "ChatMessage": { "properties": { "id": {