diff --git a/docs/assistant.md b/docs/assistant.md
index f5cad029a..45ebfc39d 100644
--- a/docs/assistant.md
+++ b/docs/assistant.md
@@ -49,7 +49,13 @@ then.
the chat's model does. A long stretch folded at once (`/compact`, or a conversation from before summaries) is
written in as many calls as it takes to fit that model's context, each in the [activity ledger](activity.md) as the
conversation's. Settings › AI assistant › Summarise long conversations (`ai.compact`, on) turns it off; off, or with
- no model, only the last six messages are read.
+ no model, only the last six messages are read. When the model server still says a request is too long for the
+ model (its context was smaller than Lens knew), the answer is tried again rather than failed: what tools returned
+ in that answer is shortened first (each result keeps its start and end), then all but the last two messages are
+ folded into the summary now and half of the excerpts are left out; the context the server named is kept as the
+ model's, so the next conversation is summarised before it fills. In the web app, a line under the last message a
+ summary covers says how many messages it stands for (or that `/clear` stopped the model reading them), and Show
+ summary opens the summary; `GET /api/v1/chats/` gives it as `summary` (`text`, `upto`, `covers`).
- **Titles:**{#titles} a conversation nobody named is titled with its first question's words as soon as it's asked,
then, after the first answer, the model gives it a short title of a few words from the question and the answer
(streamed as `title` at the end of the answer's stream). It's one small call, written by **Model for summaries**
diff --git a/docs/requirements.md b/docs/requirements.md
index a7f3a2b3c..a6f273dd1 100644
--- a/docs/requirements.md
+++ b/docs/requirements.md
@@ -86,7 +86,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` |
| Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` |
| Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` |
-| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app` |
+| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again`, `test_what_tools_returned_is_cut_to_fit_the_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app`; [chat-summary-mark.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-summary-mark.test.tsx) `says how many messages the summary stands for, and shows it on request` |
| A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` |
| Telegram and Slack built in, with voice notes and files | [test_telegram.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_telegram.py) `test_it_answers_private_chats_and_what_is_said_to_it_in_groups`, `test_voice_notes_are_heard`, `test_files_are_held_for_the_assistant_to_import`; [test_slack.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_slack.py) `test_it_answers_direct_messages_mentions_and_the_slash_command`, `test_voice_clips_are_heard_and_files_held` |
| Other chat rooms through Matterbridge | [test_bridge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_bridge.py) `test_it_answers_what_is_said_to_it_and_nothing_else`; [test_matterbridge_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_matterbridge_run.py) `test_chat_networks_are_set_in_the_app_and_become_matterbridges_config`, `test_the_bridge_talks_to_the_matterbridge_lens_runs`; [matterbridge.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/matterbridge.yml) `Lens writes a config, and Matterbridge starts with it` |
diff --git a/fastapi_backend/app/api/v1/routes/chats.py b/fastapi_backend/app/api/v1/routes/chats.py
index 8c64eaaa0..20806980c 100644
--- a/fastapi_backend/app/api/v1/routes/chats.py
+++ b/fastapi_backend/app/api/v1/routes/chats.py
@@ -219,7 +219,8 @@ def _chat_view(db: DB, acl: Access, c: dict[str, Any]) -> Chat:
for m in msgs:
m["versions"] = versions.get(m["id"])
_readable_citations(db, acl, msgs)
- return Chat.model_validate(sign_urls({**c, "messages": msgs, "tree": nodes}, full=True))
+ view = {**c, "messages": msgs, "tree": nodes, "summary": chat.covered(db, c["id"], msgs)}
+ return Chat.model_validate(sign_urls(view, full=True))
def _readable_citations(db: DB, acl: Access, msgs: list[dict[str, Any]]) -> None:
@@ -681,6 +682,10 @@ def stopped(text: str, cited: list[dict[str, Any]]) -> Iterator[str]:
yield _ev("stopped", {"message": "Stopped"})
yield from save(text or "(stopped)", cited, "stopped", stopped=True)
+ def shrink() -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
+ """The model said the conversation doesn't fit its context: fold it into the summary now (chat.shrink)."""
+ return chat.shrink(db, cfg, cid, past)
+
def answer(on: chat.Answering) -> Iterator[str]:
nonlocal notice
if llm.configured(cfg) and cfg["ai"].get("tools") and tools != []: # given no tools, it answers from a search
@@ -708,7 +713,17 @@ def answer(on: chat.Answering) -> Iterator[str]:
# setting the server up, or working in the person's browser, takes more steps than looking things up
steps_max = max(cfg["ai"].get("max_steps") or 6, 12 if setup else 0, browser.steps(cfg) if box.browser else 0)
answering = chat.tool_answer(
- cfg, box, asked, past, steps_max, model, setup=setup, summary=summary, record=record, max_tool_calls=limit
+ cfg,
+ box,
+ asked,
+ past,
+ steps_max,
+ model,
+ setup=setup,
+ summary=summary,
+ record=record,
+ max_tool_calls=limit,
+ shrink=shrink,
)
for kind, data in answering:
if kind == "step":
@@ -745,7 +760,7 @@ def answer(on: chat.Answering) -> Iterator[str]:
text, error = "", None
try:
if llm.configured(cfg):
- pieces = llm.stream_chat(cfg, chat.messages_for(asked, passages, past, summary, cfg), model=model)
+ pieces = chat.stream_fitted(cfg, asked, passages, past, summary, model=model, shrink=shrink)
for piece in pieces:
text += piece
yield _ev("token", {"text": piece})
diff --git a/fastapi_backend/app/domain/bridge.py b/fastapi_backend/app/domain/bridge.py
index df6978c2c..093fb0f2f 100644
--- a/fastapi_backend/app/domain/bridge.py
+++ b/fastapi_backend/app/domain/bridge.py
@@ -375,6 +375,10 @@ def _answer(db, cfg, base, account, cid, q, files, act_on, outside):
steps, approvals, notice = [], [], None
wrote = cfg["llm"].get("model") if llm.configured(cfg) else None
+ def shrink():
+ """The model said the conversation doesn't fit its context: fold it into the summary now (chat.shrink)."""
+ return chat.shrink(db, cfg, cid, past)
+
def said(words, cited, outcome, **kw):
"""The answer, saved with its record."""
meta = record.meta(db, outcome, spent)
@@ -397,7 +401,8 @@ def said(words, cited, outcome, **kw):
)
try:
text, how = None, "answered"
- for kind, data in chat.tool_answer(cfg, box, asked, past, cfg["ai"].get("max_steps") or 6, summary=summary, record=record):
+ steps_max = cfg["ai"].get("max_steps") or 6
+ for kind, data in chat.tool_answer(cfg, box, asked, past, steps_max, summary=summary, record=record, shrink=shrink):
if kind == "step":
steps.append(data)
elif kind == "direct" and passages and not box.cited(data): # unless it cites what it remembers
@@ -415,7 +420,7 @@ def said(words, cited, outcome, **kw):
return "I couldn't answer that: " + str(e), box.approvals, keep | box.links
error = None
try:
- text = llm.chat(cfg, chat.messages_for(asked, passages, past, summary, cfg)) if llm.configured(cfg) else chat.fallback(passages)
+ text = chat.reply_fitted(cfg, asked, passages, past, summary, shrink=shrink) if llm.configured(cfg) else chat.fallback(passages)
except llm.LLMError as e:
text, error = "I couldn't answer that: " + str(e), str(e)
how = "error" if error else "fallback" if notice or not llm.configured(cfg) else "answered"
diff --git a/fastapi_backend/app/domain/chat.py b/fastapi_backend/app/domain/chat.py
index 273edc23b..b0fab6035 100644
--- a/fastapi_backend/app/domain/chat.py
+++ b/fastapi_backend/app/domain/chat.py
@@ -8,6 +8,7 @@
from __future__ import annotations
+import contextlib
import contextvars
import json
import logging
@@ -598,6 +599,15 @@ def memory(db, cfg, cid, line=None):
return {**had, "window": RECENT} if had.get("upto") else None
+def covered(db, cid, line):
+ """The summary as the app shows it: the one stored for the version `line` is, with "covers", how many of its
+ messages it stands for; None before the first summary (or /clear)."""
+ had = _stored(db, cid, line)
+ if not had.get("upto"):
+ return None
+ return {**had, "covers": sum(1 for m in line if m["id"] <= had["upto"])}
+
+
def summary_note(summary):
text = ((summary or {}).get("text") or "").strip()
return (
@@ -652,11 +662,12 @@ def _batches(cfg, model, had, old):
yield run
-def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None):
+def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None, keep=None):
"""Fold the messages older than the latest RECENT into the conversation's summary, once COMPACT_AFTER of them have
built up or they fill ai.compact_at % of the chat model's context window (window; `full_only`: only then;
ai.compact, on by default; needs a model). `force` (/compact) folds every message in now, whatever the setting,
- with the person's `focus` (what to keep) if any. The summary is written by ai.compact_model when set, a cheaper
+ with the person's `focus` (what to keep) if any, but the latest `keep` (shrink). The summary is written by
+ ai.compact_model when set, a cheaper
model than the chat's, in as many calls as it takes to fit that model's window (_batches), each in the ledger as
the conversation's. It's of the version `line` is (by default the head's), afresh when the summary was another
version's. Returns the new summary (with "folded": how many messages), or None when nothing changed."""
@@ -665,7 +676,8 @@ def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None):
line = path(db, cid) if line is None else line
had = _stored(db, cid, line)
fresh = [m for m in line if m["id"] > (had.get("upto") or 0)] # ids grow along a version
- old = fresh if force else fresh[:-RECENT]
+ keep = (0 if force else RECENT) if keep is None else keep
+ old = fresh[:-keep] if keep else fresh
if not old:
return None
if not force and (full_only or len(fresh) < RECENT + COMPACT_AFTER):
@@ -696,6 +708,81 @@ def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None):
return {**out, "folded": folded}
+# ---------- fitting an answer's request into the model's context ----------
+FIT_KEEP = 2 # the latest messages still read word for word once a conversation didn't fit the model
+MIN_RESULT = 400 # characters of a tool's result kept, however little room is left
+
+
+def shrink(db, cfg, cid, line):
+ """What to answer with once the model said the conversation doesn't fit its context: (the messages, the summary),
+ with all but the last FIT_KEEP messages of the version `line` folded into the summary now, whatever the count. With
+ summaries off (ai.compact), or when one can't be written, the last FIT_KEEP messages alone, for this answer."""
+ if cfg["ai"].get("compact", True):
+ try:
+ compact(db, cfg, cid, force=True, keep=FIT_KEEP, line=line)
+ except (llm.LLMError, activity.OverBudget):
+ log.warning("chat %s: couldn't fold the conversation into its summary to fit the model", cid, exc_info=True)
+ return list(line)[-FIT_KEEP:], memory(db, cfg, cid, line)
+
+
+def room(cfg, model=None, tools=None):
+ """About how many tokens an answer's messages may take: the model's context window less room for the answer and
+ for the tools offered."""
+ w = window(cfg, model)
+ return w - min(w // 4, 8192) - len(json.dumps(tools or [])) // 4
+
+
+def fit(msgs, budget):
+ """Shorten what tools returned, oldest first, until the messages take at most `budget` tokens: each loses its middle,
+ so the mark wrapped round outside text (untrusted.wrap) stays whole, and keeps at least MIN_RESULT characters.
+ Returns whether it cut anything."""
+ over, cut = tokens(msgs) - budget, False
+ for m in msgs:
+ if over <= 0:
+ break
+ text = m.get("content") or ""
+ if m["role"] != "tool" or len(text) <= MIN_RESULT:
+ continue
+ keep = max(MIN_RESULT, len(text) - over * 4)
+ if keep >= len(text):
+ continue
+ m["content"] = (
+ f"{text[: keep - keep // 4]}\n…({len(text) - keep} characters left out, to fit the model's context)…\n{text[-(keep // 4) :]}"
+ )
+ over -= (len(text) - len(m["content"])) // 4
+ cut = True
+ return cut
+
+
+def stream_fitted(cfg, question, passages, history=(), summary=None, model=None, shrink=None):
+ """llm.stream_chat of messages_for. When the server says they don't fit the model's context before any of the answer
+ came, it's asked once more with the conversation shrunk (`shrink()`: (history, summary), see shrink) and the first
+ half of the excerpts."""
+ try:
+ with contextlib.closing(llm.stream_chat(cfg, messages_for(question, passages, history, summary, cfg), model=model)) as pieces:
+ for piece in pieces:
+ yield piece
+ return
+ except llm.ContextFull:
+ if shrink is None:
+ raise
+ history, summary = shrink()
+ shorter = passages[: max(1, len(passages) // 2)]
+ with contextlib.closing(llm.stream_chat(cfg, messages_for(question, shorter, history, summary, cfg), model=model)) as pieces:
+ yield from pieces
+
+
+def reply_fitted(cfg, question, passages, history=(), summary=None, model=None, shrink=None):
+ """llm.chat of messages_for, asked once more as stream_fitted is when the server says they don't fit."""
+ try:
+ return llm.chat(cfg, messages_for(question, passages, history, summary, cfg), model=model)
+ except llm.ContextFull:
+ if shrink is None:
+ raise
+ history, summary = shrink()
+ return llm.chat(cfg, messages_for(question, passages[: max(1, len(passages) // 2)], history, summary, cfg), model=model)
+
+
# ---------- commands: /clear and /compact, typed in the app or in a chat room ----------
COMMAND = re.compile(r"^/(clear|compact)(?:\s+(.*))?$", re.S | re.I)
@@ -1052,37 +1139,65 @@ def _finish(cfg, msgs, tools, model):
return text
-def tool_answer(cfg, toolbox, question, history=(), max_steps=6, model=None, setup=False, summary=None, record=None, max_tool_calls=None):
+def tool_answer(
+ cfg, toolbox, question, history=(), max_steps=6, model=None, setup=False, summary=None, record=None, max_tool_calls=None, shrink=None
+):
"""The tool loop: yields ("step", {...}) for each tool call, then ("answer", text), or ("direct", text) when the model
answered without looking anything up (it never saw the archive, so the caller can answer from a search instead).
Every turn ends in an answer: when the steps run out, or the model replies with nothing, it's asked to answer from
what it found, with the tools off. Raises llm.ToolsUnsupported. `record` (a Record) learns the tools offered and
counts the rounds and tool calls. `max_tool_calls` (tool_calls_allowed): at most that many calls, in as many rounds
as it takes (a round makes one call or more), instead of `max_steps` rounds; a round that asks for more makes the
- first of them, and the answer is written from what they found (its record's outcome is limit)."""
+ first of them, and the answer is written from what they found (its record's outcome is limit). Each request is
+ fitted into the model's context (fit: what tools returned is shortened); when the server still says it doesn't
+ fit, what tools returned is cut down further, then the conversation is shrunk (`shrink()`: (history, summary),
+ once)."""
limit = tool_calls_allowed(cfg, max_tool_calls)
max_steps = limit or max_steps
system = TOOL_SYSTEM + untrusted.rules(cfg) + ("\n\n" + SETUP_SYSTEM if setup else "")
note = getattr(toolbox, "system_note", None) # skills, and context from hooks (extensions.py)
system += note() if note else ""
- system += summary_note(summary)
if limit:
- system += (
+ budget_note = (
f"\n\nYou have up to {limit} tool call{'s' if limit > 1 else ''} for this question in all (make calls that don't depend "
"on each other in the same round), then you answer. Answer as soon as you have enough."
)
else:
- system += (
+ budget_note = (
f"\n\nYou have up to {max_steps} rounds of tool calls for this question (make calls that don't depend on each other "
"in the same round), then you answer. Answer as soon as you have enough."
)
- msgs = [{"role": "system", "content": system}] + past_turns(history, summary=summary, cfg=cfg)
- msgs.append({"role": "user", "content": question})
+
+ def opening(history, summary):
+ msgs = [{"role": "system", "content": system + summary_note(summary) + budget_note}] + past_turns(history, summary=summary, cfg=cfg)
+ return msgs + [{"role": "user", "content": question}]
+
+ msgs = opening(history, summary)
+ start = len(msgs) # where this answer's own rounds begin
tools, looked, made = toolbox.specs(), False, set()
record = record or Record(cfg)
record.offer(toolbox, tools, max_steps)
+ budget, shrunk = room(cfg, model, tools), shrink is None
+
+ def ask(call):
+ nonlocal start, budget, shrunk
+ while True:
+ fit(msgs, budget)
+ try:
+ return call()
+ except llm.ContextFull:
+ smaller = min(room(cfg, model, tools), tokens(msgs)) * 3 // 4 # the window it said (llm keeps it), or less than was sent
+ if smaller < budget and fit(msgs, smaller): # what tools returned can still be cut
+ budget = smaller
+ elif not shrunk:
+ shrunk = True
+ head = opening(*shrink())
+ msgs[:start], start = head, len(head)
+ else:
+ raise
+
for _ in range(max_steps):
- msg = llm.chat_message(cfg, msgs, tools=tools, model=model)
+ msg = ask(lambda: llm.chat_message(cfg, msgs, tools=tools, model=model))
if not msg["tool_calls"]:
if msg["content"].strip():
yield ("answer" if looked else "direct"), msg["content"]
@@ -1125,7 +1240,10 @@ def tool_answer(cfg, toolbox, question, history=(), max_steps=6, model=None, set
break
else: # the rounds ran out
record.limited = True
- text = _finish(cfg, msgs, tools, model) or "I couldn't put an answer together from what I found. Try again, or ask something narrower."
+ text = (
+ ask(lambda: _finish(cfg, msgs, tools, model))
+ or "I couldn't put an answer together from what I found. Try again, or ask something narrower."
+ )
yield ("answer" if looked else "direct"), text
diff --git a/fastapi_backend/app/domain/llm.py b/fastapi_backend/app/domain/llm.py
index 7bfe387a2..a7bda8ffc 100644
--- a/fastapi_backend/app/domain/llm.py
+++ b/fastapi_backend/app/domain/llm.py
@@ -18,6 +18,22 @@ class LLMError(RuntimeError):
pass
+class ContextFull(LLMError):
+ """The server refused a request too long for the model's context. `window`: the context it said the model has, when
+ it said (kept as the model's, so the next conversation is summarised before it fills that)."""
+
+ def __init__(self, message, window=None):
+ super().__init__(message)
+ self.window = window
+
+
+# how servers say a request doesn't fit: OpenAI's, vLLM's and OpenRouter's "maximum context length", llama.cpp's
+# "exceeds the available context size", Groq's context_length_exceeded, Mistral's "too large for model with N maximum
+# context length", Anthropic's "prompt is too long", Gemini's "exceeds the maximum number of tokens"
+FULL_SAYS = re.compile(r"context[ _-]?(?:length|size|window)|maximum context|prompt is too long|exceeds the maximum number of tokens", re.I)
+WINDOW_SAYS = re.compile(r"maximum context length is (\d{3,7})|with (\d{3,7}) maximum context length|\"n_ctx\"\s*:\s*(\d{3,7})", re.I)
+
+
def configured(cfg):
return bool(cfg["llm"].get("base_url") and cfg["llm"].get("model"))
@@ -49,7 +65,14 @@ def _post(cfg, payload):
try:
return urllib.request.urlopen(req, timeout=l.get("timeout") or 300)
except urllib.error.HTTPError as e:
- raise LLMError(f"{e.code} from the LLM server: {e.read().decode('utf-8', 'replace')[:300]}") from None
+ said = e.read().decode("utf-8", "replace")
+ if e.code in (400, 413) and FULL_SAYS.search(said):
+ m = WINDOW_SAYS.search(said)
+ window = _tokens(next(g for g in m.groups() if g)) if m else None
+ if window: # what the server said, rather than what it lists or nothing
+ _WINDOWS[(l["base_url"].rstrip("/"), payload.get("model") or l["model"])] = (time.monotonic(), window)
+ raise ContextFull(f"{e.code} from the LLM server: {said[:300]}", window) from None
+ raise LLMError(f"{e.code} from the LLM server: {said[:300]}") from None
except (urllib.error.URLError, TimeoutError, OSError) as e:
raise LLMError(f"can't reach the LLM server at {l['base_url']}: {e}") from None
@@ -327,6 +350,8 @@ def chat_message(cfg, messages, tools=None, model=None, temperature=0.2, tool_ch
try:
with telemetry.model_call(cfg, payload) as call, _post(cfg, payload) as r:
j = call.reply(_load(r))
+ except ContextFull: # a 400 too, but the tools aren't why
+ raise
except LLMError as e:
if tools and str(e)[:3] in ("400", "404", "422", "501"):
raise ToolsUnsupported(str(e)) from None
diff --git a/fastapi_backend/app/schemas/chats.py b/fastapi_backend/app/schemas/chats.py
index 7f9caf38a..d487ea657 100644
--- a/fastapi_backend/app/schemas/chats.py
+++ b/fastapi_backend/app/schemas/chats.py
@@ -307,11 +307,24 @@ class StopResult(ResponseModel):
stopping: bool = Field(description="an answer was being written, and stops after its current piece or step")
+class ChatMemory(ResponseModel):
+ """What the model reads instead of the messages a conversation's summary covers (long conversations are summarised;
+ /compact and /clear)."""
+
+ text: str = Field(description="the summary; empty after /clear, when the messages it covers aren't read at all")
+ upto: int = Field(description="the last message it covers, of the version showing")
+ at: str | None = Field(None, description="when it was written")
+ covers: int = Field(description="how many messages of the version showing it stands for")
+
+
class Chat(ChatSummary):
account: int
head: int | None = Field(None, description="the last message of the version showing, which the next question follows")
messages: list[ChatMessage] = Field(default_factory=list, description="the version showing, from the first message to `head`")
tree: list[ChatNode] = Field(default_factory=list, description="every message of every version, oldest first")
+ summary: ChatMemory | None = Field(
+ None, description="what the model reads instead of the messages up to its `upto`; none before the first summary"
+ )
class HeadUpdate(RequestModel):
diff --git a/fastapi_backend/tests/api/_assist.py b/fastapi_backend/tests/api/_assist.py
index 7a87d81c1..adf4a78b4 100644
--- a/fastapi_backend/tests/api/_assist.py
+++ b/fastapi_backend/tests/api/_assist.py
@@ -26,7 +26,7 @@ def sse(text):
def start_llm(cfg):
fake_llm.Handler.tool_script, fake_llm.Handler.reject_tools, fake_llm.Handler.decision = [], False, None
fake_llm.Handler.ignore_tool_choice, fake_llm.Handler.pick = False, None
- fake_llm.Handler.about, fake_llm.Handler.props = {}, None
+ fake_llm.Handler.about, fake_llm.Handler.props, fake_llm.Handler.context = {}, None, None
fake_llm.Handler.title = None
fake_llm.Handler.names, fake_llm.Handler.cypher_script = None, []
fake_llm.Handler.research_plan = fake_llm.Handler.research_report = None
diff --git a/fastapi_backend/tests/api/test_compaction.py b/fastapi_backend/tests/api/test_compaction.py
index 32a4bccaf..385ed1b0e 100644
--- a/fastapi_backend/tests/api/test_compaction.py
+++ b/fastapi_backend/tests/api/test_compaction.py
@@ -2,6 +2,7 @@
from __future__ import annotations
+import json
from urllib.parse import urlsplit
import pytest
@@ -184,6 +185,72 @@ def test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model(db, cf
assert len(rows) == 3 and all(f"chat:{cid}" in r["resources"] for r in rows)
+def test_the_server_saying_the_context_is_full_is_understood(cfg, llm):
+ """A 400 that says the request is too long is llm.ContextFull, and the window it names is kept as the model's."""
+ forget_windows()
+ llm.context = 500
+ with pytest.raises(llm_mod.ContextFull) as e:
+ llm_mod.chat(cfg, [{"role": "user", "content": "word " * 1000}])
+ assert e.value.window == 500 and llm_mod.context_window(cfg) == 500 and chat.window(cfg) == 500
+ assert llm_mod.chat(cfg, [{"role": "user", "content": "hi"}]) == "OK"
+
+
+class BigBox:
+ """A toolbox with one tool, whose result is far too long for a small model."""
+
+ flagged: list[str] = []
+ approvals: list[dict] = []
+
+ def specs(self):
+ return [{"type": "function", "function": {"name": "read_big", "parameters": {"type": "object", "properties": {}}}}]
+
+ def call(self, name, args, model=False):
+ return json.dumps({"text": "line " * 6000}), "Read something big"
+
+ def take_detail(self):
+ return {}
+
+
+def test_what_tools_returned_is_cut_to_fit_the_model(cfg, llm):
+ forget_windows()
+ llm.context = 2000
+ llm.tool_script = [
+ {"content": "", "tool_calls": [{"id": "1", "type": "function", "function": {"name": "read_big", "arguments": "{}"}}]},
+ {"content": "Done [1]."},
+ ]
+ out = list(chat.tool_answer(cfg, BigBox(), "what's in it?", (), 6))
+ assert out[-1] == ("answer", "Done [1].") and out[0][0] == "step"
+ sent = [b for b in llm.seen if b.get("tools")]
+ results = [m["content"] for b in sent for m in b["messages"] if m["role"] == "tool"]
+ assert len(results[0]) > 20000 and "left out, to fit the model's context" not in results[0] # as sent first: refused
+ assert "left out, to fit the model's context" in results[-1] and len(results[-1]) < 2000 * 4 # cut, then answered
+ assert results[-1].startswith('{"text": "line') and results[-1].endswith('"}') # its ends kept, the middle gone
+ assert llm_mod.context_window(cfg) == 2000 # what the server said, kept for the next answer
+
+
+def test_a_conversation_that_doesnt_fit_is_folded_in_and_asked_again(client, db, cfg, folder, llm):
+ """The model says the conversation is too long for it (nothing said it had a small context): all but the last two
+ messages are folded into the summary now, and the question is asked again with that."""
+ cfg["ai"]["tools"] = False
+ seed(db, cfg, folder)
+ make_user(db, "root@x.io", "root password 1", admin=True)
+ h = login(client, "root@x.io", "root password 1")
+ cid = client.post("/api/v1/chats", headers=h, json={}).json()["id"]
+ long_talk(db, cid, 10)
+ forget_windows()
+ llm.context = 900 # ten long messages don't fit; two and a summary do
+ out = sse(client.post(f"/api/v1/chats/{cid}/messages", headers=h, json={"content": "shipment Friday?"}).text)
+ assert "error" not in out and "".join(t["text"] for t in out["token"]) == "The shipment leaves on Friday [1]."
+ folded = summaries(llm.seen)
+ assert folded and "long message 7 " in folded[-1]["messages"][1]["content"] and "long message 8 " not in str(folded)
+ answered = [b for b in llm.seen if b.get("stream")]
+ assert len(answered) == 2 and "long message 0 " in str(answered[0]["messages"]) # refused, then asked again
+ assert "Earlier in this conversation" in answered[1]["messages"][0]["content"]
+ assert [m["content"][:14] for m in answered[1]["messages"][1:]] == ["long message 8", "long message 9", "Excerpts:\n\n Handler.context:
+ said = f"This model's maximum context length is {Handler.context} tokens. However, you requested {asked} tokens."
+ data = json.dumps({"error": {"message": said, "type": "invalid_request_error", "code": "context_length_exceeded"}}).encode()
+ self.send_response(400)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(data)))
+ self.end_headers()
+ self.wfile.write(data)
+ return
if body.get("tools"):
if Handler.reject_tools:
data = b'{"error": "tools are not supported by this model"}'
diff --git a/nextjs-frontend/__tests__/chat-summary-mark.test.tsx b/nextjs-frontend/__tests__/chat-summary-mark.test.tsx
new file mode 100644
index 000000000..e13f55625
--- /dev/null
+++ b/nextjs-frontend/__tests__/chat-summary-mark.test.tsx
@@ -0,0 +1,26 @@
+import { fireEvent, render, screen } from "@testing-library/react";
+import "@testing-library/jest-dom";
+
+import { SummaryMark } from "@/components/chat/summary-mark";
+
+describe("the mark under the last message a summary covers", () => {
+ it("says how many messages the summary stands for, and shows it on request", () => {
+ render(
+ ,
+ );
+ expect(screen.getByRole("note", { name: "Summarised up to here" })).toHaveTextContent(
+ "the assistant reads a summary of the 14 messages above instead of them",
+ );
+ expect(screen.queryByText("Alice wants the capsid samples by Friday.")).not.toBeInTheDocument();
+ fireEvent.click(screen.getByRole("button", { name: "Show summary" }));
+ expect(screen.getByText("Alice wants the capsid samples by Friday.")).toBeInTheDocument();
+ fireEvent.click(screen.getByRole("button", { name: "Hide summary" }));
+ expect(screen.queryByText("Alice wants the capsid samples by Friday.")).not.toBeInTheDocument();
+ });
+
+ it("says the messages above are no longer read after /clear", () => {
+ render();
+ expect(screen.getByRole("note", { name: "Cleared here" })).toHaveTextContent("no longer reads the 1 message above");
+ expect(screen.queryByRole("button")).not.toBeInTheDocument();
+ });
+});
diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts
index eb6bd463e..95d9b740b 100644
--- a/nextjs-frontend/app/openapi-client/index.ts
+++ b/nextjs-frontend/app/openapi-client/index.ts
@@ -319,6 +319,7 @@ export type {
ChatCompletion,
ChatCompletionRequest,
ChatCreate,
+ ChatMemory,
ChatMessage,
ChatNode,
ChatScope,
diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts
index 91c73b405..23c99b6de 100644
--- a/nextjs-frontend/app/openapi-client/types.gen.ts
+++ b/nextjs-frontend/app/openapi-client/types.gen.ts
@@ -3437,6 +3437,10 @@ export type Chat = {
* every message of every version, oldest first
*/
tree?: Array;
+ /**
+ * what the model reads instead of the messages up to its `upto`; none before the first summary
+ */
+ summary?: ChatMemory | null;
[key: string]: unknown;
};
@@ -3608,6 +3612,40 @@ export type ChatCreate = {
max_tool_calls?: number | null;
};
+/**
+ * ChatMemory
+ *
+ * What the model reads instead of the messages a conversation's summary covers (long conversations are summarised;
+ * /compact and /clear).
+ */
+export type ChatMemory = {
+ /**
+ * Text
+ *
+ * the summary; empty after /clear, when the messages it covers aren't read at all
+ */
+ text: string;
+ /**
+ * Upto
+ *
+ * the last message it covers, of the version showing
+ */
+ upto: number;
+ /**
+ * At
+ *
+ * when it was written
+ */
+ at?: string | null;
+ /**
+ * Covers
+ *
+ * how many messages of the version showing it stands for
+ */
+ covers: number;
+ [key: string]: unknown;
+};
+
/**
* ChatMessage
*/
diff --git a/nextjs-frontend/components/chat/chat-app.tsx b/nextjs-frontend/components/chat/chat-app.tsx
index d30806d67..1a9d7be51 100644
--- a/nextjs-frontend/components/chat/chat-app.tsx
+++ b/nextjs-frontend/components/chat/chat-app.tsx
@@ -31,6 +31,7 @@ import {
type TurnState,
} from "@/components/chat/stream";
import { onlyFiles, UserBubble } from "@/components/chat/user-bubble";
+import { SummaryMark } from "@/components/chat/summary-mark";
import { switched, VersionSwitch } from "@/components/chat/versions";
import { SharedFrom } from "@/components/page-chat/page-chat";
import { useRecordingIndex, useSpeakerDirectory } from "@/components/search/data";
@@ -558,6 +559,8 @@ export function ChatApp() {
}, [chat.data?.id, chat.data]);
const msgs = chat.data?.messages ?? [];
+ // where the model stops reading the conversation word for word: under the last message its summary covers
+ const memory = chat.data?.summary ?? null;
const items = useMemo(() => {
const here = live && live.chatId === activeId;
// a version being written streams in place of the one beside it (what came before it has smaller ids)
@@ -799,6 +802,7 @@ export function ChatApp() {
/>
)}
+ {memory && it.q && it.a && memory.upto === it.q.id && }
{it.a?.kind === "research_plan" && it.a.meta?.research != null ? (
) : it.a ? (
@@ -875,6 +879,9 @@ export function ChatApp() {
)
)}
+ {memory && (it.a ? memory.upto === it.a.id : memory.upto === it.q?.id) && (
+
+ )}
);
})}
diff --git a/nextjs-frontend/components/chat/summary-mark.tsx b/nextjs-frontend/components/chat/summary-mark.tsx
new file mode 100644
index 000000000..9171a43f9
--- /dev/null
+++ b/nextjs-frontend/components/chat/summary-mark.tsx
@@ -0,0 +1,51 @@
+"use client";
+
+import { useState } from "react";
+
+import type { ChatMemory } from "@/app/openapi-client/types.gen";
+
+/**
+ * Under the last message a conversation's summary covers: the model reads the summary instead of the messages above
+ * it (or, after /clear, nothing of them). Show summary opens the text, so what the model remembers is never hidden.
+ */
+export function SummaryMark({ summary }: { summary: ChatMemory }) {
+ const [open, setOpen] = useState(false);
+ const n = summary.covers;
+ const some = `${n} message${n === 1 ? "" : "s"}`;
+ const cleared = !summary.text.trim();
+ return (
+
+
+
+
+ {cleared
+ ? `Cleared here: the assistant no longer reads the ${some} above.`
+ : `Summarised up to here: the assistant reads a summary of the ${some} above instead of them.`}
+ {!cleared && (
+ <>
+ {" "}
+
+ >
+ )}
+
+
+
+ {open && !cleared && (
+
+ {summary.text}
+
+ )}
+
+ );
+}
diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json
index 21e0aae43..679a29ecd 100644
--- a/nextjs-frontend/openapi.json
+++ b/nextjs-frontend/openapi.json
@@ -38900,6 +38900,17 @@
"type": "array",
"title": "Tree",
"description": "every message of every version, oldest first"
+ },
+ "summary": {
+ "anyOf": [
+ {
+ "$ref": "#/components/schemas/ChatMemory"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "what the model reads instead of the messages up to its `upto`; none before the first summary"
}
},
"additionalProperties": true,
@@ -39177,6 +39188,46 @@
"type": "object",
"title": "ChatCreate"
},
+ "ChatMemory": {
+ "properties": {
+ "text": {
+ "type": "string",
+ "title": "Text",
+ "description": "the summary; empty after /clear, when the messages it covers aren't read at all"
+ },
+ "upto": {
+ "type": "integer",
+ "title": "Upto",
+ "description": "the last message it covers, of the version showing"
+ },
+ "at": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "At",
+ "description": "when it was written"
+ },
+ "covers": {
+ "type": "integer",
+ "title": "Covers",
+ "description": "how many messages of the version showing it stands for"
+ }
+ },
+ "additionalProperties": true,
+ "type": "object",
+ "required": [
+ "text",
+ "upto",
+ "covers"
+ ],
+ "title": "ChatMemory",
+ "description": "What the model reads instead of the messages a conversation's summary covers (long conversations are summarised;\n/compact and /clear)."
+ },
"ChatMessage": {
"properties": {
"id": {