Repository navigation
feat(chat): summarise long chats at the model's real context window - #528
Merged
Merged
Conversation
A conversation was folded into its summary early only once it filled half of a fixed 16000-token guess, so a local model started with 4096 overflowed first. The window is now the conversation's model's: ai.context_tokens when set, Lens's own local model's context, what the model server lists (or llama.cpp's /props), else 16000. ai.compact_at sets the share (50 %). A long stretch folded at once is written in runs that fit the summary model, each call in the activity ledger as the conversation's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZAK1YALs8mQRwgGnyju4F
The settings test's AI section lacked the new ai.compact_at, so Save did nothing; the field now sits with the other long-conversation options. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZAK1YALs8mQRwgGnyju4F
…-w4pz1s # Conflicts: # README.md # docs/assistant.md # docs/requirements.md # fastapi_backend/app/domain/store.py # fastapi_backend/tests/api/_assist.py
…-w4pz1s # Conflicts: # fastapi_backend/tests/api/_assist.py
20 of 23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: a long conversation was folded into its running summary early only once it filled half of a fixed 16000-token guess (unless an admin typed the model's context into Settings). Lens's own local model starts with a 4096-token context, so a chat on it overflowed long before that. A
/compactof a long conversation sent every message to the summary model in one request, which a small model couldn't take.After: the threshold is the conversation's model's real context window: Chat model's context (
ai.context_tokens) when set; Lens's own local model'slocal_llm.contextwhen it's the provider; what the model server lists (context_length,max_model_len,max_context_length,context_window, as OpenRouter, vLLM, Mistral, Groq and others give it) or llama.cpp's/props; else 16000. A new setting, Summarise when a conversation fills (ai.compact_at, 50 %), sets the share. A long stretch folded at once is written in runs that fit the summary model's own context, each call in the activity ledger as the conversation's (chat:<id>).How:
llm.context_windowreads the model list it already fetched (now kept with each entry) and caches the answer, a miss too, for ten minutes, so a server that doesn't say isn't asked before every answer.chat.windowpicks the source;chat.compactlooks it up only when the message count hasn't already triggered a fold./compactstays all-or-nothing; an automatic fold keeps the runs it finished if a later one fails.Motivation and Context
Adeel: "auto compress and manage the context of larger chats!" This builds on #269 rather than adding a second summariser.
Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md. (n/a: the title says it)Done questions (AGENTS.md)
ai.compact_atandai.context_tokensare settable throughPUT /api/v1/settings/aiand the assistant's settings tool; folding happens the same for API, MCP and chat-room conversations.ai.compact_modelwrites the summaries; each call is in the ledger under the conversation; budgets' hard stop applies as before.ai.compactstays on as before; the default share is the old fixed 50 %. The only change for existing installs withoutai.context_tokensis that a model whose server states its context is measured against that instead of 16000.test_the_context_window_is_the_model_servers,test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model; docs/assistant.md, README, docs/requirements.md./models(their own APIs do), so they still fall back to 16000 unless set.chat.compactandllm.list_models; no second summariser.Generated by Claude Code