Skip to content

feat(chat): summarise long chats at the model's real context window - #528

Merged
adeelahmad merged 4 commits into
mainfrom
claude/project-thread-w4pz1s
Oct 18, 2026
Merged

adeelahmad merged 4 commits into
mainfrom
claude/project-thread-w4pz1s

Conversation

@adeelahmad

Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: a long conversation was folded into its running summary early only once it filled half of a fixed 16000-token guess (unless an admin typed the model's context into Settings). Lens's own local model starts with a 4096-token context, so a chat on it overflowed long before that. A /compact of a long conversation sent every message to the summary model in one request, which a small model couldn't take.

After: the threshold is the conversation's model's real context window: Chat model's context (ai.context_tokens) when set; Lens's own local model's local_llm.context when it's the provider; what the model server lists (context_length, max_model_len, max_context_length, context_window, as OpenRouter, vLLM, Mistral, Groq and others give it) or llama.cpp's /props; else 16000. A new setting, Summarise when a conversation fills (ai.compact_at, 50 %), sets the share. A long stretch folded at once is written in runs that fit the summary model's own context, each call in the activity ledger as the conversation's (chat:<id>).

How: llm.context_window reads the model list it already fetched (now kept with each entry) and caches the answer, a miss too, for ten minutes, so a server that doesn't say isn't asked before every answer. chat.window picks the source; chat.compact looks it up only when the message count hasn't already triggered a fold. /compact stays all-or-nothing; an automatic fold keeps the runs it finished if a later one fails.

Motivation and Context

Adeel: "auto compress and manage the context of larger chats!" This builds on #269 rather than adding a second summariser.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md. (n/a: the title says it)

Done questions (AGENTS.md)

  • Goal: summaries now happen before the real context fills; merged on green.
  • Agents: ai.compact_at and ai.context_tokens are settable through PUT /api/v1/settings/ai and the assistant's settings tool; folding happens the same for API, MCP and chat-room conversations.
  • Zero setup: the window is read from the provider; nothing to fill in.
  • Fewest clicks: none needed.
  • Configurable: Settings › AI assistant › Summarise when a conversation fills (50 %), out of the wizard.
  • Light: one cached GET to the model server, none for Lens's own local model.
  • Private and secure: n/a, no new surface (the same model server Lens already calls).
  • Cost: ai.compact_model writes the summaries; each call is in the ledger under the conversation; budgets' hard stop applies as before.
  • User decides: ai.compact stays on as before; the default share is the old fixed 50 %. The only change for existing installs without ai.context_tokens is that a model whose server states its context is measured against that instead of 16000.
  • Upgrades: n/a, no change to stored data.
  • Honest: docs say where the window comes from.
  • Tests, docs: test_the_context_window_is_the_model_servers, test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model; docs/assistant.md, README, docs/requirements.md.
  • Open standards: OpenAI-compatible model list; no new dependencies.
  • Reported in the project thread; refine later: Ollama and LM Studio don't list the context on their OpenAI-compatible /models (their own APIs do), so they still fall back to 16000 unless set.
  • Reuse: extends chat.compact and llm.list_models; no second summariser.

Generated by Claude Code

A conversation was folded into its summary early only once it filled half
of a fixed 16000-token guess, so a local model started with 4096 overflowed
first. The window is now the conversation's model's: ai.context_tokens when
set, Lens's own local model's context, what the model server lists (or
llama.cpp's /props), else 16000. ai.compact_at sets the share (50 %).

A long stretch folded at once is written in runs that fit the summary
model, each call in the activity ledger as the conversation's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZAK1YALs8mQRwgGnyju4F
@adeelahmad adeelahmad self-assigned this Oct 17, 2026
The settings test's AI section lacked the new ai.compact_at, so Save did
nothing; the field now sits with the other long-conversation options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZAK1YALs8mQRwgGnyju4F
…-w4pz1s

# Conflicts:
#	README.md
#	docs/assistant.md
#	docs/requirements.md
#	fastapi_backend/app/domain/store.py
#	fastapi_backend/tests/api/_assist.py
…-w4pz1s

# Conflicts:
#	fastapi_backend/tests/api/_assist.py
@adeelahmad
adeelahmad merged commit 7c3956e into main Oct 18, 2026
13 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-w4pz1s branch October 18, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants