Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
of each conversation's own. Regenerating an answer or editing a question makes a new version beside the old one, and
nothing is lost. Each answer keeps a record of what produced it (the model, its tools, how long it took and what it
cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine
(opt-in), any message in one click, and `/clear` and `/compact` keep long conversations short. Conversations title
themselves after their first answer, unless you name them. Each namespace can have its own assistant with
(opt-in), any message in one click. Long conversations are summarised before they fill the model's context, and
`/clear` and `/compact` start afresh or summarise now. Conversations title themselves after their first answer, unless
you name them. Each namespace can have its own assistant with
its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in
Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you
email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages),
Expand Down
15 changes: 11 additions & 4 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,18 @@ then.
every answer gets that summary plus the messages it doesn't cover yet. Room conversations (chat-rooms.md) go on for
good, so they rely on it most. A summary covers the version it was written on: on a version that leaves what it
covers, the model reads that version's messages, and the next summary is made from them. A conversation that fills
half of the model's context window (**Chat model's context**, `ai.context_tokens`, 16000 when empty; about four
characters a token) is folded in early, before the next answer, whatever the count.
half of its model's context window (**Summarise when a conversation fills**, `ai.compact_at`, 50 %; about four
characters a token) is folded in early, before the next answer, whatever the count, so the excerpts, tools and the
answer fit in the rest. The context window is the conversation's model's: **Chat model's context**
(`ai.context_tokens`) when it's set; otherwise the context Lens's own [local model](local-models.md) runs with
(`local_llm.context`) when it's the provider; otherwise what the model server says (its model list's
`context_length`, `max_model_len`, `max_context_length` or `context_window`, as OpenRouter, vLLM, Mistral, Groq and
others give it, or llama.cpp's `/props`), asked once every ten minutes; otherwise 16000 tokens.
**Model for summaries** (`ai.compact_model`) writes them with a smaller, cheaper model on the same provider; empty,
the chat's model does. Settings › AI assistant › Summarise long conversations (`ai.compact`, on) turns it off; off,
or with no model, only the last six messages are read.
the chat's model does. A long stretch folded at once (`/compact`, or a conversation from before summaries) is
written in as many calls as it takes to fit that model's context, each in the [activity ledger](activity.md) as the
conversation's. Settings › AI assistant › Summarise long conversations (`ai.compact`, on) turns it off; off, or with
no model, only the last six messages are read.
- **Titles:**{#titles} a conversation nobody named is titled with its first question's words as soon as it's asked,
then, after the first answer, the model gives it a short title of a few words from the question and the answer
(streamed as `title` at the end of the answer's stream). It's one small call, written by **Model for summaries**
Expand Down
2 changes: 1 addition & 1 deletion docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` |
| Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` |
| Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` |
| `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app` |
| Long conversations summarised before they fill the model's context, and `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_older_messages_are_folded_into_a_summary`, `test_the_context_window_is_the_model_servers`, `test_a_long_conversation_is_folded_in_runs_that_fit_the_summary_model`, `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app` |
| A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` |
| Telegram and Slack built in, with voice notes and files | [test_telegram.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_telegram.py) `test_it_answers_private_chats_and_what_is_said_to_it_in_groups`, `test_voice_notes_are_heard`, `test_files_are_held_for_the_assistant_to_import`; [test_slack.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_slack.py) `test_it_answers_direct_messages_mentions_and_the_slash_command`, `test_voice_clips_are_heard_and_files_held` |
| Other chat rooms through Matterbridge | [test_bridge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_bridge.py) `test_it_answers_what_is_said_to_it_and_nothing_else`; [test_matterbridge_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_matterbridge_run.py) `test_chat_networks_are_set_in_the_app_and_become_matterbridges_config`, `test_the_bridge_talks_to_the_matterbridge_lens_runs`; [matterbridge.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/matterbridge.yml) `Lens writes a config, and Matterbridge starts with it` |
Expand Down
85 changes: 64 additions & 21 deletions fastapi_backend/app/domain/chat.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
from collections import Counter, defaultdict
from urllib.parse import urlsplit

from . import __version__, activity, llm, recsets, render, semantic, store, textindex, untrusted
from . import __version__, activity, llm, local_llm, recsets, render, semantic, store, textindex, untrusted

R = store.R
log = logging.getLogger("lens")
Expand Down Expand Up @@ -562,8 +562,9 @@ def migrate_tree(db):
RECENT = 6 # the latest messages, always given word for word
COMPACT_AFTER = 8 # how many more build up before they're folded into the summary (one model call per that many)
TURN_CHARS = 1500 # of each message, for the summary
CONTEXT_TOKENS = 16000 # the model's context window when ai.context_tokens isn't set
FULL = 0.5 # the share of it the conversation may take before it's folded in early (the rest: excerpts, tools, answer)
CONTEXT_TOKENS = 16000 # a model's context window when neither ai.context_tokens nor its server says (window)
COMPACT_AT = 50 # the % of it a conversation may fill before it's folded in early (ai.compact_at; the rest: excerpts, tools, answer)
SUMMARY_TOKENS = 800 # the most a summary is written in
COMPACT_SYSTEM = (
"You keep the running summary of a conversation between a person and their assistant, so the assistant still "
"knows what was said once the messages themselves are out of view. Fold the new messages into the summary so far: "
Expand Down Expand Up @@ -619,18 +620,46 @@ def tokens(messages):
return sum(len(m.get("content") or "") + len(attached_note(m.get("attachments"))) for m in messages) // 4


def full(cfg, messages):
"""True when the messages fill more than FULL of the model's context window (ai.context_tokens)."""
return tokens(messages) > (cfg["ai"].get("context_tokens") or CONTEXT_TOKENS) * FULL
def window(cfg, model=None):
"""A model's context window in tokens (by default the chat's): ai.context_tokens when it's set (for the chat's
models); the local model's (local_llm.context) when this machine's llama-server is the provider; what the model
server says (llm.context_window); else CONTEXT_TOKENS."""
if cfg["ai"].get("context_tokens"):
return cfg["ai"]["context_tokens"]
if local_llm.serves(cfg, model):
return local_llm.conf(cfg)["context"]
return llm.context_window(cfg, model) or CONTEXT_TOKENS


def full(cfg, messages, model=None):
"""True when the messages fill more than ai.compact_at % (COMPACT_AT) of the model's context window."""
return tokens(messages) > window(cfg, model) * (cfg["ai"].get("compact_at") or COMPACT_AT) / 100


def _batches(cfg, model, had, old):
"""`old` in runs that each fit the summary's model with the summary so far and room to write the next one, so a
long conversation folded at once (/compact, or one from before summaries) never overflows a small model."""
room = window(cfg, model) * 3 // 4 - SUMMARY_TOKENS - (len(COMPACT_SYSTEM) + len(had.get("text") or "") + 2000) // 4
run, size = [], 0
for m in old:
n = len(_said(m)) // 4
if run and size + n > room:
yield run
run, size = [], 0
run.append(m)
size += n
if run:
yield run


def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None):
"""Fold the messages older than the latest RECENT into the conversation's summary, once COMPACT_AFTER of them have
built up or they fill FULL of the model's context window (`full_only`: only then; ai.compact, on by default; needs a
model). `force` (/compact) folds every message in now, whatever the setting, with the person's `focus` (what to
keep) if any. The summary is written by ai.compact_model when set, a cheaper model than the chat's. It's of the
version `line` is (by default the head's), afresh when the summary was another version's. Returns the new summary
(with "folded": how many messages), or None when nothing changed."""
built up or they fill ai.compact_at % of the chat model's context window (window; `full_only`: only then;
ai.compact, on by default; needs a model). `force` (/compact) folds every message in now, whatever the setting,
with the person's `focus` (what to keep) if any. The summary is written by ai.compact_model when set, a cheaper
model than the chat's, in as many calls as it takes to fit that model's window (_batches), each in the ledger as
the conversation's. It's of the version `line` is (by default the head's), afresh when the summary was another
version's. Returns the new summary (with "folded": how many messages), or None when nothing changed."""
if not llm.configured(cfg) or not (force or cfg["ai"].get("compact", True)):
return None
line = path(db, cid) if line is None else line
Expand All @@ -639,18 +668,32 @@ def compact(db, cfg, cid, force=False, focus=None, full_only=False, line=None):
old = fresh if force else fresh[:-RECENT]
if not old:
return None
if not force and not full(cfg, fresh) and (full_only or len(fresh) < RECENT + COMPACT_AFTER):
return None
ask = f"Summary so far:\n{had.get('text') or '(none yet)'}\n\nNew messages:\n\n" + "\n\n".join(_said(m) for m in old)
if focus:
ask += f"\n\nThe person asked the summary to keep: {focus[:500]}"
msgs = [{"role": "system", "content": COMPACT_SYSTEM}, {"role": "user", "content": ask}]
text = llm.chat(cfg, msgs, model=cfg["ai"].get("compact_model") or None, max_tokens=800).strip()
if not text:
if not force and (full_only or len(fresh) < RECENT + COMPACT_AFTER):
model = (db.one("SELECT model FROM $r", r=R("chat", cid)) or {}).get("model") # the conversation's, if picked
if not full(cfg, fresh, model):
return None
writer = cfg["ai"].get("compact_model") or None
out, folded = None, 0
with activity.scope(db, f"chat:{cid}", cfg=cfg):
for run in _batches(cfg, writer, had, old):
ask = f"Summary so far:\n{had.get('text') or '(none yet)'}\n\nNew messages:\n\n" + "\n\n".join(_said(m) for m in run)
if focus:
ask += f"\n\nThe person asked the summary to keep: {focus[:500]}"
msgs = [{"role": "system", "content": COMPACT_SYSTEM}, {"role": "user", "content": ask}]
try:
text = llm.chat(cfg, msgs, model=writer, max_tokens=SUMMARY_TOKENS).strip()
except (llm.LLMError, activity.OverBudget):
if force or out is None:
raise
break # what's folded so far is kept; the rest is folded in next time
if not text:
break
had = out = {"text": text, "upto": run[-1]["id"], "at": store.now()}
folded += len(run)
if out is None or (force and folded < len(old)): # /compact folds everything or changes nothing
return None
out = {"text": text, "upto": old[-1]["id"], "at": store.now()}
db.q("UPDATE $r SET summary = $s", r=R("chat", cid), s=out)
return {**out, "folded": len(old)}
return {**out, "folded": folded}


# ---------- commands: /clear and /compact, typed in the app or in a chat room ----------
Expand Down
92 changes: 75 additions & 17 deletions fastapi_backend/app/domain/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,7 @@ def _post(cfg, payload):
l = cfg["llm"]
if not configured(cfg):
raise LLMError("no language model is configured (Settings → LLM provider)")
headers = {"Content-Type": "application/json"}
key = l.get("api_key") or (os.environ.get(l["api_key_env"]) if l.get("api_key_env") else None)
if key:
headers["Authorization"] = f"Bearer {key}"
headers = {"Content-Type": "application/json", **_headers(cfg)}
req = urllib.request.Request(
l["base_url"].rstrip("/") + "/chat/completions", data=json.dumps(payload).encode(), headers=headers, method="POST"
)
Expand All @@ -57,29 +54,90 @@ def _post(cfg, payload):
raise LLMError(f"can't reach the LLM server at {l['base_url']}: {e}") from None


_MODELS = {} # base_url -> (when, names): the server's list, kept a minute
_MODELS = {} # base_url -> (when, {name: what the server says of it}): the server's list, kept a minute
_WINDOWS = {} # (base_url, model) -> (when, tokens or None): each model's context window, kept ten minutes


def list_models(cfg, ttl=60):
"""The models the server offers (GET /models), kept for `ttl` seconds."""
def _headers(cfg):
l = cfg["llm"]
key = l.get("api_key") or (os.environ.get(l["api_key_env"]) if l.get("api_key_env") else None)
return {"Authorization": f"Bearer {key}"} if key else {}


def _listing(cfg, ttl=60):
"""The models the server offers (GET /models), each name with what the server says of it, kept for `ttl` seconds."""
l = cfg["llm"]
if not configured(cfg):
return []
url = l["base_url"].rstrip("/")
hit = _MODELS.get(url)
if hit and time.monotonic() - hit[0] < ttl:
return hit[1]
headers = {}
key = l.get("api_key") or (os.environ.get(l["api_key_env"]) if l.get("api_key_env") else None)
if key:
headers["Authorization"] = f"Bearer {key}"
try:
with urllib.request.urlopen(urllib.request.Request(url + "/models", headers=headers), timeout=10) as r:
names = sorted({str(m["id"]) for m in json.load(r).get("data") or [] if isinstance(m, dict) and m.get("id")})
with urllib.request.urlopen(urllib.request.Request(url + "/models", headers=_headers(cfg)), timeout=10) as r:
listed = {str(m["id"]): m for m in json.load(r).get("data") or [] if isinstance(m, dict) and m.get("id")}
except (urllib.error.URLError, TimeoutError, OSError, ValueError, AttributeError) as e:
raise LLMError(f"can't list the models at {l['base_url']}: {e}") from None
_MODELS[url] = (time.monotonic(), names)
return names
_MODELS[url] = (time.monotonic(), listed)
return listed


def list_models(cfg, ttl=60):
"""The models the server offers (GET /models), kept for `ttl` seconds."""
return sorted(_listing(cfg, ttl)) if configured(cfg) else []


# what a GET /models entry calls the model's context window: LM Studio's loaded context first (what it runs with), then
# OpenRouter's, Together's and others' context_length, vLLM's max_model_len, Mistral's max_context_length, Groq's
# context_window and max_input_tokens
WINDOW_KEYS = ("loaded_context_length", "context_length", "max_model_len", "max_context_length", "context_window", "max_input_tokens")


def _tokens(v):
if isinstance(v, str) and v.strip().isdigit():
v = int(v)
return int(v) if isinstance(v, (int, float)) and not isinstance(v, bool) and v >= 256 else None


def _window_of(entry):
for k in WINDOW_KEYS:
if n := _tokens(entry.get(k)):
return n
return _tokens((entry.get("top_provider") or {}).get("context_length")) if isinstance(entry.get("top_provider"), dict) else None


def _llamacpp_window(cfg):
"""The context llama.cpp's server runs each request with (GET /props at its root), which its model list doesn't
give (only the context the model was trained for)."""
root = re.sub(r"/v1/?$", "", cfg["llm"]["base_url"].rstrip("/"))
try:
with urllib.request.urlopen(urllib.request.Request(root + "/props", headers=_headers(cfg)), timeout=10) as r:
j = json.load(r)
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
raise LLMError(f"can't ask llama.cpp's server for its context: {e}") from None
if not isinstance(j, dict):
return None
gen = j.get("default_generation_settings")
return _tokens(gen.get("n_ctx")) if isinstance(gen, dict) else _tokens(j.get("n_ctx"))


def context_window(cfg, model=None, ttl=600):
"""The model's context window in tokens as its server says: its GET /models entry (WINDOW_KEYS; a server listing
one model speaks for whichever is asked for), or llama.cpp's /props. None when the server doesn't say or can't be
asked. Kept `ttl` seconds, a miss too, so a server that doesn't say isn't asked before every answer."""
if not configured(cfg):
return None
model = model or cfg["llm"]["model"]
key = (cfg["llm"]["base_url"].rstrip("/"), model)
hit = _WINDOWS.get(key)
if hit and time.monotonic() - hit[0] < ttl:
return hit[1]
try:
listed = _listing(cfg)
entry = listed.get(model) or (next(iter(listed.values())) if len(listed) == 1 else {})
n = _window_of(entry) or (_llamacpp_window(cfg) if entry.get("owned_by") == "llamacpp" else None)
except LLMError:
n = None
_WINDOWS[key] = (time.monotonic(), n)
return n


def _load(r):
Expand Down
Loading
Loading