Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions changelog.d/582.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
- **A new conversation's first answer no longer goes missing.** A quick answer in a new conversation sometimes showed
"No answer was saved for this question" though it was saved: the conversation's first read, made before the answer
was saved, was used in place of reading it again. The web app (Chat, the page assistant and a recording's Chat tab)
now drops a read under way and reads the conversation afresh once the answer is saved. And an answer is now written
on a thread of its own, so closing the tab, losing the connection or moving to another page no longer drops it: it's
finished and saved all the same, and only Stop ends it early ([Chat](docs/assistant.md#chat)).
3 changes: 3 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ then.
([Titles](#titles)).
- **Stopping:** Stop asks the server to end the answer after the piece or tool step it's on (whichever server process
is writing it); what came before is saved, marked stopped. A model call already under way finishes first.
- **Leaving mid-answer:** an answer is written on a thread of its own, not by the connection reading it, so closing the
tab, losing the connection or moving to another page doesn't stop it: it's finished and saved, and opening the
conversation again shows it. Only Stop ends an answer early.
- **Reopening:** a conversation's answers keep the tools the assistant used (and with what), any notice (the model
couldn't use tools), the error when there was no answer, their latest source check and what produced them.
- **Versions:** a conversation is a tree, and nothing in it is deleted. Regenerating an answer writes a new answer to
Expand Down
3 changes: 3 additions & 0 deletions docs/frontend.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,9 @@ four-colour loop (Transcribe blue, Diarize red, Analyze green, Summarize and Rep
yet. Refreshing data already on screen (polling, refetching after a change) and anything quicker than 150 ms don't
show it.
* Streams (job events, chat answers) use `streamSSE()` with the session's access token.
* **Reading again after a change** the server just made: `refetchFresh(qc, key)` (`components/chat/data.ts`), not
`invalidateQueries` alone. A query's first read still on its way was asked before the change, and invalidating waits
for that read rather than making a new one, so a new conversation showed its question without the answer just saved.
* Media links in API responses are signed relative URLs; the proxy serves them from the app's origin, so they work in
`<audio>`, `<video>` and `<img>`.

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Deep research: a goal gets a plan to approve, runs in the background and comes back as a cited report, saved as a note or a podcast episode | [test_research.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_research.py) `test_a_goal_gets_a_plan_that_waits_for_approval_and_can_be_revised`, `test_an_approved_plan_runs_and_its_report_comes_back_checked`, `test_a_report_becomes_a_note_and_a_podcast`; [chat-research.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-research.test.tsx) `approves the plan with Enter, sending the edits`, `saves it as a note, asking where when it could go in several namespaces`, `turns the next message into a research goal, and says when it can't`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `chat-research`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `chat-research` ≤ 4 clicks |
| Regenerating an answer or editing a question makes a new version beside the old one | [test_chat_versions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_versions.py) `test_regenerate_edit_switch_and_go_on`; [chat-versions.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-versions.test.tsx) `regenerating an answer`, `the version switch`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `chat-regenerate`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `chat-regenerate` ≤ 4 clicks |
| Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` |
| An answer is finished, saved and shown even when the person leaves before it's done; only Stop ends it early | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_an_answer_is_saved_when_the_client_leaves`, `test_stopping_an_answer`; [chat-fresh-answer.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-fresh-answer.test.tsx) `shows the answer even when the conversation's first read left before it was saved` |
| Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` |
| Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` |
| Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` |
Expand Down
40 changes: 38 additions & 2 deletions fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,11 @@

from __future__ import annotations

import contextvars
import json
import logging
import queue
import threading
import urllib.error
import urllib.parse
from collections.abc import Iterator
Expand Down Expand Up @@ -431,6 +434,39 @@ def _ev(name: str, data: Any) -> str:
return f"event: {name}\ndata: {json.dumps(sign_urls(data, full=True), default=str)}\n\n"


_END = object()


def _detached(events: Iterator[str], name: str) -> Iterator[str]:
"""`events` run to the end on a thread of their own, handed over as they come. A client that leaves mid-answer (a
closed tab, a dropped connection, a page that moved on) stops the stream, which no longer stops the work: the answer
is still saved, so the question isn't left unanswered when the conversation is opened again. The work's own cleanup
runs on its thread too, never on whichever thread happens to collect an abandoned generator."""
out: queue.Queue[Any] = queue.Queue()
left = threading.Event() # the client went away: nobody reads what comes next

def work() -> None:
try:
for ev in events:
if not left.is_set():
out.put(ev)
except Exception: # noqa: BLE001 - the stream ends; `events` saves its own answer when it breaks off
log.exception("%s failed", name)
finally:
out.put(_END)

threading.Thread(target=contextvars.copy_context().run, args=(work,), daemon=True, name=name).start()
ended = False
try:
while (ev := out.get()) is not _END:
yield ev
ended = True
finally:
if not ended:
left.set()
log.info("%s: the client left before the end; it's finished and saved all the same", name)


def _research_reply(
cid: int,
c: dict[str, Any],
Expand Down Expand Up @@ -809,8 +845,8 @@ def gen() -> Iterator[str]:
if titled := chat.title_soon(db, cfg, cid, q, answered["text"]):
yield _ev("title", {"title": titled})

# A sync generator: Starlette iterates it in the threadpool, so the domain calls inside don't block the loop.
return StreamingResponse(gen(), media_type="text/event-stream", headers=SSE_HEADERS)
# Answered on a thread of its own (the domain calls inside don't block the loop) and saved even if the client leaves
return StreamingResponse(_detached(gen(), f"chat-answer-{cid}"), media_type="text/event-stream", headers=SSE_HEADERS)


@router.post("/chats/{cid}/stop")
Expand Down
77 changes: 77 additions & 0 deletions fastapi_backend/tests/api/test_chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@

import io
import json
import threading
import time

import anyio
import pytest

from app.domain import ai_tools, auth, chat, store, topics
Expand Down Expand Up @@ -215,6 +218,80 @@ def broken(*_a, **_k):
)


async def _leave_after_first_token(app, path, headers, body, left):
"""POST `path` as a browser that leaves once the first piece of the answer has arrived (a closed tab, a page that
moved on): the server tells the app the client disconnected, as uvicorn does. What it was sent before leaving."""
got, token, asked = [], anyio.Event(), False

async def receive():
nonlocal asked
if not asked:
asked = True
return {"type": "http.request", "body": body, "more_body": False}
await token.wait()
left.set()
return {"type": "http.disconnect"}

async def send(message):
if message["type"] == "http.response.body":
got.append(message.get("body", b""))
if b"event: token" in b"".join(got):
token.set()

scope = {
"type": "http",
"asgi": {"version": "3.0", "spec_version": "2.3"},
"http_version": "1.1",
"method": "POST",
"scheme": "http",
"path": path,
"raw_path": path.encode(),
"root_path": "",
"query_string": b"",
"headers": [(b"host", b"127.0.0.1"), (b"content-type", b"application/json")]
+ [(k.lower().encode(), v.encode()) for k, v in headers.items()],
"client": ("127.0.0.1", 50000),
"server": ("127.0.0.1", 80),
}
await app(scope, receive, send)
return b"".join(got).decode()


def test_an_answer_is_saved_when_the_client_leaves(plain, app, client, db, cfg, folder, monkeypatch):
seed(db, cfg, folder)
make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor", "calls": "editor"})
h = login(client, "ed@x.io", "editor password 1")
cid = client.post("/api/v1/chats", headers=h, json={}).json()["id"]
left = threading.Event()

def stream(*_a, **_k): # the rest of the answer comes after the browser has gone
yield "The shipment "
assert left.wait(10)
yield "leaves on Friday [1]."

monkeypatch.setattr(llm_mod, "stream_chat", stream)
body = json.dumps({"content": "When does the shipment leave?"}).encode()
seen = anyio.run(_leave_after_first_token, app, f"/api/v1/chats/{cid}/messages", h, body, left)
assert "event: token" in seen and "event: done" not in seen # it left before the end

# the answer is finished and saved all the same, so opening the conversation again shows it, not a bare question
for _ in range(100):
msgs = client.get(f"/api/v1/chats/{cid}", headers=h).json()["messages"]
if len(msgs) == 2:
break
time.sleep(0.1)
assert [(m["role"], m["content"]) for m in msgs] == [
("user", "When does the shipment leave?"),
("assistant", "The shipment leaves on Friday [1]."),
]
assert msgs[1]["passages"] and not msgs[1].get("stopped")
for _ in range(50): # and it's no longer marked as being written, so Stop has nothing to stop
if not (db.one("SELECT answering FROM $r", r=store.R("chat", cid)) or {}).get("answering"):
break
time.sleep(0.1)
assert client.post(f"/api/v1/chats/{cid}/stop", headers=h).json()["stopping"] is False


def test_stopping_between_tool_steps(client, db, cfg, folder, llm, monkeypatch):
seed(db, cfg, folder)
make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"})
Expand Down
52 changes: 52 additions & 0 deletions nextjs-frontend/__tests__/chat-fresh-answer.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import { QueryClient, QueryObserver } from "@tanstack/react-query";

import { refetchFresh } from "@/components/chat/data";

jest.mock("@/lib/api/browser", () => ({ data: jest.fn(), useApiClient: jest.fn() }));

describe("a conversation read again after its answer is saved", () => {
it("shows the answer even when the conversation's first read left before it was saved", async () => {
// A new conversation: the page opens it (its first read) while the answer is still streaming; the server reads it
// before the answer is saved, and that reply arrives only after the stream has ended.
const saved: string[] = ["Who said what about Raspberry Pi?"];
const late: Array<() => void> = [];
const reads: number[] = [];
const queryFn = () => {
const now = [...saved]; // what the server has when the read reaches it
reads.push(now.length);
return reads.length === 1
? new Promise<string[]>((resolve) => late.push(() => resolve(now)))
: Promise.resolve(now);
};
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const observer = new QueryObserver(qc, { queryKey: ["chat", 2], queryFn, staleTime: 10_000 });
const unsubscribe = observer.subscribe(() => undefined);
expect(reads).toEqual([1]); // the first read is on its way

saved.push("This is the survey's stand-in model answering."); // the stream ends with `done`: the answer is saved
const refreshed = refetchFresh(qc, ["chat", 2]);
late.forEach((reply) => reply()); // the first read's reply, from before the save, comes back
await refreshed;

expect(reads).toEqual([1, 2]); // read again after the save, not only the read already on its way
expect(observer.getCurrentResult().data).toEqual([
"Who said what about Raspberry Pi?",
"This is the survey's stand-in model answering.",
]);
unsubscribe();
qc.clear();
});

it("reads a conversation that is already showing again", async () => {
let answer = "first";
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const observer = new QueryObserver(qc, { queryKey: ["chat", 3], queryFn: async () => answer, staleTime: 10_000 });
const unsubscribe = observer.subscribe(() => undefined);
await qc.getQueryCache().find({ queryKey: ["chat", 3] })?.promise;
answer = "second";
await refetchFresh(qc, ["chat", 3]);
expect(observer.getCurrentResult().data).toBe("second");
unsubscribe();
qc.clear();
});
});
7 changes: 2 additions & 5 deletions nextjs-frontend/components/chat/chat-app.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import { AttachmentChips, useAttachments } from "@/components/chat/attachments";
import { shortTitle } from "@/components/chat/cite";
import { Composer, ScopeBar } from "@/components/chat/composer";
import { ConversationList } from "@/components/chat/conversations";
import { useApprovals, useChat, useChats, useLlmStatus, useStopAnswer } from "@/components/chat/data";
import { refetchFresh, useApprovals, useChat, useChats, useLlmStatus, useStopAnswer } from "@/components/chat/data";
import { EmptyChat } from "@/components/chat/empty";
import { ExportMenu, useChatExport } from "@/components/chat/export";
import { NamespaceOffer, spokenPick } from "@/components/chat/namespace-offer";
Expand Down Expand Up @@ -313,10 +313,7 @@ export function ChatApp() {
[mid]: { steps: turn.steps, notice: turn.notice, error: turn.error },
}));
if (turn.status === "done") setAnnounce(`Answer: ${turn.text.replace(/\[\d+(?:,\s*\d+)*\]/g, "")}`);
await Promise.all([
qc.invalidateQueries({ queryKey: ["chat", cid] }),
qc.invalidateQueries({ queryKey: ["approvals", cid] }),
]);
await Promise.all([refetchFresh(qc, ["chat", cid]), refetchFresh(qc, ["approvals", cid])]);
qc.invalidateQueries({ queryKey: ["chats"] });
if (mid == null) setPartials((p) => ({ ...p, [`${cid}:${question}`]: turn }));
setLive(null);
Expand Down
12 changes: 11 additions & 1 deletion nextjs-frontend/components/chat/data.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,21 @@
"use client";

import { useQuery } from "@tanstack/react-query";
import { type QueryClient, type QueryKey, useQuery } from "@tanstack/react-query";
import { useCallback } from "react";

import { Chats } from "@/app/openapi-client";
import { data, useApiClient } from "@/lib/api/browser";

/**
* Reads `key` again once something has changed on the server (an answer just saved). A first read still on its way
* was asked before the change, and invalidating alone waits for that read instead of making a new one, which would show
* a new conversation's question without the answer that was just saved; so a read already under way is dropped first.
*/
export async function refetchFresh(qc: QueryClient, key: QueryKey) {
await qc.cancelQueries({ queryKey: key });
await qc.invalidateQueries({ queryKey: key });
}

export function useChats() {
const client = useApiClient();
return useQuery({
Expand Down
4 changes: 2 additions & 2 deletions nextjs-frontend/components/page-chat/page-chat.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { Chats } from "@/app/openapi-client";
import type { ChatMessage, Passage, SharedContext } from "@/app/openapi-client/types.gen";
import { NO_ANSWER } from "@/components/chat/cite";
import { CitationChip } from "@/components/chat/citation";
import { useChat, useLlmStatus, useStopAnswer } from "@/components/chat/data";
import { refetchFresh, useChat, useLlmStatus, useStopAnswer } from "@/components/chat/data";
import { RichText } from "@/components/chat/rich-text";
import { SharedFiles } from "@/components/chat/shared-files";
import { ToolSteps } from "@/components/chat/steps";
Expand Down Expand Up @@ -312,7 +312,7 @@ function ChatPanel({
}
if (cid == null) return setLive({ chatId: -1, turn, context: shown });
setLive({ chatId: cid, turn, context: shown });
await qc.invalidateQueries({ queryKey: ["chat", cid] });
await refetchFresh(qc, ["chat", cid]);
void qc.invalidateQueries({ queryKey: ["chats"] });
// a saved answer is in the conversation now; one that wasn't saved stays shown
if (turn.messageId != null) setLive(null);
Expand Down
4 changes: 2 additions & 2 deletions nextjs-frontend/components/recording/chat-tab.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { useEffect, useRef, useState } from "react";

import { Chats } from "@/app/openapi-client";
import { citeLabel } from "@/components/chat/cite";
import { useStopAnswer } from "@/components/chat/data";
import { refetchFresh, useStopAnswer } from "@/components/chat/data";
import { usePlayerApi } from "@/components/player/media";
import {
applyChatEvent,
Expand Down Expand Up @@ -121,7 +121,7 @@ function ChatPanel() {
setAnswer((a) => (a ? applyChatEvent(a, ev) : a));
}
setAnswer((a) => (a && a.status === "streaming" ? { ...a, status: "done" } : a));
await qc.invalidateQueries({ queryKey: ["chat", target] });
await refetchFresh(qc, ["chat", target]);
setAnswer((a) => (a?.status === "done" || a?.saved ? null : a)); // a saved one shows in the conversation now
} catch (e) {
if (ctl.signal.aborted) setAnswer((a) => (a ? { ...a, status: "stopped" } : a));
Expand Down
Loading