Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Read any message aloud.** Every question and answer in a conversation has a speaker button: one click reads it
aloud by the speech model and voice picked in Settings → AI assistant → Voice (else the browser's own speech),
another stops it. One message plays at a time, long answers go in pieces under every provider's limit, and each
piece's audio is kept in the page, so playing it again doesn't call the model again. Each call is logged in the
activity ledger ([Read a message aloud](docs/assistant.md#read-a-message-aloud)).
- **Conversations title themselves.** After a conversation's first answer, the model gives it a short title from the
question and the answer, with the model for summaries (`ai.compact_model`) when one is set, in one call counted for
the conversation in the activity ledger; until then, and when there's no model, it's titled with the first
Expand Down
11 changes: 11 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

Plans and progress for work in flight. Newest first.

## 2026-10-17 · Read any chat message aloud

Adeel: play any message in the chat through the TTS model. A speaker button on every question and answer reads it by
`POST /voice/speak` (so the provider layer, the voice picker's voice and the activity ledger are voice mode's), one
message at a time, in pieces of at most 1,500 characters (Deepgram's limit is 2,000; `speech.speak` cuts at 4,000),
with the next piece asked for while one plays. Audio is kept in the page by text, the 32 most recent pieces.

Refine later: keep audio across reloads (encrypted on the server, or the browser's Cache Storage); a shorter first
piece so long answers start sooner; voice mode reading answers through the same pieces; pause and resume rather than
stop; the speaker button in an asset's Chat tab; drop kept audio when the voice is changed.

## 2026-10-17 · Dates read as days

Adeel: dates extraction. `dates.py` reads the dates each line gives (written out, numeric, and relative to the day)
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
of each conversation's own. Regenerating an answer or editing a question makes a new version beside the old one, and
nothing is lost. Each answer keeps a record of what produced it (the model, its tools, how long it took and what it
cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine
(opt-in), and `/clear` and `/compact` keep long conversations short. Conversations title themselves after their
first answer, unless you name them. Each namespace can have its own assistant with
(opt-in), any message in one click, and `/clear` and `/compact` keep long conversations short. Conversations title
themselves after their first answer, unless you name them. Each namespace can have its own assistant with
its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in
Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you
email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages),
Expand Down
11 changes: 11 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,17 @@ Voice goes through one hook, `useVoice()` in `nextjs-frontend/lib/voice.ts` (`li
or by ElevenLabs, Deepgram, Google Gemini or macOS `say` ([Speech providers](speech-providers.md#voices)); else by
the browser.

#### Read a message aloud

Every question and answer in a conversation has a speaker button: one click reads it aloud, another stops it. It goes
through the same `POST /voice/speak` as voice mode, so it uses the speech model and the voice picked in Settings → AI
assistant → Voice, without citation marks or Markdown, and falls back to the browser's own speech when the server has
none. One message is read at a time (starting another stops the first, and so does leaving the conversation). Long
answers are sent in pieces of at most 1,500 characters, cut after a sentence, under every provider's limit; the next
piece is asked for while one plays. Each piece's audio is kept in the page (the 32 most recently played), so playing
a message again doesn't call the speech model again. Each call is a `model.speech` row in the
[activity ledger](activity.md), like voice mode's.

Settings → AI assistant → Voice holds these. The API: `GET /voice` (what the server does; it also starts loading the
engine), `POST /voice/transcribe` (a clip as the raw body; nothing is kept), `POST /voice/speak` (`{text, voice?}`;
204 when the browser should read it), `GET /voice/voices` (the voices a provider reads in).
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Regenerating an answer or editing a question makes a new version beside the old one | [test_chat_versions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_versions.py) `test_regenerate_edit_switch_and_go_on`; [chat-versions.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-versions.test.tsx) `regenerating an answer`, `the version switch` |
| Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` |
| Answers read aloud by a speech provider, macOS `say` or StyleTTS2 (opt-in) | [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_answers_are_read_aloud_by_a_speech_model`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_macos_say_reads_aloud_on_a_mac`, `test_styletts2_reads_aloud_from_its_own_service`; [styletts2.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/styletts2.yml) `Read a sentence aloud` |
| Any message read aloud in one click, one at a time, its audio kept so a replay costs nothing | [chat-read-aloud.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-read-aloud.test.tsx) `plays an answer by the speech model in one click, stops on the next, and keeps its audio`, `reads one message at a time: starting another stops the first`, `reads a long answer piece by piece, asking for the next while one plays` |
| Conversations title themselves after their first answer, and a name you give is kept | [test_chat_titles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_titles.py) `test_a_first_answer_titles_the_conversation`, `test_a_title_you_give_is_kept`, `test_agents_rename_and_retitle_over_mcp` |
| `/clear` and `/compact` | [test_compaction.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_compaction.py) `test_clear_starts_afresh_and_compact_folds_everything_now`, `test_commands_in_the_app` |
| A namespace's own assistant, with its own name, instructions and memory | [test_ns_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ns_assistant.py) `test_off_by_default_and_owners_turn_it_on`, `test_remembers_across_conversations_with_citations`; [ns-assistant.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ns-assistant.test.tsx) `saving a name and forgetting a memory` |
Expand Down
187 changes: 187 additions & 0 deletions nextjs-frontend/__tests__/chat-read-aloud.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { act, fireEvent, render, screen, waitFor } from "@testing-library/react";
import "@testing-library/jest-dom";

import { Voice as VoiceApi } from "@/app/openapi-client";
import { Answer } from "@/components/chat/answer";
import { UserBubble } from "@/components/chat/user-bubble";
import { TooltipProvider } from "@/components/ui/tooltip";
import { forgetVoice, pieces, PIECE_CHARS, stopReading } from "@/lib/voice";

jest.mock("@/app/openapi-client", () => ({
Voice: { voiceInfo: jest.fn(), speakText: jest.fn() },
Chats: { checkMessage: jest.fn() },
Search: {},
}));
jest.mock("@/lib/api/browser", () => ({
useApiClient: () => ({}),
data: (p: Promise<{ data: unknown }>) => p.then((r) => r.data),
}));
jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) }));
jest.mock("@/lib/hooks/session", () => ({
useArchive: () => ({ can: () => true, namespaces: [], roleIn: () => "editor" }),
}));
jest.mock("@/components/chat/attachments", () => ({ SentFiles: () => null }));

const m = (f: unknown) => f as jest.Mock;

class FakeAudio {
static playing: FakeAudio[] = [];
static played: string[] = [];
onended: (() => void) | null = null;
onerror: (() => void) | null = null;
paused = false;
constructor(public src: string) {}
play() {
FakeAudio.played.push(this.src);
FakeAudio.playing.push(this);
return Promise.resolve();
}
pause() {
this.paused = true;
}
/** It plays to the end. */
end() {
this.onended?.();
}
}

let clipN = 0;
beforeAll(() => {
window.matchMedia = ((query: string) => ({
matches: false,
media: query,
addEventListener: () => {},
removeEventListener: () => {},
})) as unknown as typeof window.matchMedia;
});
beforeEach(() => {
jest.clearAllMocks();
forgetVoice();
stopReading();
FakeAudio.playing = [];
FakeAudio.played = [];
(window as unknown as Record<string, unknown>).Audio = FakeAudio;
URL.createObjectURL = jest.fn(() => `blob:${++clipN}`);
URL.revokeObjectURL = jest.fn();
m(VoiceApi.voiceInfo).mockResolvedValue({ data: { transcribe: false, speak: true }, response: { status: 200 } });
m(VoiceApi.speakText).mockImplementation(() =>
Promise.resolve({ data: new Blob(["mp3"]), response: { status: 200 } }),
);
});

function show(ui: React.ReactNode) {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
return render(
<QueryClientProvider client={qc}>
<TooltipProvider>{ui}</TooltipProvider>
</QueryClientProvider>,
);
}

const answer = (text: string, messageId = 7) => (
<Answer
chatId={1}
messageId={messageId}
question="When does it leave?"
text={text}
passages={[]}
status="done"
scope={{}}
model="m"
hover={null}
onHover={() => {}}
onPreview={() => {}}
/>
);

describe("reading a message aloud", () => {
it("cuts long text after sentences, under every provider's limit", () => {
const sentence = "The ship leaves on Friday at noon. ";
const long = sentence.repeat(100);
const out = pieces(long);
expect(out.length).toBeGreaterThan(1);
expect(out.every((p) => p.length <= PIECE_CHARS && p.endsWith("."))).toBe(true);
expect(out.join(" ")).toBe(long.trim());
expect(pieces("one two three four", 9)).toEqual(["one two", "three", "four"]);
expect(pieces(" ")).toEqual([]);
});

it("has nothing to read in an answer that is only code", () => {
show(answer("```\nls -la\n```"));
expect(screen.queryByRole("button", { name: "Read aloud" })).not.toBeInTheDocument();
});

it("plays an answer by the speech model in one click, stops on the next, and keeps its audio", async () => {
show(answer("It leaves on **Friday** [1]."));
fireEvent.click(screen.getByRole("button", { name: "Read aloud" }));
await waitFor(() => expect(FakeAudio.played).toHaveLength(1));
expect(m(VoiceApi.speakText).mock.calls[0][0].body).toEqual({ text: "It leaves on Friday." });
const stop = screen.getByRole("button", { name: "Stop reading aloud" });
expect(stop).toHaveAttribute("aria-pressed", "true");

fireEvent.click(stop);
expect(FakeAudio.playing[0].paused).toBe(true);
expect(screen.getByRole("button", { name: "Read aloud" })).toBeInTheDocument();

fireEvent.click(screen.getByRole("button", { name: "Read aloud" })); // again: the kept audio, no second call
await waitFor(() => expect(FakeAudio.played).toHaveLength(2));
expect(VoiceApi.speakText).toHaveBeenCalledTimes(1);
act(() => FakeAudio.playing[1].end());
await waitFor(() => expect(screen.getByRole("button", { name: "Read aloud" })).toBeInTheDocument());
});

it("reads one message at a time: starting another stops the first", async () => {
show(
<>
<UserBubble text="When does the ship leave?" />
{answer("It leaves on Friday.")}
</>,
);
const [question, reply] = screen.getAllByRole("button", { name: "Read aloud" });
fireEvent.click(reply);
await waitFor(() => expect(FakeAudio.played).toHaveLength(1));
fireEvent.click(question);
await waitFor(() => expect(FakeAudio.played).toHaveLength(2));
expect(FakeAudio.playing[0].paused).toBe(true);
expect(m(VoiceApi.speakText).mock.calls[1][0].body).toEqual({ text: "When does the ship leave?" });
expect(screen.getAllByRole("button", { name: "Stop reading aloud" })).toHaveLength(1);
expect(screen.getAllByRole("button", { name: "Read aloud" })).toHaveLength(1);
});

it("stops when the message leaves the page", async () => {
const { unmount } = show(answer("It leaves on Friday."));
fireEvent.click(screen.getByRole("button", { name: "Read aloud" }));
await waitFor(() => expect(FakeAudio.played).toHaveLength(1));
unmount();
expect(FakeAudio.playing[0].paused).toBe(true);
});

it("reads a long answer piece by piece, asking for the next while one plays", async () => {
show(answer(Array.from({ length: 120 }, (_, i) => `The ship number ${i} leaves on Friday at noon.`).join(" ")));
fireEvent.click(screen.getByRole("button", { name: "Read aloud" }));
await waitFor(() => expect(FakeAudio.played).toHaveLength(1));
await waitFor(() => expect(VoiceApi.speakText).toHaveBeenCalledTimes(2)); // the next piece, while this one plays
act(() => FakeAudio.playing[0].end());
await waitFor(() => expect(FakeAudio.played).toHaveLength(2));
for (const [{ body }] of m(VoiceApi.speakText).mock.calls)
expect(body.text.length).toBeLessThanOrEqual(PIECE_CHARS);
});

it("is read by the browser when the server has no speech model", async () => {
m(VoiceApi.voiceInfo).mockResolvedValue({ data: { transcribe: false, speak: false }, response: { status: 200 } });
const said: string[] = [];
const speak = jest.fn((u: { text: string; onend?: () => void }) => said.push(u.text));
Object.assign(window, { speechSynthesis: { speak, cancel: jest.fn() } });
(window as unknown as Record<string, unknown>).SpeechSynthesisUtterance = class {
lang = "";
onend: (() => void) | null = null;
onerror: (() => void) | null = null;
constructor(public text: string) {}
};
show(answer("It leaves on Friday [1]."));
fireEvent.click(screen.getByRole("button", { name: "Read aloud" }));
await waitFor(() => expect(speak).toHaveBeenCalled());
expect(said).toEqual(["It leaves on Friday."]);
});
});
1 change: 1 addition & 0 deletions nextjs-frontend/__tests__/chat.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { parseBlocks, RichText } from "@/components/chat/rich-text";
import { applyEvent, newTurn, savedSteps, stepCall } from "@/components/chat/stream";
import { UserBubble } from "@/components/chat/user-bubble";

jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) }));
jest.mock("@/components/chat/attachments", () => ({
SentFiles: ({ files }: { files: { filename: string }[] }) => (
<ul aria-label="Files sent">
Expand Down
2 changes: 2 additions & 0 deletions nextjs-frontend/components/chat/answer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import { CitationChip } from "@/components/chat/citation";
import { AnswerDetails } from "@/components/chat/details";
import { AnswerMenu, type ExportFormat } from "@/components/chat/export";
import { keywords } from "@/components/chat/keywords";
import { ReadAloud } from "@/components/chat/read-aloud";
import { RichText } from "@/components/chat/rich-text";
import { scopeWords, type Scope } from "@/components/chat/scope";
import { SharedFiles } from "@/components/chat/shared-files";
Expand Down Expand Up @@ -364,6 +365,7 @@ export function Answer(p: AnswerProps) {
{check.isPending ? "Checking sources…" : "Check sources"}
</Button>
)}
{!streaming && !blank && !noModel && !nothing && !failed && <ReadAloud text={p.text} />}
{!streaming && !blank && !noModel && <AnswerMenu text={p.text} passages={p.passages} onExport={p.onExport} />}
</div>
{check.isError && (
Expand Down
25 changes: 25 additions & 0 deletions nextjs-frontend/components/chat/read-aloud.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
"use client";

import { Loader2, Square, Volume2 } from "lucide-react";

import { IconButton } from "@/components/ui/button";
import { cn } from "@/lib/utils";
import { speakable, useReadAloud } from "@/lib/voice";

/** The play button on a chat message (docs/assistant.md, "Read a message aloud"): one click reads it aloud by the
* text-to-speech model in the voice picked in Settings (else the browser), another stops it. */
export function ReadAloud({ text, size = 32, className }: { text: string; size?: number; className?: string }) {
const r = useReadAloud(text);
if (!speakable(text)) return null; // only code, say: nothing to read
return (
<IconButton
label={r.on ? "Stop reading aloud" : "Read aloud"}
size={size}
active={r.on}
onClick={r.toggle}
className={cn("print:hidden [&_svg]:size-4", className, r.on && "opacity-100")}
>
{r.loading ? <Loader2 className="animate-spin" /> : r.on ? <Square /> : <Volume2 />}
</IconButton>
);
}
2 changes: 2 additions & 0 deletions nextjs-frontend/components/chat/user-bubble.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { useState } from "react";

import { SentFiles } from "@/components/chat/attachments";
import { CopyQuestion } from "@/components/chat/copy";
import { ReadAloud } from "@/components/chat/read-aloud";
import { Button, IconButton } from "@/components/ui/button";
import { cn } from "@/lib/utils";

Expand Down Expand Up @@ -74,6 +75,7 @@ export function UserBubble({
}
return (
<div className="group flex items-start justify-end gap-1 self-end">
{!filesOnly && <ReadAloud text={text} className={REVEAL} />}
{!filesOnly && <CopyQuestion text={text} className={REVEAL} />}
{onEdit && (
<IconButton
Expand Down
Loading
Loading