Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/workflows/styletts2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: StyleTTS2

# The StyleTTS2 service Lens can read aloud with (the optional styletts2 service, styletts2/): it builds, lists its
# voices, and reads a sentence aloud as WAV (fetching the model on the way).
on:
pull_request:
paths:
- styletts2/**
- .github/workflows/styletts2.yml
push:
branches: [main]
paths:
- styletts2/**
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
contents: read

jobs:
styletts2:
name: Build and read aloud
runs-on: ${{ fromJSON(vars.CI_RUNNER || '"ubuntu-latest"') }}
timeout-minutes: 45
steps:
- uses: actions/checkout@v4

- name: Build
run: docker build -t lens-styletts2:ci styletts2

- name: Start
run: |
docker run -d --name st -p 127.0.0.1:8880:8880 lens-styletts2:ci
for _ in $(seq 1 30); do curl -fsS http://127.0.0.1:8880/health && break; sleep 2; done
curl -fsS http://127.0.0.1:8880/v1/audio/voices | tee /dev/stderr | grep -q '"default"'

- name: Read a sentence aloud
run: |
code=$(curl -sS -o speech.wav -w '%{http_code}' --max-time 1500 \
-H 'Content-Type: application/json' \
-d '{"model":"styletts2","input":"Lens reads this sentence aloud.","voice":"default"}' \
http://127.0.0.1:8880/v1/audio/speech)
echo "HTTP $code, $(stat -c %s speech.wav) bytes"
test "$code" = 200
head -c 4 speech.wav | grep -q RIFF
test "$(stat -c %s speech.wav)" -gt 20000
code=$(curl -sS -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' \
-d '{"input":"Hi","voice":"nobody"}' http://127.0.0.1:8880/v1/audio/speech)
test "$code" = 404

- name: Logs
if: always()
run: docker logs st 2>&1 | tail -50
5 changes: 3 additions & 2 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ Todo:

- [x] Gemini TTS and macOS `say` as text-to-speech providers; `GET /voice/voices` lists each provider's voices;
`POST /voice/speak` takes a voice to try
- [ ] StyleTTS2 as an opt-in local service (off by default; heavy)
- [x] StyleTTS2 as an opt-in local service (`styletts2/`, Compose profile `styletts2`; voices are clips in a volume)
- [x] Settings → AI assistant → Voice: pick a voice from the list and hear it; Gemini in Speech providers

Refine later: pick a second `say` voice for podcast hosts from what the Mac has; Gemini's multi-speaker mode for a
whole podcast in one call; ElevenLabs voice settings (stability, style); cache the voice lists.
whole podcast in one call; ElevenLabs voice settings (stability, style); cache the voice lists; upload StyleTTS2 voice
clips from Settings (or use a speaker's own voiceprint clips); start the StyleTTS2 service from Settings; GPU build.

## 2026-10-10 · Podcasts from picked knowledge

Expand Down
16 changes: 16 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,20 @@ services:
restart: unless-stopped
networks: [lens]

# StyleTTS2 (docs/speech-providers.md#styletts2): spoken answers and podcasts read on this machine, in a voice like a
# short clip you put in its voices volume. Heavy (PyTorch and a 1 GB model, fetched on first use), so opt-in:
# `docker compose --profile styletts2 up -d`, then Settings → AI assistant → Voice → StyleTTS2.
styletts2:
security_opt: ["no-new-privileges:true"]
image: lens-styletts2:${LENS_VERSION:-prod}
build: ./styletts2
profiles: [styletts2]
volumes:
- styletts2-data:/data
- styletts2-voices:/voices
restart: unless-stopped
networks: [lens]

# Fedora 6 (docs/fedora.md), the repository Lens can keep a copy of the archive in. Opt-in, and not needed on small
# machines: `docker compose --profile fedora up`, with LENS_FEDORA_URL=http://fedora:8080/fcrepo/rest (and
# LENS_FEDORA_USER / LENS_FEDORA_PASSWORD) in .env, or set it in Settings → Fedora.
Expand Down Expand Up @@ -172,6 +186,8 @@ services:
volumes:
surreal-data:
matterbridge-data:
styletts2-data:
styletts2-voices:
archive-data:
fedora-data:
opensearch-data:
Expand Down
15 changes: 15 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,19 @@ services:
- matterbridge-data:/matterbridge
networks: [lens]

# StyleTTS2 (docs/speech-providers.md#styletts2): spoken answers and podcasts read on this machine, in a voice like a
# short clip you put in its voices volume. Heavy (PyTorch and a 1 GB model, fetched on first use), so opt-in:
# `docker compose --profile styletts2 up -d`, then Settings → AI assistant → Voice → StyleTTS2.
styletts2:
security_opt: ["no-new-privileges:true"]
image: lens-styletts2:dev
build: ./styletts2
profiles: [styletts2]
volumes:
- styletts2-data:/data
- styletts2-voices:/voices
networks: [lens]

# Fedora 6 (docs/fedora.md), the repository Lens can keep a copy of the archive in. Opt-in, and not needed on small
# machines: `docker compose --profile fedora up`, with LENS_FEDORA_URL=http://fedora:8080/fcrepo/rest (and
# LENS_FEDORA_USER / LENS_FEDORA_PASSWORD) in .env, or set it in Settings → Fedora.
Expand All @@ -149,6 +162,8 @@ services:
volumes:
surreal-data:
matterbridge-data:
styletts2-data:
styletts2-voices:
archive-data:
fedora-data:

Expand Down
23 changes: 22 additions & 1 deletion docs/speech-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ summaries, chapters and entities as optional imports, per-minute prices in the a
| Deepgram | Nova | yes | detected | from sentiment (English) | no | Aura |
| Google Gemini | no | no | no | no | no | `gemini-2.5-flash-preview-tts`, 30 prebuilt voices |
| macOS `say` | no | no | no | no | no | the Mac's own voices; no key, nothing leaves the Mac |
| StyleTTS2 | no | no | no | no | no | on this machine, in a voice like a clip you give it ([below](#styletts2)) |

Sentiment becomes emotion as positive → Happy, negative → Sad, neutral → Neutral (turn it off with
`speech.sentiment`). SenseVoice's own emotion tags stay richer; a provider only reports sentiment.
Expand Down Expand Up @@ -63,7 +64,27 @@ every Mac.
| `speech.timeout` | 1800 seconds |
| `voice.stt` | `same` |
| `voice.tts_provider` | `openai` |
| `voice.tts_voice` | the provider's default (alloy, a premade ElevenLabs voice, aura-2-thalia-en, Kore, the Mac's) |
| `voice.tts_voice` | the provider's default (alloy, a premade ElevenLabs voice, aura-2-thalia-en, Kore, the Mac's, `default`) |
| `voice.styletts2_url` | `http://styletts2:8880` |

## StyleTTS2

[StyleTTS2](https://github.com/yl4579/StyleTTS2) reads aloud on your own machine: nothing leaves it, and nothing is
billed. It is heavy (PyTorch, a 2 GB image and a 1 GB model fetched from Hugging Face on first use; a sentence takes
a few seconds on a recent CPU), so it is off by default and runs as its own service, `styletts2/`:

```sh
docker compose --profile styletts2 up -d
```

Then pick **StyleTTS2** in Settings → AI assistant → Voice. `voice.styletts2_url` is where Lens reaches it
(`http://styletts2:8880`, the Compose service). It speaks the OpenAI audio API (`POST /v1/audio/speech`,
`GET /v1/audio/voices`).

Its voices are clips: put a few seconds of someone speaking (`.wav`, `.mp3` or `.flac`, named after the voice, e.g.
`grandpa.wav`) in the `styletts2-voices` volume and it reads in a voice like theirs. `default` is a public-domain
LibriVox reader. The model's authors ask that listeners are told the speech is synthesized, unless you have
permission to use the voice.

## What is sent

Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/domain/podcasts.py
Original file line number Diff line number Diff line change
Expand Up @@ -718,8 +718,8 @@ def timed(lines, wpm=150, pause_ms=400):

def voices(cfg, req=None):
"""{a, b}: the hosts' voices for the text-to-speech provider: what the episode asked for, else podcasts.voice_a /
voice_b, else voice.tts_voice and a second voice of the provider's (say has no voice every Mac is sure to have, so
both hosts get the same one unless podcasts.voice_b names another)."""
voice_b, else voice.tts_voice and a second voice of the provider's (say and StyleTTS2 have no second voice they're
sure to have, so both hosts get the same one unless podcasts.voice_b names another)."""
from . import speech

p = cfg.get("podcasts") or {}
Expand Down
6 changes: 5 additions & 1 deletion fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
("decisions", "engine"): {"auto", "jev", "laya", "llm", "off"},
("voice", "input"): {"auto", "server", "browser"},
("voice", "stt"): {"same", "sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"},
("voice", "tts_provider"): {"openai", "elevenlabs", "deepgram", "gemini", "say"},
("voice", "tts_provider"): {"openai", "elevenlabs", "deepgram", "gemini", "say", "styletts2"},
("mail", "security"): {"starttls", "ssl", "none"},
("bridge", "answer"): {"mention", "all"},
("bridge", "approve"): {"off", "low_risk"},
Expand Down Expand Up @@ -426,6 +426,10 @@ def _check(section, key, value, default):
if not (isinstance(value, str) and VIEWER_URL.match(value.strip())):
raise ValueError("voice.tts_base_url is the http(s) address of an OpenAI-compatible server with /audio/speech")
return value.strip().rstrip("/")
if (section, key) == ("voice", "styletts2_url"):
if not (isinstance(value, str) and VIEWER_URL.match(value.strip())):
raise ValueError("voice.styletts2_url is the http(s) address of the styletts2 service")
return value.strip().rstrip("/")
if section == "voice" and key in ("tts_model", "tts_voice", "tts_api_key"):
if value in (None, ""):
return None
Expand Down
44 changes: 40 additions & 4 deletions fastapi_backend/app/domain/speech.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@
* gemini: Google Gemini text to speech (its prebuilt voices).

Text to speech can also come from macOS `say` when Lens runs on a Mac itself (not in Docker): nothing to set up, the
Mac's own voices. voices() lists what each one can read in, for Settings and podcast hosts.
Mac's own voices; or from StyleTTS2, the optional `styletts2` service (styletts2/), which reads in a voice like a
short clip of someone speaking. voices() lists what each one can read in, for Settings and podcast hosts.

A provider's transcript is the same list of segments a local engine returns, plus `provider_speaker`: the speaker
separation step uses those labels (diarize.engine provider, or auto), and voice IDs still come from local voiceprints.
Expand Down Expand Up @@ -53,8 +54,9 @@
"deepgram": "Deepgram",
"gemini": "Google Gemini",
"say": "macOS say",
"styletts2": "StyleTTS2",
}
TTS_PROVIDERS = ("openai", "elevenlabs", "deepgram", "gemini", "say")
TTS_PROVIDERS = ("openai", "elevenlabs", "deepgram", "gemini", "say", "styletts2")
CHUNK_SECONDS = 600 # OpenAI-compatible servers take up to 25 MB a request; ten minutes of Opus is a few MB
POLL_SECONDS = 3
MAX_WORDS = 40
Expand All @@ -68,7 +70,15 @@
DEEPGRAM_TTS_MODEL = "aura-2-thalia-en"
GEMINI_VOICE = "Kore"
# what each provider reads in when nothing is picked (say: the Mac's own voice)
DEFAULT_VOICE = {"openai": "alloy", "elevenlabs": ELEVENLABS_VOICE, "deepgram": DEEPGRAM_TTS_MODEL, "gemini": GEMINI_VOICE, "say": None}
DEFAULT_VOICE = {
"openai": "alloy",
"elevenlabs": ELEVENLABS_VOICE,
"deepgram": DEEPGRAM_TTS_MODEL,
"gemini": GEMINI_VOICE,
"say": None,
"styletts2": "default",
}
STYLETTS2_SECONDS = 600 # it reads on the CPU, a long answer slowly; the first call also loads the model
OPENAI_VOICES = ("alloy", "ash", "ballad", "coral", "echo", "fable", "nova", "onyx", "sage", "shimmer", "verse")
# Gemini's prebuilt voices and how Google describes each
GEMINI_VOICES = {
Expand Down Expand Up @@ -491,6 +501,8 @@ def tts_ready(cfg):
return bool(v.get("tts_model") and (v.get("tts_base_url") or (cfg.get("llm") or {}).get("base_url")))
if p == "say":
return say_available()
if p == "styletts2":
return bool(v.get("styletts2_url")) # its own service; when it isn't running the browser reads answers
return p in TTS_PROVIDERS and bool(_key(cfg, p))


Expand Down Expand Up @@ -583,7 +595,25 @@ def _speak_openai(cfg, text, voice):
return m, lambda: _request(base + "/audio/speech", data=json.dumps(body).encode(), headers=headers, timeout=60)


SPEAK = {"openai": _speak_openai, "elevenlabs": _speak_elevenlabs, "deepgram": _speak_deepgram, "gemini": _speak_gemini, "say": _speak_say}
def _styletts2_url(cfg):
return str((cfg.get("voice") or {}).get("styletts2_url") or "").rstrip("/")


def _speak_styletts2(cfg, text, voice):
body = {"model": "styletts2", "input": text, "voice": voice, "response_format": "wav"}
url = _styletts2_url(cfg) + "/v1/audio/speech"
headers = {"Content-Type": "application/json"}
return "styletts2", lambda: _request(url, data=json.dumps(body).encode(), headers=headers, timeout=STYLETTS2_SECONDS)


SPEAK = {
"styletts2": _speak_styletts2,
"openai": _speak_openai,
"elevenlabs": _speak_elevenlabs,
"deepgram": _speak_deepgram,
"gemini": _speak_gemini,
"say": _speak_say,
}


def speak(cfg, text, voice=None):
Expand Down Expand Up @@ -666,6 +696,12 @@ def voices(cfg, provider=None):
return _say_voices(), None
except (OSError, subprocess.TimeoutExpired) as e:
return [], f"say didn't list its voices: {e}"
if p == "styletts2":
try:
names = _json(_styletts2_url(cfg) + "/v1/audio/voices", timeout=15).get("voices") or []
except (ProviderError, AttributeError) as e:
return [], f"StyleTTS2 isn't running ({e}): docker compose --profile styletts2 up -d"
return [_voice(n, detail="its own reader" if n == "default" else "from your clip") for n in names if isinstance(n, str)], None
if not _key(cfg, p):
return [], f"{LABELS[p]} has no API key yet: add it in Settings → Speech providers"
try:
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,8 @@ def labels():
# browser's; spoken answers come from tts_model (an OpenAI-compatible /audio/speech), else the browser reads them
# stt: the engine that hears voice chat ("same": the transcription engine); tts_provider: who reads answers aloud
# (openai: the OpenAI-compatible speech server below; elevenlabs, deepgram, gemini: the keys in speech; say: macOS's
# own voices, when Lens runs on a Mac itself); tts_voice: the voice they read in (null: the provider's default)
# own voices, when Lens runs on a Mac itself; styletts2: the optional service at styletts2_url); tts_voice: the
# voice they read in (null: the provider's default)
"voice": {
"input": "auto",
"stt": "same",
Expand All @@ -301,6 +302,7 @@ def labels():
"tts_model": None,
"tts_voice": None,
"tts_api_key": None,
"styletts2_url": "http://styletts2:8880",
},
# podcast episodes made from what someone picked (domain/podcasts.py): where they go, the hosts' names, how much
# source text the model reads (characters), the longest episode (minutes), the speaking pace, and a model for
Expand Down
37 changes: 37 additions & 0 deletions fastapi_backend/tests/domain/test_speech.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ def do_GET(self):
)
if self.path.startswith("/gm/v1beta/models"):
return self._send({"models": [{"name": "models/gemini-2.5-flash-preview-tts"}, {"name": "models/gemini-2.5-pro"}]})
if self.path == "/st/v1/audio/voices":
return self._send({"voices": ["default", "grandpa"]})
if self.path == "/oai/v1/audio/voices":
return self._send({"voices": ["af_heart", "am_adam"]})
if self.path.endswith("/models") or self.path.endswith("/v1/models"):
Expand Down Expand Up @@ -138,6 +140,8 @@ def do_POST(self):
}
}
)
if p == "/st/v1/audio/speech":
return self._send(speech.wav(b"\x00\x00" * 24, 24000), "audio/wav")
if p.startswith("/gm/v1beta/models/"):
pcm = base64.b64encode(b"\x00\x01" * 2400).decode()
return self._send(
Expand Down Expand Up @@ -349,6 +353,39 @@ def test_each_provider_lists_its_voices(db, cfg, fake):
speech.voices(eff, "nope")


def test_styletts2_reads_aloud_from_its_own_service(db, cfg, fake):
settings.save(db, cfg, "voice", {"tts_provider": "styletts2"}, "t")
eff = settings.effective(db, cfg)
assert eff["voice"]["styletts2_url"] == "http://styletts2:8880" and voice.can_speak(eff) # the compose service
settings.save(db, cfg, "voice", {"styletts2_url": fake + "/st/"}, "t")
eff = settings.effective(db, cfg)
data, ctype = voice.speak(eff, "Hello", voice="grandpa")
assert ctype == "audio/wav" and data[:4] == b"RIFF"
method, path, headers, body = Fake.seen[-1]
assert path == "/st/v1/audio/speech" and json.loads(body) == {
"model": "styletts2",
"input": "Hello",
"voice": "grandpa",
"response_format": "wav",
}
voice.speak(eff, "Hi")
assert json.loads(Fake.seen[-1][3])["voice"] == "default"
assert speech.voices(eff) == (
[
{"id": "default", "name": "default", "detail": "its own reader"},
{"id": "grandpa", "name": "grandpa", "detail": "from your clip"},
],
None,
)
settings.save(db, cfg, "voice", {"styletts2_url": "http://127.0.0.1:9"}, "t")
eff = settings.effective(db, cfg)
found, err = speech.voices(eff)
assert found == [] and err.startswith("StyleTTS2 isn't running") and "--profile styletts2" in err
assert voice.speak(eff, "Hello") is None # not running: the browser reads it
with pytest.raises(ValueError):
settings.save(db, cfg, "voice", {"styletts2_url": "styletts2:8880"}, "t")


SAY_VOICES = """Albert en_US # Hello! My name is Albert.
Eddy (English (UK)) en_GB # Hello! My name is Eddy.
Bad News en_US # Hello! My name is Bad News.
Expand Down
10 changes: 9 additions & 1 deletion nextjs-frontend/components/settings/bodies.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -1302,8 +1302,16 @@ function AiBody({ ctx }: { ctx: BodyCtx }) {
state={ctx.state("voice.tts_voice")}
/>
{tts === "openai" && <F ctx={ctx} id="voice.tts_base_url" />}
{tts === "styletts2" && <F ctx={ctx} id="voice.styletts2_url" />}
</div>
{tts === "say" ? (
{tts === "styletts2" ? (
<p className="text-[12.5px] text-fg-secondary">
Reads on this machine, in a voice like a short clip of someone speaking: put a few seconds of a .wav in the
service’s voices folder and pick it here. Heavy, so it runs only when started with{" "}
<code className="font-mono text-[12px]">docker compose --profile styletts2 up -d</code>. Tell listeners the
speech is synthesized unless you may use the voice.
</p>
) : tts === "say" ? (
<p className="text-[12.5px] text-fg-secondary">
Uses the voices installed on the Mac Lens runs on (System Settings → Accessibility → Spoken Content adds
more). Not available when Lens runs in Docker.
Expand Down
Loading
Loading