Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Long recordings go to a speech provider in pieces that fit it.** A recording longer or larger than a provider takes
in one request is cut at pauses into pieces within its limits, and their transcripts are joined back into one, where
only OpenAI-compatible servers had ten-minute parts before. Each provider starts with its published limits (10 hours
for ElevenLabs and AssemblyAI, 2 GB for Deepgram, 25 MB for OpenAI), editable in Settings → Speech providers
([docs](docs/speech-providers.md#long-recordings)).
- **Running the install line again keeps `.env` readable only by you.** Each update rewrote `.env` with the shell's
default permissions, so after an update its secrets could be read by other users of the machine. It is now written
for its owner alone, and an install an earlier update left open is closed again the next time the line runs.
Expand Down
14 changes: 14 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,20 @@

Plans and progress for work in flight. Newest first.

## 2026-10-14 · Long recordings in pieces a speech provider takes

Adeel: large audio files are segmented automatically to each transcription provider's limits. `speech.Engine` cuts a
recording longer than `speech.{provider}_max_minutes` at the quietest tenth of a second in the half minute before the
limit, halves a piece whose encoded file is over `speech.{provider}_max_mb` (again at a pause), sends each piece and
joins the segments on the recording's timeline. Defaults follow each provider's published limits
(docs/speech-providers.md, Long recordings); they are in Settings → Speech providers. This replaces the ten-minute
parts only OpenAI-compatible servers had.

Refine later: link a new voice heard in two pieces into one speaker by voiceprint (today it shows as two until merged);
decode and send a very long recording a piece at a time instead of holding all its samples in memory; send pieces in
parallel for providers that allow it; overlap pieces a little and drop the words heard twice, for a cut with no pause
near it.

## 2026-10-14 · MCP servers that run as a program

Admins add a stdio MCP server (`command`, `args`, `env`, as in Claude's `.mcp.json`) as an `mcp` extension, and a
Expand Down
33 changes: 31 additions & 2 deletions docs/speech-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,35 @@ Then pick where each task goes:
- **AI assistant → Voice**: *Engine that hears it* (default: the transcription engine) and *Spoken answers by* (an
OpenAI-compatible speech server, ElevenLabs, Deepgram Aura, Google Gemini or macOS `say`) and *Voice*.

## Long recordings

Each provider takes so much audio in one request. A recording longer or larger than that is cut into pieces, each
piece is transcribed on its own, and the transcripts are joined back into one, timed from the start of the recording.
Nothing needs setting up: every provider starts with a limit from its published one.

| Provider | Longest piece | Largest file | Where the default comes from |
| --- | --- | --- | --- |
| OpenAI-compatible | 10 minutes | 25 MB | [OpenAI](https://platform.openai.com/docs/guides/speech-to-text): 25 MB a file; GPT-4o transcription takes up to 25 minutes, and shorter pieces suit servers with smaller limits |
| ElevenLabs | 600 minutes | 3000 MB | [ElevenLabs](https://elevenlabs.io/docs/capabilities/speech-to-text): up to 10 hours and 3 GB |
| AssemblyAI | 600 minutes | 2200 MB | [AssemblyAI](https://www.assemblyai.com/docs/faq/are-there-any-limits-on-file-size-or-file-duration-for-files-submitted-to-the-api): up to 10 hours; 2.2 GB through its upload endpoint |
| Deepgram | 120 minutes | 2000 MB | [Deepgram](https://developers.deepgram.com/docs/pre-recorded-audio): 2 GB; it stops a request after 10 minutes of work, so two hours at a time |

Change them in **Settings → Speech providers** (*Longest piece sent at once*, *Largest file sent at once*), or as
`speech.{provider}_max_minutes` and `speech.{provider}_max_mb` through the settings API or the assistant's
`change_settings` tool: lower them for a proxy or a compatible server that takes less (Groq's free tier, a small
self-hosted server), raise them for one that takes more.

How a recording is cut:

- A piece ends at the quietest tenth of a second in the half minute before the limit, so a cut falls in a pause rather
in the middle of a word. Pieces meet exactly: nothing is sent twice or left out.
- Each piece is encoded to send; one whose file is still larger than the limit is halved, again at a pause, until each
part fits.
- The provider numbers speakers afresh in every request, so a speaker label belongs to its piece (`c0:S1`, `c1:S1`).
Voice IDs still come from local voiceprints and match voices already known; a new voice heard in two pieces shows as
two speakers until you merge them.
- The activity ledger keeps one row for the recording, with its length in seconds and the number of pieces.

## Voices

`GET /voice/voices?provider=…` lists the voices a provider reads in, as `{id, name, detail}`: ElevenLabs lists the
Expand Down Expand Up @@ -89,7 +118,7 @@ permission to use the voice.
## What is sent

Only the audio of the recording (or the clip said into the mic), encoded as Ogg Opus at 32 kbit/s (FLAC when ffmpeg
has no Opus). Recordings longer than ten minutes go to OpenAI-compatible servers in ten-minute parts, since they take
at most 25 MB a request; speaker labels then belong to each part. Text to speech sends the answer's text.
has no Opus), in pieces when it is longer or larger than the provider takes at once ([above](#long-recordings)). Text
to speech sends the answer's text.

Code: `fastapi_backend/app/domain/speech.py`.
9 changes: 9 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -630,9 +630,18 @@ def _speech_setting(key, value):
if not (isinstance(value, int) and not isinstance(value, bool) and 30 <= value <= 86400):
raise ValueError("speech.timeout is a whole number of seconds from 30 to 86400")
return value
for end, (lo, hi), what in (("_max_minutes", SPEECH_MINUTES, "minutes"), ("_max_mb", SPEECH_MB, "MB")):
if key.endswith(end):
if not (isinstance(value, int) and not isinstance(value, bool) and lo <= value <= hi):
raise ValueError(f"speech.{key} is a whole number of {what} from {lo} to {hi}")
return value
raise ValueError(f"unknown setting speech.{key}")


SPEECH_MINUTES = (1, 1440) # the longest piece of a recording sent to a speech provider at once
SPEECH_MB = (1, 10_000) # its largest file


PODCAST_RANGES = {"context_chars": (2000, 400_000), "max_minutes": (1, 120), "words_per_minute": (80, 250)}


Expand Down
176 changes: 119 additions & 57 deletions fastapi_backend/app/domain/speech.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

A provider's transcript is the same list of segments a local engine returns, plus `provider_speaker`: the speaker
separation step uses those labels (diarize.engine provider, or auto), and voice IDs still come from local voiceprints.

Each provider takes so much audio in one request ({p}_max_minutes and {p}_max_mb in the speech settings, defaulting to
its published limits): a longer recording is cut into pieces at pauses, each piece is transcribed on its own, and their
transcripts are joined back on the recording's timeline.
"""

from __future__ import annotations
Expand Down Expand Up @@ -57,7 +61,8 @@
"styletts2": "StyleTTS2",
}
TTS_PROVIDERS = ("openai", "elevenlabs", "deepgram", "gemini", "say", "styletts2")
CHUNK_SECONDS = 600 # OpenAI-compatible servers take up to 25 MB a request; ten minutes of Opus is a few MB
QUIET_SECONDS = 30 # a recording too long for one request is cut at the quietest moment in the half minute before a limit
QUIET_WINDOW = SR // 10 # ... measured in tenths of a second
POLL_SECONDS = 3
MAX_WORDS = 40
GAP_MS = 1500
Expand Down Expand Up @@ -293,55 +298,44 @@ def _label(v):


# ---------- providers ----------
def _openai(cfg, audio, seconds):
def _openai(cfg, clip, seconds):
base, m, key = base_url(cfg, "openai"), model(cfg, "openai"), _key(cfg, "openai")
diarize = "diarize" in m
data, name, ctype = clip
lang = _language(cfg)
out, many = [], len(audio) > CHUNK_SECONDS * SR
for k, start in enumerate(range(0, max(len(audio), 1), CHUNK_SECONDS * SR)):
data, name, ctype = encode(audio[start : start + CHUNK_SECONDS * SR])
if diarize:
fields = [("model", m), ("response_format", "diarized_json"), ("chunking_strategy", "auto")]
elif m.startswith("gpt-"):
fields = [("model", m), ("response_format", "json")]
else:
fields = [("model", m), ("response_format", "verbose_json"), ("timestamp_granularities[]", "segment"),
("timestamp_granularities[]", "word")] # fmt: skip
if lang:
fields.append(("language", lang))
body, ctype_ = multipart(fields, [("file", name, data, ctype)])
headers = {"Content-Type": ctype_, **({"Authorization": f"Bearer {key}"} if key else {})}
j = _json(base + "/audio/transcriptions", data=body, headers=headers, timeout=section(cfg)["timeout"])
off = start * 1000 // SR
got = j.get("language") or lang
segs = j.get("segments") or []
words = j.get("words") or []
if not segs:
end = min(len(audio), start + CHUNK_SECONDS * SR)
segs = [{"start": 0, "end": (end - start) / SR, "text": j.get("text") or ""}]
for s in segs:
t0, t1 = off + int(float(s.get("start") or 0) * 1000), off + int(float(s.get("end") or 0) * 1000)
ws = [
[w["word"].strip(), off + int(w["start"] * 1000), off + int(w["end"] * 1000)]
for w in words
if t0 <= off + w["start"] * 1000 < t1
]
spk = _label(s.get("speaker"))
text = (s.get("text") or "").strip()
if any(ch.isalnum() for ch in text):
out.append(
{
"t0": t0,
"t1": max(t1, t0 + 1),
"text": text,
"raw_text": None,
"emotion": None,
"event": None,
"lang": _short(got),
"words": json.dumps(ws) if ws else None,
"provider_speaker": (f"c{k}:{spk}" if many else spk) if spk else None,
} # fmt: skip
)
if "diarize" in m:
fields = [("model", m), ("response_format", "diarized_json"), ("chunking_strategy", "auto")]
elif m.startswith("gpt-"):
fields = [("model", m), ("response_format", "json")]
else:
fields = [("model", m), ("response_format", "verbose_json"), ("timestamp_granularities[]", "segment"),
("timestamp_granularities[]", "word")] # fmt: skip
if lang:
fields.append(("language", lang))
body, ctype_ = multipart(fields, [("file", name, data, ctype)])
headers = {"Content-Type": ctype_, **({"Authorization": f"Bearer {key}"} if key else {})}
j = _json(base + "/audio/transcriptions", data=body, headers=headers, timeout=section(cfg)["timeout"])
got = j.get("language") or lang
segs = j.get("segments") or [{"start": 0, "end": seconds, "text": j.get("text") or ""}]
words = j.get("words") or []
out = []
for s in segs:
t0, t1 = int(float(s.get("start") or 0) * 1000), int(float(s.get("end") or 0) * 1000)
ws = [[w["word"].strip(), int(w["start"] * 1000), int(w["end"] * 1000)] for w in words if t0 <= w["start"] * 1000 < t1]
text = (s.get("text") or "").strip()
if any(ch.isalnum() for ch in text):
out.append(
{
"t0": t0,
"t1": max(t1, t0 + 1),
"text": text,
"raw_text": None,
"emotion": None,
"event": None,
"lang": _short(got),
"words": json.dumps(ws) if ws else None,
"provider_speaker": _label(s.get("speaker")),
} # fmt: skip
)
return out


Expand All @@ -356,8 +350,8 @@ def _short(lang):
return names.get(lang) or lang.split("-")[0].split("_")[0][:3]


def _elevenlabs(cfg, audio, seconds):
data, name, ctype = encode(audio)
def _elevenlabs(cfg, clip, seconds):
data, name, ctype = clip
fields = [("model_id", model(cfg, "elevenlabs")), ("diarize", "true"), ("tag_audio_events", "true"), ("timestamps_granularity", "word")]
if _language(cfg):
fields.append(("language_code", _language(cfg)))
Expand Down Expand Up @@ -386,9 +380,9 @@ def _elevenlabs(cfg, audio, seconds):
return group_words(words, _short(j.get("language_code")))


def _assemblyai(cfg, audio, seconds):
def _assemblyai(cfg, clip, seconds):
base, key, t = base_url(cfg, "assemblyai"), _key(cfg, "assemblyai"), section(cfg)["timeout"]
data, _, ctype = encode(audio)
data, _, ctype = clip
up = _json(base + "/v2/upload", data=data, headers={"Authorization": key, "Content-Type": "application/octet-stream"}, timeout=t)
ask = {"audio_url": up["upload_url"], "speaker_labels": True, "punctuate": True, "format_text": True}
m = model(cfg, "assemblyai")
Expand Down Expand Up @@ -426,8 +420,8 @@ def _assemblyai(cfg, audio, seconds):
return add_sentiment(segs, spans)


def _deepgram(cfg, audio, seconds):
data, _, ctype = encode(audio)
def _deepgram(cfg, clip, seconds):
data, _, ctype = clip
q = {"model": model(cfg, "deepgram"), "smart_format": "true", "punctuate": "true", "diarize": "true", "utterances": "true"}
if _language(cfg):
q["language"] = _language(cfg)
Expand Down Expand Up @@ -465,9 +459,67 @@ def _deepgram(cfg, audio, seconds):
return add_sentiment(segs, spans)


# each takes (cfg, an encoded clip as (bytes, file name, media type), its seconds) and returns its segments, timed
# from the clip's start
TRANSCRIBE = {"openai": _openai, "elevenlabs": _elevenlabs, "assemblyai": _assemblyai, "deepgram": _deepgram}


# ---------- long recordings ----------
def limits(cfg, p):
"""The most a provider takes in one request: (samples, bytes) from {p}_max_minutes and {p}_max_mb."""
s = section(cfg)
return int(s[f"{p}_max_minutes"]) * 60 * SR, int(s[f"{p}_max_mb"]) * 1_000_000


def quiet_cut(audio, lo, hi):
"""The quietest tenth of a second between samples lo and hi (the latest, of equally quiet ones), so a cut falls in
a pause rather than in a word; hi when there's too little to look at."""
k = (hi - lo) // QUIET_WINDOW
if k < 2:
return hi
energy = np.square(audio[lo : lo + k * QUIET_WINDOW].reshape(k, QUIET_WINDOW)).mean(axis=1)
return lo + (k - 1 - int(np.argmin(energy[::-1]))) * QUIET_WINDOW + QUIET_WINDOW // 2


def pieces(audio, most):
"""Where to cut samples into pieces of at most `most` samples each, each cut at a pause: [(start, end)]."""
out, start = [], 0
while len(audio) - start > most:
hi = start + most
cut = quiet_cut(audio, max(start + most // 2, hi - QUIET_SECONDS * SR), hi)
out.append((start, cut))
start = cut
out.append((start, len(audio)))
return out


def clips(audio, start, end, most_bytes):
"""A piece encoded to send, halved at a pause until each part's file fits the provider's limit: (start, end, file)s."""
clip = encode(audio[start:end])
if len(clip[0]) <= most_bytes:
yield start, end, clip
return
if end - start < 2 * SR:
mb = most_bytes // 1_000_000
raise ProviderError(f"two seconds of audio already make a file larger than {mb} MB: raise the limit in Settings → Speech providers")
mid = (start + end) // 2
mid = quiet_cut(audio, max(start + (end - start) // 4, mid - QUIET_SECONDS * SR), mid)
yield from clips(audio, start, mid, most_bytes)
yield from clips(audio, mid, end, most_bytes)


def _shift(segs, ms, piece):
"""A piece's segments on the recording's timeline; with more than one piece, its speakers are its own (cN:label),
as a provider numbers the speakers of each request afresh."""
for s in segs:
s["t0"], s["t1"] = s["t0"] + ms, s["t1"] + ms
if ms and s.get("words"):
s["words"] = json.dumps([[w, a + ms, b + ms] for w, a, b in json.loads(s["words"])])
if piece is not None and s.get("provider_speaker"):
s["provider_speaker"] = f"c{piece}:{s['provider_speaker']}"
return segs


class Engine:
"""A provider as a transcription engine: transcribe(16 kHz mono samples) -> segments, like SenseVoice's."""

Expand All @@ -478,9 +530,19 @@ def __init__(self, cfg, provider):
self.cfg, self.provider, self.name = cfg, provider, provider

def transcribe(self, audio):
seconds = round(len(audio) / SR, 2)
with _ledger("speech.transcribe", self.cfg, f"{self.provider}:{model(self.cfg, self.provider)}", {"seconds": seconds}):
segs = TRANSCRIBE[self.provider](self.cfg, audio, seconds)
"""The recording's transcript: in one request when it fits the provider's limits, else piece by piece."""
most, most_bytes = limits(self.cfg, self.provider)
detail = {"seconds": round(len(audio) / SR, 2)} # the ledger row reads it when the call ends
parts = []
with _ledger("speech.transcribe", self.cfg, f"{self.provider}:{model(self.cfg, self.provider)}", detail):
for a, b in pieces(audio, most):
for start, end, clip in clips(audio, a, b, most_bytes):
parts.append((start, TRANSCRIBE[self.provider](self.cfg, clip, round((end - start) / SR, 2))))
if len(parts) > 1:
detail["pieces"] = len(parts)
log.info("speech: %s s of audio went to %s in %d pieces", detail["seconds"], LABELS[self.provider], len(parts))
many = len(parts) > 1
segs = [s for k, (start, part) in enumerate(parts) for s in _shift(part, start * 1000 // SR, k if many else None)]
return sorted(segs, key=lambda s: s["t0"])


Expand Down
10 changes: 10 additions & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -339,19 +339,29 @@ def labels():
},
# speech providers (domain/speech.py): transcription, speakers and text to speech by a service instead of this
# server, picked in transcribe.engine, diarize.engine and voice. A base URL can be a proxy or a compatible server.
# {p}_max_minutes and {p}_max_mb are the most audio a provider takes in one request: a longer recording is cut into
# pieces at pauses and their transcripts joined (docs/speech-providers.md, Long recordings, says where each comes from)
"speech": {
"openai_base_url": "https://api.openai.com/v1",
"openai_model": "whisper-1",
"openai_api_key": None,
"openai_max_minutes": 10, # 25 MB a file; GPT-4o transcription takes up to 25 minutes
"openai_max_mb": 25,
"elevenlabs_base_url": "https://api.elevenlabs.io",
"elevenlabs_model": "scribe_v1",
"elevenlabs_api_key": None,
"elevenlabs_max_minutes": 600, # up to 10 hours and 3 GB
"elevenlabs_max_mb": 3000,
"assemblyai_base_url": "https://api.assemblyai.com",
"assemblyai_model": "universal",
"assemblyai_api_key": None,
"assemblyai_max_minutes": 600, # up to 10 hours; 2.2 GB through its upload endpoint
"assemblyai_max_mb": 2200,
"deepgram_base_url": "https://api.deepgram.com",
"deepgram_model": "nova-3",
"deepgram_api_key": None,
"deepgram_max_minutes": 120, # 2 GB; it stops a request after 10 minutes of work, so two hours at a time
"deepgram_max_mb": 2000,
"gemini_base_url": "https://generativelanguage.googleapis.com",
"gemini_model": "gemini-2.5-flash-preview-tts",
"gemini_api_key": None,
Expand Down
Loading
Loading