diff --git a/README.md b/README.md index 30b91b73..f2d512c4 100644 --- a/README.md +++ b/README.md @@ -182,7 +182,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p (noise, volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before analysis (stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server you run), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each - off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place. Translate a transcript into another language when you ask, with the estimate shown first: it's kept + off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place. The language + transcription hears is kept on the asset, and you can correct it with a region (en-GB or en-US, pt-BR or pt-PT), which + transcribing again keeps. Translate a transcript into another language when you ask, with the estimate shown first: it's kept beside the asset as a translation whose lines keep their times, found by search like an uploaded one. A date said ("let's meet next Tuesday at 3") becomes a calendar event in one click, prefilled from the line and added only once you approve it ([propose an event](docs/processing.md#propose-an-event)). @@ -195,7 +197,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p [cleaning up audio](docs/processing.md#cleaning-up-audio), [your own names](docs/processing.md#your-own-names-in-transcription), [cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates), - [unsure words](docs/processing.md#unsure-words), + [unsure words](docs/processing.md#unsure-words), [an asset's language](docs/api.md#an-assets-language), [translations](docs/api.md#translating-a-transcript), [captions](docs/video.md#captions); [checks](docs/requirements.md#understand-it)) - **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its diff --git a/changelog.d/asset-language.md b/changelog.d/asset-language.md new file mode 100644 index 00000000..c81866ca --- /dev/null +++ b/changelog.d/asset-language.md @@ -0,0 +1,7 @@ +- **Correct an asset's language, with its region.** The language transcription hears stays on the asset, and editors + can set or correct it as a BCP 47 tag with a region when it matters (`en-GB` or `en-US`, `pt-BR` or `pt-PT`) in the + asset's Details tab, with `PATCH /api/v1/resources/ {"language": "en-gb"}` or the MCP tool `set_language`. The + tag is checked and written the usual way (`en-GB`, `zh-Hant-TW`), the IIIF manifest and Dublin Core record give it, + and transcribing the asset again keeps a person's language instead of the detected one; `null` forgets it. A + pipeline step's `languages` condition of `en` now takes `en-GB` and `en-US` too. Languages already stored stay as + they are ([An asset's language](docs/api.md#an-assets-language)). diff --git a/docs/api.md b/docs/api.md index c154bc7a..f0f3e853 100644 --- a/docs/api.md +++ b/docs/api.md @@ -728,7 +728,7 @@ many changed. `GET /resources/tags` (`ns` for one namespace) lists the tags in u most used first. `PATCH /resources/{rid}` with `{"title": …}` renames an asset (editors; whitespace is collapsed, at most 200 -characters). It is audited as `recording.rename`; the asset's report page follows the new title and is rebuilt, +characters), and with `{"language": …}` sets its language ([An asset's language](#an-assets-language)). It is audited as `recording.rename`; the asset's report page follows the new title and is rebuilt, and a published asset shows up as an Update in the IIIF change feed. `DELETE /resources/{rid}` deletes an asset (owners). Everything Lens made from it goes: its text and analysis @@ -788,6 +788,20 @@ all of the asset (owners). `PUT …/{gid}` opens the asset to a group that opens that opens `everything` already), `DELETE …/{gid}` closes it again (404 when it wasn't open). Both answer with the list and are audited as `recording.ip_group.open` and `recording.ip_group.close`. +### An asset's language + +Transcription keeps the language it hears on the asset (`language`): SenseVoice's, the one Whisper detects when +Settings → Transcription leaves the language on auto, or a speech provider's, as a short code such as `en`. A person +can set or correct it as a BCP 47 tag, with a region when it matters (`en-GB` or `en-US`, `pt-BR` or `pt-PT`): +`PATCH /resources/{rid}` with `{"language": "en-gb"}` (editors) checks the tag and writes it the usual way (`en-GB`, +`zh-Hant-TW`, `es-419`); one that isn't a tag is a 400. `GET /resources/{rid}` gives `language` and `language_set`, +true when a person set it: transcribing it again, importing a transcript over it or reading it again keeps their +language instead of the detected one. `{"language": null}` forgets it, so the next transcription says again. Changes +are audited as `recording.language`, and a published asset shows up as an Update in the IIIF change feed (its +manifest and Dublin Core record give the tag). The MCP tool `set_language` does the same, and the asset's Details tab +in the web app shows the language by name with its tag, which editors change there. The Language filter counts `en-GB` +apart from `en`; a pipeline step's `languages` condition of `en` takes `en-GB` and `en-US` too. + ### Documents and images A document (a PDF; a Word, PowerPoint or spreadsheet file, OpenDocument or RTF; text or Markdown; a saved web page; an diff --git a/docs/database.md b/docs/database.md index 6ae1795b..f081714b 100644 --- a/docs/database.md +++ b/docs/database.md @@ -94,7 +94,9 @@ as missing on older rows: a chat message's record (`meta`, none. [Deep research](assistant.md#deep-research) is another: its tables (`research`, and `research_run` for the claim of the process running one) are new, a message's `kind` is missing on older ones, and notes accept `chat` links without rewriting any. [Import runs](api.md#import-runs) are a third: the `import_run` table and the index on a -recording's `import_run` are new, and assets imported before have no run. Anything else, such as rewriting stored values, moving data between +recording's `import_run` are new, and assets imported before have no run. [An asset's +language](api.md#an-assets-language) is a fourth: `language_set` is new, assets from before have none (their language +reads as detected), and the codes already stored (`en`, `de`, `zh`) stay valid as they are. Anything else, such as rewriting stored values, moving data between tables, or changing or removing an index that already exists, is a named step in `app/domain/migrations.py`: * Every process that opens the database (the API, workers, `lens watch`, `lens` commands) runs the steps the database diff --git a/docs/mcp.md b/docs/mcp.md index 8faec5dd..b8703177 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -127,6 +127,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `nsfw_settings` | how [NSFW classification](nsfw.md) works in a namespace: on or off, the threshold, how flagged resources show, and how many are classified, flagged and not classified yet | | `set_nsfw` | turn NSFW classification on or off, set its threshold and display, for a namespace (owners) or the archive (admins), and optionally classify what's there. Needs a write-scope token | | `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access | +| `set_language` | set or correct the [language an asset is in](api.md#an-assets-language) as a BCP 47 tag with a region if it matters (`en-GB`, `pt-BR`), written the usual way and kept over the one transcription detects; left out, it's forgotten. Needs a write-scope token and editor access | | `enhance` | [enhance an image](enhance.md) whatever the settings, keeping the original, and read it again from the enhanced copy. Needs a write-scope token and the process permission | | `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there | | `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token | @@ -212,7 +213,7 @@ app (merges and deletes can be undone), `write_note`, which adds a note, `create queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [published feed](feeds.md), `save_circle`, `share_with_circle` and `undo_share`, which change your [circles](circles.md) and what you share with them, `write_spec`, which queues a spec, `translate_transcript`, which queues a translation, `draft_reply` and `edit_reply`, which draft and change [email replies](email-replies.md) (nothing is sent), `send_reply`, which asks to send one (it waits in Approvals), `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and -editors), `enhance`, which enhances an image and reads it again (the process permission), +editors), `set_language`, which sets an asset's language (editors), `enhance`, which enhances an image and reads it again (the process permission), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `install_plugin`, which installs a plugin from a marketplace as your own extension, `add_mcp_server`, which adds an MCP server (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), `set_mail_routing`, which changes where an email account's messages go (admins), `add_mesh_node`, diff --git a/docs/processing.md b/docs/processing.md index 9eacbd56..07fbbff6 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -9,7 +9,8 @@ How assets move through Lens: where they come from, what each step does, and how file). What one namespace's scan brings in is one [import run](api.md#import-runs), undone whole. - `clean_audio` (only in a pipeline that has it) cleans up the sound before it's transcribed: noise, volume, rumble, echo, and silence left out of what's transcribed ([Cleaning up audio](#cleaning-up-audio)). -- `transcribe` uses SenseVoice, faster-whisper or mlx-whisper. When the configured engine isn't installed on a worker (or +- `transcribe` uses SenseVoice, faster-whisper or mlx-whisper, and keeps the language it heard on the asset, unless a + person set one ([An asset's language](api.md#an-assets-language)). When the configured engine isn't installed on a worker (or doesn't import there), it uses the next one that is and says so in the job log; with none at all the step fails, saying how to add one. A file that fails is marked and the batch carries on. For a document or an image it draws the pages and reads their text instead: a PDF's own text, and OCR for scans and images @@ -864,7 +865,8 @@ audio skip themselves for imported text. Steps: - `convert`: makes new assets from one, a video's soundtrack as audio or a document's pages as images ([Converting assets](#converting-assets)). -Any step can carry a condition: `min_minutes`, `max_minutes`, `source` (audio or transcript), `languages`. +Any step can carry a condition: `min_minutes`, `max_minutes`, `source` (audio or transcript), `languages` (the +asset's [language](api.md#an-assets-language), ignoring case; `en` takes `en-GB` and `en-US` too). Templates are versioned (publish, history, diff) and rendered in a sandboxed Jinja environment. It can't reach Python internals, caps output size, and escapes HTML in reports. Templates see `recording`, `speakers`, `segments`, diff --git a/docs/requirements.md b/docs/requirements.md index 91beed0b..f60e85a1 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -201,6 +201,7 @@ request, to another check that proves the same thing; loosening a requirement or | Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) | | One asset's voice split off from the person it was matched to, with Undo, and a voice never matched or suggested again as someone it was said not to be | [test_speakers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_speakers.py) `test_detach_one_assets_voice`; [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_a_voice_never_goes_to_someone_it_was_said_not_to_be` | | How sure transcription was of each word kept (Whisper, Deepgram, AssemblyAI, ElevenLabs), words from before still loading, the unsure ones marked in the transcript with how sure, a threshold in Settings (0 for none), and `unsure` words per line over MCP | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_providers_keep_how_sure_they_were_of_each_word`, `test_whisper_keeps_how_sure_it_was_of_each_word`; [test_split_merge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_split_merge.py) `test_words_keep_how_sure_transcription_was`, `test_words_from_before_how_sure_was_kept_still_load`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_get_transcript_says_which_words_transcription_was_unsure_of`; [recording-split-words.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-split-words.test.ts) `keeps how sure transcription was of each word, and words from before it was kept`, `marks the words below the threshold, and none when it's off`; [recording-unsure-words.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-unsure-words.test.tsx) `are underlined with how sure, and a click still plays the line` | +| The language transcription hears kept on the asset, and a person sets or corrects it as a BCP 47 tag with a region (en-GB, pt-BR), checked and written the usual way, kept over the detected one when it is transcribed again; through the Details tab, the API and MCP | [test_language.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_language.py) `test_tags_are_written_the_usual_way`, `test_what_isnt_a_tag_is_refused`, `test_editors_set_and_correct_an_assets_language`, `test_transcribing_again_keeps_the_language_a_person_set`, `test_a_pipeline_condition_on_en_takes_its_regions`, `test_agents_set_it_over_mcp`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_whisper_keeps_the_language_it_heard`; [recording-language.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-language.test.tsx) `sends a tidy tag, null to forget it, and nothing when it's the same`, `lets an editor correct it with Enter, and refuses what isn't a tag` | | People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`, `test_merging_keeps_who_they_are_not`, `test_upgrade_gives_pairs_lost_to_merges_to_who_they_were_merged_into`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` | | An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` | | Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` | diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 65f2cf32..b606c999 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -920,7 +920,10 @@ def list_speakers(ctx: Context, namespace: str | None, query: str | None, limit: def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]: rec = recording_routes.get_recording(recording_id, ctx.acl, ctx.db, ctx.cfg) raw = ( - ctx.db.one("SELECT recorded_at, duration_ms, source, media, language, pages, analyzed_at FROM $r", r=R("recording", recording_id)) + ctx.db.one( + "SELECT recorded_at, duration_ms, source, media, language, language_set, pages, analyzed_at FROM $r", + r=R("recording", recording_id), + ) or {} ) sections = ctx.db.rows("SELECT idx, t0, t1, title FROM section WHERE recording = $r ORDER BY idx", r=recording_id) @@ -937,6 +940,7 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]: "media": render.kind(raw), "pages": raw.get("pages"), "language": raw.get("language"), + "language_set": bool(raw.get("language_set")) or None, # a person set it; transcription leaves it "status": rec.status, "tags": rec.tags or None, "your_role": rec.role, @@ -2960,6 +2964,32 @@ def mark_nsfw(ctx: Context, recording_id: int, verdict: str) -> dict[str, Any]: return {"recording_id": recording_id, "nsfw": got, "url": ctx.link(recording_id)} +# ---------- an asset's language ---------- +@tool( + "set_language", + "Set an asset's language", + "Set or correct the language an asset is in, as a BCP 47 tag with a region if it matters (en-GB, en-US, pt-BR, " + "pt-PT); it is written the usual way (en-gb becomes en-GB) and kept over the language transcription detects. " + "Left out, it forgets it, so the next transcription says again. get_recording gives its language and whether a person " + "set it (language_set). Editors, with a token that has the write scope.", + Arg("recording_id", "integer", "the asset", required=True), + Arg("language", "string", "a BCP 47 tag such as en, en-GB or pt-BR; left out, it's forgotten", max_length=40), + writes=True, +) +def set_language(ctx: Context, recording_id: int, language: str | None) -> dict[str, Any]: + if not ctx.user.can_write: + raise ToolError("this token is read-only: ask for a token with the write scope to set languages") + ctx.acl.recording(recording_id, "edit") + try: + before, after = library.set_language(ctx.db, ctx.cfg, recording_id, language) + except ValueError as e: + raise ToolError(str(e)) from None + if before != after: + detail = {"from": before, "to": after, "via": "mcp"} + auth.audit(ctx.db, ctx.user.as_audit(), "recording.language", f"recording:{recording_id}", detail) + return {"recording_id": recording_id, "language": after, "language_set": after is not None, "url": ctx.link(recording_id)} + + # ---------- enhancing low quality images ---------- @tool( "enhance", diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py index 6adefcdd..665c37db 100644 --- a/fastapi_backend/app/api/v1/routes/recordings.py +++ b/fastapi_backend/app/api/v1/routes/recordings.py @@ -387,11 +387,17 @@ def _attached_to(acl: Access, db: DB, link: dict[str, Any] | None) -> dict[str, @router.patch("/{rid}") def update_recording(rid: int, body: RecordingUpdate, acl: Acl, user: Writer, db: Db, cfg: Cfg) -> Recording: - """Rename an asset or replace its tags (editors). A renamed asset's report is rebuilt with the new title.""" + """Rename an asset, replace its tags or set its language (editors). A renamed asset's report is rebuilt with the + new title. The language is a BCP 47 tag, written the usual way (en-gb → en-GB), kept over the one transcription + detects; null forgets it. Audited as `recording.rename` and `recording.language`.""" rec = acl.recording(rid, "edit") - if body.title is None and body.tags is None: - raise HTTPException(400, "send a title or tags") + if body.title is None and body.tags is None and "language" not in body.model_fields_set: + raise HTTPException(400, "send a title, tags or a language") with domain_errors(): + if "language" in body.model_fields_set: + before, after = library.set_language(db, cfg, rid, body.language) + if before != after: + auth.audit(db, user.as_audit(), "recording.language", f"recording:{rid}", {"from": before, "to": after}) if body.tags is not None: library.set_tags(db, rid, body.tags) if body.title is not None: diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index b82942d5..b3fb3e6d 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -544,7 +544,10 @@ def _language(segs): def write_transcript(db, rid, nid, segs, patch): - """Replace a recording's transcript and everything derived from it, atomically.""" + """Replace a recording's transcript and everything derived from it, atomically. A language a person set + (library.set_language) stays over the one transcription gives.""" + if "language" in patch and (db.one("SELECT language_set FROM $r", r=store.R("recording", rid)) or {}).get("language_set"): + patch = {k: v for k, v in patch.items() if k != "language"} rows = segment_rows(rid, nid, segs) # overwrite segments in place and drop the extra ones (see store.DOWNSTREAM) db.run( store.DOWNSTREAM + ["FOR $s IN $segs { UPSERT $s.id CONTENT $s; }", "UPDATE $rec MERGE $patch", "UPDATE $rec SET embedded = NONE"], diff --git a/fastapi_backend/app/domain/library.py b/fastapi_backend/app/domain/library.py index 576d50c3..517d72ca 100644 --- a/fastapi_backend/app/domain/library.py +++ b/fastapi_backend/app/domain/library.py @@ -201,11 +201,18 @@ def in_scope(spaces, within=None): return "space IN $spaces", {"spaces": sorted(spaces)} +def _shown_language(v): + try: + return metadata.language_tag(v) + except ValueError: # what an engine says without a tag (nospeech) stays as it is + return str(v).strip().lower() + + def language_counts(db, spaces, within=None): """[{language, recordings}] in these namespaces (and the collections in `within`), the most recordings first; null - is "not known".""" + is "not known". A language tag is written the usual way (en-GB), whatever its case.""" cond, p = in_scope(spaces, within) - n = Counter((str(v or "").strip().lower() or None) for v in db.values(f"SELECT VALUE language FROM recording WHERE {cond}", **p)) + n = Counter(_shown_language(v) for v in db.values(f"SELECT VALUE language FROM recording WHERE {cond}", **p)) return sorted(({"language": k, "recordings": c} for k, c in n.items()), key=lambda x: (-x["recordings"], x["language"] or "~")) @@ -581,6 +588,24 @@ def tag_counts(db, spaces, within=None): return [{"tag": spellings[k].most_common(1)[0][0], "recordings": n} for k, n in sorted(counts.items(), key=lambda x: (-x[1], x[0]))] +def set_language(db, cfg, rid, language): + """A person's language for a recording, as a BCP 47 tag with its region if they like ("en-gb" → "en-GB"). It is + kept over the one transcription detects (write_transcript leaves it); None forgets it, so the next transcription + says again. Returns (old, new).""" + tag = metadata.language_tag(language) + rec = db.one("SELECT language, language_set FROM $r", r=R("recording", rid)) + if not rec: + raise KeyError(rid) + if rec.get("language") == tag and bool(rec.get("language_set")) == (tag is not None): + return tag, tag + if tag: + db.q("UPDATE $r SET language = $l, language_set = true", r=R("recording", rid), l=tag) + else: + db.q("UPDATE $r SET language = NONE, language_set = NONE", r=R("recording", rid)) + metadata.touched(db, cfg, rid) # IIIF harvesters see an Update for a published recording + return rec.get("language"), tag + + def rename(db, cfg, rid, title): """A new title (whitespace collapsed, at most 200 characters). Returns (old, new). diff --git a/fastapi_backend/app/domain/metadata.py b/fastapi_backend/app/domain/metadata.py index d142933d..d8fdaade 100644 --- a/fastapi_backend/app/domain/metadata.py +++ b/fastapi_backend/app/domain/metadata.py @@ -50,6 +50,27 @@ "featured", ) LANG_RX = re.compile(r"^(none|[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*)$") +# A BCP 47 language tag (RFC 5646), lowercased: a language, then a script, a region, variants, extensions and a private +# part, each but the language optional (en, en-GB, pt-BR, zh-Hant-TW, sr-Latn-RS, es-419, de-CH-1996) +TAG_RX = re.compile( + r"^(?P[a-z]{2,3})(?:-(?P