diff --git a/README.md b/README.md index e789b991d..24635e483 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p beside the asset as a translation whose lines keep their times, found by search like an uploaded one. A date said ("let's meet next Tuesday at 3") becomes a calendar event in one click, prefilled from the line and added only once you approve it ([propose an event](docs/processing.md#propose-an-event)). + Videos play their transcript, and timed captions and translations added beside them, as captions the browser + itself shows (its captions menu, full screen, picture in picture), in the workspace and on public pages. ([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages), [similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md), [topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md), @@ -194,7 +196,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p [your own names](docs/processing.md#your-own-names-in-transcription), [cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates), [unsure words](docs/processing.md#unsure-words), - [translations](docs/api.md#translating-a-transcript); [checks](docs/requirements.md#understand-it)) + [translations](docs/api.md#translating-a-transcript), [captions](docs/video.md#captions); + [checks](docs/requirements.md#understand-it)) - **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its source, read aloud in two voices, by Kokoro on this machine or a speech provider. Turn a voice session or a meeting into a technical spec, in a note linked to it, that keeps every decision, every no and every constraint in the speaker's own words, linked to when they said them, diff --git a/changelog.d/captions-track.md b/changelog.d/captions-track.md new file mode 100644 index 000000000..193357a2d --- /dev/null +++ b/changelog.d/captions-track.md @@ -0,0 +1,8 @@ +- **Captions the browser knows about.** A video now plays its transcript as a WebVTT ``, in the workspace and on + its public page, so the browser's captions menu, full screen, picture in picture and assistive technology get it, + with timed supplementary captions and translations as more tracks to pick. The links are signed, so they work for + visitors and share links too. In the workspace the Captions toggle still draws the line and its speaker over the + video, and the browser draws the track only where that can't be seen (full screen, picture in picture), so captions + never show twice; on the public page the browser draws them once the visitor turns them on with the new + Captions button. `GET /api/v1/recordings//player` and `GET /api/v1/public/recordings/` list the tracks as + `captions` ([Captions](docs/video.md#captions)). diff --git a/docs/access.md b/docs/access.md index 0f852ebf5..a92b8e369 100644 --- a/docs/access.md +++ b/docs/access.md @@ -51,7 +51,8 @@ and translations are open when its text is, indexes when its index is, and thumb attachments (release forms, notes, anything else) always need permission. Decided with the project owner. People with permission download all of them; the public page lists the files a visitor may download and says how many more need permission, which signed-in visitors may ask for. IIIF lists every file, the closed ones behind the Authorization -Flow. +Flow. A video's player offers the ones it may read that say when their lines are as caption tracks, beside the +transcript's ([Captions](video.md#captions)); with downloads off they still play, as the transcript does. ## Who is asking diff --git a/docs/requirements.md b/docs/requirements.md index 617b3e178..db158a06b 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -234,6 +234,7 @@ request, to another check that proves the same thing; loosening a requirement or | Dates said in an asset read as days ("next Tuesday", "last week"), off by default, with dates that read two ways left out | [test_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_dates.py) `test_dates_are_read_as_days`, `test_what_could_be_two_dates_is_left_out`, `test_off_by_default_reads_none` | | A transcript translated into another language only when someone asks, the estimate first: in batches of lines that keep their times and speaker, the transcript marked as data, every call in the activity ledger and within budgets, kept as a translation file like an uploaded one; through the web app's Files tab, the API, MCP and the assistant (asking first) | [test_translations.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_translations.py) `test_translating_writes_a_translation_file`, `test_the_transcript_reaches_the_model_as_data`, `test_lines_go_in_batches_and_keep_their_place`, `test_a_spent_budget_refuses`, `test_the_estimate_comes_first`, `test_what_stops_a_translation`, `test_agents_translate`, `test_the_assistant_asks_first`; [recording-translate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-translate.test.tsx) `is the first the browser reads that the transcript isn't in`, `translates into the browser's language with Enter, after showing the estimate` | | A date said in an asset is proposed as a calendar event in one click, prefilled from its line (the day, a time said, the line as its title, who it names), and added only once the person approves it, signed in; declined, nothing is added; offered only to admins with a calendar connection that may add events, and agents propose over MCP but can't approve | [test_event_proposals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_event_proposals.py) `test_a_date_said_is_proposed_and_added_only_once_approved`, `test_a_declined_proposal_adds_nothing`, `test_offered_only_to_admins_with_a_calendar_that_adds_events`, `test_agents_propose_over_mcp_but_cannot_approve`, `test_the_time_of_day_is_read_from_the_line`; [said-dates.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/said-dates.test.tsx) `proposes an event from a date in one click, and says it waits in Approvals`, `offers no Propose event without a calendar that adds events, and shows nothing without dates` | +| Videos play their transcript and timed supplementary captions and translations as WebVTT tracks with signed links, in the workspace (drawn by the browser only where the overlay can't be seen) and on the public page (off until a button turns them on), with only what the viewer may read | [test_caption_tracks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_caption_tracks.py) `test_the_player_offers_the_transcript_and_timed_files_as_tracks`, `test_the_public_page_offers_what_the_visitor_may_read_as_tracks`; jest [caption-tracks.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/caption-tracks.test.tsx) `stay undrawn while the page's own overlay shows the captions, and are drawn full screen`, `plays its text as captions the browser draws once the visitor turns them on` | ## Learn from it diff --git a/docs/video.md b/docs/video.md index c60e92519..ef1cbefb4 100644 --- a/docs/video.md +++ b/docs/video.md @@ -82,6 +82,35 @@ Video recordings open in their own layout. - **Shots:** a grid of shots. - Everything seeks the one video, and the layout works down to phone widths. +## Captions + +A video plays its text as WebVTT ``s, so the browser's own captions menu, full screen and picture in picture +show them and assistive technology can read them, in the workspace and on its public page alike: + +- the transcript comes first, as `kind="captions"` in the recording's language when it's known, and is the default + while captions are on; +- then each supplementary transcript, captions or translation that says when its lines are + ([files](api.md#files)), in the order they were added; a translation is `kind="subtitles"`. Pick them from the + browser's captions menu. + +The tracks are IIIF's WebVTT (`/iiif//transcript.vtt`, `/iiif//files/.vtt`), with links signed for as long +as media links last, because a `` can't send the Authorization header. `GET /api/v1/recordings//player` +lists them as `captions`, and so does `GET /api/v1/public/recordings/`, with only what that visitor may read and +only when they may play the media. A share link (`?s=`) gets the transcript's track; supplementary files stay with +the namespace's members, as they do in the Files tab. + +Captions show once, never twice: + +- **Workspace (the video recording page):** the Captions toggle (`C`) draws the line and its speaker over the video, as + before. Its track stays loaded but undrawn while that overlay shows, and the browser draws it when the video is full + screen or picture in picture, where the overlay can't be seen. Turning Captions off turns the track off too. +- **Public page:** there is no overlay, so the browser draws the captions. They're off when the page opens, as before, + and the Captions button under the video turns them on. + +Refine later: the embeddable player (`/embed/`, `/s/`) plays sound beside its transcript, without a video, +so it has no tracks; Lens's own controls have no menu to pick a translation's track (the browser's menu does, where it +shows its own controls); and a share link's player could offer the supplementary tracks its link opens. + `The prototype's end-to-end browser test (`tests/e2e_video.py`, kept for reference in the history)` runs it end to end in Chromium, against a real server with face recognition on. It checks: - the lanes, and seeking from shots, lanes, the transcript and the grid diff --git a/fastapi_backend/app/api/media.py b/fastapi_backend/app/api/media.py index 2b8197c5c..8affa9c85 100644 --- a/fastapi_backend/app/api/media.py +++ b/fastapi_backend/app/api/media.py @@ -19,7 +19,9 @@ from collections.abc import Container from typing import Any, cast +from app.config import settings from app.core.security import sign_path +from app.domain import iiif from app.domain.store import API MEDIA_RX = re.compile(rf"^{re.escape(API)}/recordings/\d+/(audio|media|pdf|enhanced|wordcloud\.svg|frames/[\w.-]+|files/\d+/download)$") @@ -71,3 +73,18 @@ def sign_page_links(page: str, recordings: Container[int], full: bool = False) - """Sign the media links (audio, video, frames, word clouds) of these recordings in a page, including inside its embedded JSON. Links to any other recording stay unsigned: the page's text may contain them.""" return MEDIA_IN_HTML.sub(lambda m: sign_path(m.group(0), full=full) if int(m.group(1)) in recordings else m.group(0), page) + + +TRACK_RX = re.compile(r"^/iiif/(\d+)/(?:transcript|files/(\d+))\.vtt$") + + +def sign_tracks(cfg: dict[str, Any], tracks: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Sign the links of a player's caption tracks (iiif.caption_tracks) as IIIF's probe service signs them, for as long + as media links last: a can't send the Authorization header either, and share links and visitors have none. + Only hand them to someone who may read what they open.""" + for t in tracks: + m = TRACK_RX.match(t["url"]) + if m: + what = f"vtt{m.group(2)}" if m.group(2) else "transcript" + t["url"] = f"{t['url']}?{iiif.sign(cfg, int(m.group(1)), what, settings.MEDIA_URL_EXPIRE_SECONDS)}" + return tracks diff --git a/fastapi_backend/app/api/v1/routes/public.py b/fastapi_backend/app/api/v1/routes/public.py index 238f84111..299091558 100644 --- a/fastapi_backend/app/api/v1/routes/public.py +++ b/fastapi_backend/app/api/v1/routes/public.py @@ -8,7 +8,7 @@ from fastapi import APIRouter, BackgroundTasks, HTTPException, Query from app.api.deps import Acl, Cfg, Db, Writer, domain_errors -from app.api.media import sign_url +from app.api.media import sign_tracks, sign_url from app.domain import access as acc from app.domain import public, store from app.domain.store import R @@ -110,6 +110,7 @@ def get_public_recording(rid: int, acl: Acl, db: Db, cfg: Cfg) -> PublicRecordin p["image"], p["thumb"] = sign_url(p["image"], member), sign_url(p["thumb"], member) for f in d["files"]: # and only the files they may download f["url"] = sign_url(f["url"], save=owner) + d["captions"] = sign_tracks(cfg, d["captions"]) # and the text they may read, as caption tracks return PublicRecording.model_validate(d) diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py index a1f44178f..6adefcdd7 100644 --- a/fastapi_backend/app/api/v1/routes/recordings.py +++ b/fastapi_backend/app/api/v1/routes/recordings.py @@ -15,16 +15,18 @@ from app.api import evidence as evidence_api from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Writer, domain_errors -from app.api.media import sign_url, sign_urls +from app.api.media import sign_tracks, sign_url, sign_urls from app.api.streaming import LOCKED, file_response, range_response from app.domain import access as acc from app.domain import ( analyze, auth, deletion, + documents, enhance, events_content, hierarchy, + iiif, ipgroups, jobs, keyring, @@ -39,6 +41,7 @@ video, ) from app.domain import fields as fieldmod +from app.domain import files as filemod from app.domain import metadata as md from app.domain.store import API, DB from app.schemas.common import AccessLevel, Ok @@ -622,7 +625,12 @@ def get_player(rid: int, acl: Acl, db: Db, cfg: Cfg, s: str = "") -> Player: data = render.player_data(db, rid, audio_link(db, cfg, rid, s)) data["nsfw"] = _nsfw(db, cfg, rec, rid) data["unsure_below"] = cfg["transcribe"].get("confidence_highlight") or None - return sign_urls(data, full=acl.member(rec)) + member = acl.member(rec) + # its text as WebVTT tracks for the