Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,11 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off
by default). Say what a file or a page is as it comes in (a podcast, an interview, a scan), in Import, by API or
to the assistant, and its content type's pipeline runs from the start. A file, page or video the namespace already
has links to the one there instead of being imported again.
has links to the one there instead of being imported again. Media browsers can't play, such as an AVI or HEVC from a
phone, can get a copy they can (off by default).
([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika),
[import webhooks](docs/api.md#import-webhooks); [checks](docs/requirements.md#bring-in-everything))
[import webhooks](docs/api.md#import-webhooks), [playable copies](docs/video.md#playable-copies);
[checks](docs/requirements.md#bring-in-everything))
- **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a
part you pick into Markdown, lets you look it over, and sends it to a namespace. New releases reach installed copies
through the browser's store or a self-hosted update file. In Chrome and Edge, its side panel is a chat with the
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/playable-copies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Playable copies.** With Settings › Video › Playable copies on (off by default), media browsers can't play, such as
HEVC in a .mov, AVI, MTS or AMR, gets one copy they can (H.264 and AAC in MP4, at most 720 lines high by default),
made by the transcribe step. The player gets the copy; downloads still give the file itself
([Playable copies](docs/video.md#playable-copies)).

1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Import webhooks for scanners and phone shortcuts | [test_import_hooks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_hooks.py) `test_pushing_files_urls_and_text`, `test_owners_manage_hooks`; [import-hooks.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/import-hooks.test.tsx) |
| Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` |
| Uploads of audio, PDFs, Office files, photos, JSON and archives | [test_uploads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_uploads.py) `test_uploading_audio_in_chunks`, `test_json_uploaded_is_read_as_records`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_a_pdf_becomes_a_document_with_its_pages_and_text`, `test_scans_and_images_are_read_by_ocr`; [test_conversion.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversion.py) `test_office_and_text_files_become_documents`; [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_an_uploaded_zip_keeps_its_files`; [test_archives.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_archives.py) `test_reading_an_archive`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `import-file`, `import-paste` |
| Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` |
| An uploaded video, end to end | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| Apache Tika (opt-in) reads documents nothing else here can | [test_tika.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_tika.py) `test_off_by_default_nothing_changes`, `test_what_nothing_here_can_convert_tika_reads_as_text` |
| Web pages captured one at a time, from a link list or a bookmarks file, by API or by asking; private networks only when allowed | [test_webcapture.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webcapture.py) `test_a_web_page_becomes_a_document`, `test_an_admin_allows_a_private_network_in_settings`, `test_private_addresses_are_refused`; [test_web_links.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_links.py) `test_one_or_several_addresses_as_a_list`, `test_a_bookmarks_file_becomes_pages_to_capture`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_captures_web_pages` |
Expand Down
18 changes: 18 additions & 0 deletions docs/video.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,24 @@ videos, documents and images.
A published video recording becomes a IIIF Video on a Canvas with its width, height and duration. It gets a thumbnail,
a "Text on screen" annotation layer targeting `#xywh=…&t=…`, and its shots as a Range.

## Playable copies

Phones and cameras write files many browsers can't play: HEVC video in a .mov, AVI, MPEG transport streams, AMR voice
memos, Windows Media. Turn on Settings › Video › Playable copies (`video.playable_copies`, off by default, since
transcoding a long video keeps a small machine busy for a while) and the transcribe step makes one copy of such a
recording's media, even when its transcript came with it:

- video becomes H.264 at most `video.playable_height` lines high (360, 480, 720 or 1080; 720 by default) with AAC sound
in MP4, and sound alone becomes AAC in MP4;
- the copy is kept in `data_dir/playable/`, encrypted like the rest when the namespace is, and deleted with the
recording;
- the player (`/api/v1/recordings/<id>/audio`) gets the copy, while `/api/v1/recordings/<id>/media`, IIIF and feeds
still give the file itself;
- a file browsers already play (H.264, VP8, VP9 or AV1 video; AAC, MP3, Opus, Vorbis, FLAC or PCM sound, in MP4,
WebM, Ogg, MP3, FLAC or WAV) gets no copy.

Recordings transcribed before the setting was on get their copy when their transcribe step runs again.

## The video recording page

Video recordings open in their own layout.
Expand Down
15 changes: 11 additions & 4 deletions fastapi_backend/app/api/v1/routes/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
library,
moving,
nsfw,
playable,
render,
sources,
store,
Expand Down Expand Up @@ -587,10 +588,14 @@ def _media_type(rec: dict[str, Any], path: Any) -> str | None:
return None


def serve_audio(db: DB, cfg: dict[str, Any], rec: dict[str, Any], rid: int, request: Request) -> Response:
"""The recording's audio (or video) file with byte ranges: local, from a local source, from the cache, or streamed."""
def serve_audio(db: DB, cfg: dict[str, Any], rec: dict[str, Any], rid: int, request: Request, player: bool = False) -> Response:
"""The recording's audio (or video) file with byte ranges: local, from a local source, from the cache, or streamed.
For the player, its playable copy when one was made (playable.py); the file itself otherwise."""
if rec.get("source") != "audio":
raise HTTPException(404, "no audio for this recording")
copy = playable.copy_of(db, cfg, rid) if player else None
if copy:
return file_response(copy[0], request, copy[1], db, cfg)
path = render.has_audio(db, cfg, rid)
if path:
return file_response(path, request, _media_type(rec, path), db, cfg)
Expand All @@ -615,8 +620,10 @@ def serve_audio(db: DB, cfg: dict[str, Any], rec: dict[str, Any], rid: int, requ

@router.get("/{rid}/audio", response_class=Response, responses={200: {"content": {"audio/*": {}}}, 206: {"description": "a byte range"}})
def get_audio(rid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, s: str = "") -> Response:
"""The audio, with byte ranges. Accepts a bearer token, a share link (``?s=``) or a signed link."""
return serve_audio(db, cfg, acl.recording(rid, share=s), rid, request)
"""The audio (or video) for the player, with byte ranges: a copy every browser plays when Settings › Video made one,
else the file itself (``/media`` always gives the file). Accepts a bearer token, a share link (``?s=``) or a signed
link."""
return serve_audio(db, cfg, acl.recording(rid, share=s), rid, request, player=True)


@router.get("/{rid}/wordcloud.svg", response_class=Response, responses={200: {"content": {"image/svg+xml": {}}}})
Expand Down
3 changes: 2 additions & 1 deletion fastapi_backend/app/domain/deletion.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
import pathlib
import shutil

from . import access as acc, files as filemod, keyring, notebook, render, store, video
from . import access as acc, files as filemod, keyring, notebook, playable, render, store, video

R = store.R
log = logging.getLogger("lens")
Expand Down Expand Up @@ -99,6 +99,7 @@ def _files(db, cfg, rec, rid, ns):
]
if rec.get("converted_from") and rec.get("path") and keyring.owned(cfg, rec["path"]):
out.append(pathlib.Path(rec["path"]))
out += playable.copies(cfg, rid)
exports = data / "exports" / (ns or "_")
mine = {
(o.get("value") or {}).get("file")
Expand Down
14 changes: 14 additions & 0 deletions fastapi_backend/app/domain/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,20 @@ def _read(db, cfg, rid, say, spec=None):


def _transcribe(db, cfg, rid, say, spec=None):
"""A recording transcribed (or a document read), then, with Settings › Video › Playable copies on, a copy of its
media every browser plays made (playable.py), even when the transcript came with the media."""
from . import playable

try:
done = _transcribe_media(db, cfg, rid, say, spec)
except Skip:
playable.ensure(db, cfg, rid, say)
raise
playable.ensure(db, cfg, rid, say)
return done


def _transcribe_media(db, cfg, rid, say, spec=None):
rec = db.one("SELECT source, engine, status, envelope, web, path, space, title FROM $r", r=R("recording", rid)) or {}
if rec.get("source") in ("document", "image"): # a run queued before documents had a step of their own
return _read(db, cfg, rid, say, spec)
Expand Down
130 changes: 130 additions & 0 deletions fastapi_backend/app/domain/playable.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
"""Playable copies: media that every browser plays, made of what phones and cameras record (`video.playable_copies`).

Phones and camcorders write files many browsers can't play: HEVC video in a .mov, AVI, MPEG transport streams (.mts),
AMR voice memos, Windows Media. With the setting on (off by default: transcoding a long video takes a small machine a
while), the transcribe step makes a copy of such a recording's media once, next to its other files
(data_dir/playable/<recording>.mp4 or .m4a, encrypted like the rest when the namespace is): H.264 video at most
`playable_height` lines high and AAC sound in MP4, or AAC alone for sound. The player then gets the copy; the original
stays as it was, and downloads of the original still give it. A file a browser already plays (H.264, VP8, VP9 or AV1
video; AAC, MP3, Opus, Vorbis, FLAC or PCM sound, in MP4, WebM, Ogg, MP3, FLAC or WAV) gets no copy.
"""

from __future__ import annotations

import json
import pathlib
import subprocess

from . import ingest, keyring, store

R = store.R
VIDEO = {"h264", "vp8", "vp9", "av1"}
AUDIO = {"aac", "mp3", "opus", "vorbis", "flac", "pcm_s16le", "pcm_s24le", "pcm_s32le", "pcm_u8", "pcm_f32le"}
CONTAINERS = {".mp4", ".m4a", ".m4v", ".webm", ".mp3", ".ogg", ".oga", ".opus", ".wav", ".flac", ".aac"}
SECONDS = 6 * 3600 # the longest ffmpeg may take over one copy


def on(cfg):
return bool((cfg.get("video") or {}).get("playable_copies"))


def codecs(path):
"""(the video codec or None, the audio codec or None) of a media file, by ffprobe; (None, None) when unreadable."""
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-format_whitelist", ingest.MEDIA_DEMUXERS, "-show_streams", "-of", "json", str(path)],
capture_output=True,
text=True,
timeout=120,
)
streams = json.loads(out.stdout or "{}").get("streams") or []
except (OSError, ValueError, subprocess.TimeoutExpired):
return None, None
video = next(
(s.get("codec_name") for s in streams if s.get("codec_type") == "video" and not (s.get("disposition") or {}).get("attached_pic")),
None,
)
audio = next((s.get("codec_name") for s in streams if s.get("codec_type") == "audio"), None)
return video, audio


def needed(path, name=None):
"""Whether a browser may fail to play this file: its kind of file, or a codec in it, isn't one they all play."""
video, audio = codecs(path)
if video is None and audio is None:
return False # nothing ffprobe can read: nothing ffmpeg could copy either
suffix = pathlib.PurePosixPath(str(name or path)).suffix.lower()
return suffix not in CONTAINERS or (video is not None and video not in VIDEO) or (audio is not None and audio not in AUDIO)


def path_of(cfg, rid, video=True):
return pathlib.Path(cfg["data_dir"]) / "playable" / f"{int(rid)}{'.mp4' if video else '.m4a'}"


def copies(cfg, rid):
"""Both places a recording's copy may be, for deletion.py."""
return [path_of(cfg, rid, True), path_of(cfg, rid, False)]


def copy_of(db, cfg, rid):
"""The playable copy of a recording's media, when one was made and is still there: (its path, its media type), else
None."""
rec = db.one("SELECT playable FROM $r", r=R("recording", rid)) or {}
got = rec.get("playable") or {}
if not got.get("type"):
return None
p = path_of(cfg, rid, video=got["type"].startswith("video/"))
return (str(p), got["type"]) if p.is_file() else None


def ensure(db, cfg, rid, say=print, force=False):
"""Make a recording's playable copy when the setting is on and its media needs one; the copy's path, else None."""
if not on(cfg):
return None
have = copy_of(db, cfg, rid)
if have and not force:
return have[0]
rec = db.one("SELECT path, remote, space, source FROM $r", r=R("recording", rid)) or {}
if rec.get("source") != "audio":
return None
name = (rec.get("remote") or {}).get("path") or rec.get("path")
src = ingest.audio_path(db, cfg, rec)
if not needed(src, name):
return None
video, _ = codecs(src)
dest = path_of(cfg, rid, video=video is not None)
dest.parent.mkdir(parents=True, exist_ok=True)
tmp = dest.with_name("." + dest.name)
height = int((cfg.get("video") or {}).get("playable_height") or 720)
args = ["-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-pix_fmt", "yuv420p", "-vf", f"scale=-2:'min({height},ih)'"]
argv = ["ffmpeg", "-nostdin", "-y", "-v", "error", "-format_whitelist", ingest.MEDIA_DEMUXERS, "-i", str(src), "-map", "0:a:0?"]
argv += (["-map", "0:v:0", *args] if video else ["-vn"]) + [
"-c:a",
"aac",
"-b:a",
"128k",
"-movflags",
"+faststart",
"-f",
"mp4",
str(tmp),
]
try:
done = subprocess.run(argv, capture_output=True, text=True, timeout=SECONDS)
if done.returncode != 0 or not tmp.is_file():
say(f" couldn't make a playable copy: {((done.stderr or '').strip().splitlines() or ['ffmpeg failed'])[-1]}")
return None
tmp.replace(dest)
except (OSError, subprocess.TimeoutExpired) as e: # no ffmpeg, or a copy taking too long: the original still plays
say(f" couldn't make a playable copy: {e}")
return None
finally:
tmp.unlink(missing_ok=True)
keyring.protect(db, cfg, rec.get("space"), dest)
db.q(
"UPDATE $r SET playable = $p",
r=R("recording", rid),
p={"type": "video/mp4" if video else "audio/mp4", "made_at": store.now()},
)
say(f" made a copy every browser plays ({'video' if video else 'sound'}, {dest.stat().st_size // 1024} KB)")
return str(dest)
3 changes: 3 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@
# the model files (yolox_model, ultralytics_model) are startup settings only, like yunet_model
"object_engine",
"object_min_score",
"playable_copies",
"playable_height",
),
"nsfw": None,
"enhance": None,
Expand Down Expand Up @@ -178,6 +180,7 @@
("video", "face_engine"): {"opencv", "insightface", "anytopdf", "none"},
("video", "object_engine"): {"yolox", "ultralytics", "anytopdf", "off"},
("enhance", "image_engine"): {"classic", "realesrgan"},
("video", "playable_height"): {360, 480, 720, 1080},
("decisions", "engine"): {"auto", "jev", "laya", "llm", "off"},
("voice", "input"): {"auto", "server", "browser"},
("voice", "stt"): {"same", "sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"},
Expand Down
3 changes: 3 additions & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,9 @@ def labels():
"yolox_model": None,
"ultralytics_model": None,
"object_min_score": 0.4,
# playable copies (playable.py): off by default, since transcoding a long video takes a small machine a while
"playable_copies": False,
"playable_height": 720,
},
# NSFW classification (nsfw.py, docs/nsfw.md): off until an admin turns it on here, or an owner for their namespace
# (which can also choose its own threshold and display). A local model (ViT, Apache-2.0, fetched on first use) scores
Expand Down
Loading
Loading