Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
- **Stay private and safe.** Passkeys instead of passwords (one at Lens's remote address added from a signed-in one with
Open signed in), encryption at rest with a key per namespace, roles per namespace (viewer, editor, owner, or custom
roles made of permissions), who sees each asset, set for many at once from the Library, access that ends on a date
you pick (a role in a namespace or on a collection, or permission on one asset), email invitations, removing
someone for good, and an audit log. Prompt injection meets
you pick (a role in a namespace or on a collection, or permission on one asset), a downloads switch per asset (off,
it still plays and reads, but only its owners save its files), email invitations, removing someone for good, and an
audit log. Prompt injection meets
several layers: what the assistant reads (recordings, documents, web pages, tool results) reaches the model marked as
data, not instructions, and text that reads like instructions to an AI is flagged (by free patterns, or an optional
classifier model that runs on your machine and is fetched only when you pick it). Once an answer has read such text,
Expand Down
6 changes: 6 additions & 0 deletions changelog.d/asset-downloads-switch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
- **A downloads switch for each asset.** Its owners can turn downloads off in its Access dialog or with `PUT
/api/v1/resources/<id>/access {"downloads": false}`. Then everyone else who sees it still plays it, sees its pages
and reads its text, but saving its own file, the PDF made of it or its supplementary files answers 403 with the
reason; the web app disables those download buttons and says why, the file tree and IIIF manifests leave the files
out, and owners keep saving. It's on for every asset until someone turns it off. It isn't copy protection: what
plays can still be recorded ([Downloads](docs/access.md#downloads)).
22 changes: 22 additions & 0 deletions docs/access.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ closed. Who may open what depends on who is asking.
| `access` | `public`, `restricted` or `private` | the namespace's default access, else `private` |
| `open` | the parts of a public asset anyone may use: `media` (audio or video, frames; a document's or an image's pages and its file), `transcript` (the asset's text or transcript, its search and downloads) and `index` (chapters, or a document's sections) | the namespace's default parts, else all three |
| `featured` | shown on the home page when public | no |
| `downloads` | whether people below its owners may save its files ([Downloads](#downloads)) | on |

An asset's **description** (its descriptive metadata: title, date, summary, people, subjects, rights) is never a
closed part: whoever may see the asset's page sees it, as IIIF manifests always have.
Expand All @@ -22,6 +23,27 @@ its toast's Undo puts each back as it was (agents: `POST /api/v1/metadata/bulk`
Moving an asset to another namespace doesn't change who sees it: the access it had from its old namespace is pinned
on it.

## Downloads

Owners can turn an asset's **downloads** off, in its Access dialog (the Downloads switch) or with `PUT
/api/v1/resources/<id>/access {"downloads": false}` ([API](api.md#resources)). It is on for every asset until someone
turns it off, so nothing changes by itself. With downloads off, saving a copy of the asset's files is for its owners
(the publish permission in its namespace, or admin of its collection); everyone else who sees it still plays it, sees
its pages and reads its text, as its access allows:

| Still works for everyone who sees it | Only owners, with downloads off |
|---|---|
| the player: `/resources/<id>/audio` (audio and video, in the workspace, on the pages visitors see, in embeds and share links) | its own file: `/resources/<id>/media`, the Download buttons, the file tree (S3, WebDAV, SMB, NFS and `/tree`) |
| its frames and pages, and IIIF's audio and video, pages, thumbnails and captions | the PDF made of a document (`/resources/<id>/pdf`, IIIF's) |
| its text: the transcript, search, the text exports and IIIF's transcript downloads, the evidence PDF | its supplementary files (`/resources/<id>/files/<id>/download`, IIIF's), which IIIF manifests stop offering |

Anyone else gets 403 with the reason ("Downloads are turned off for this asset"), and the web app disables its download
buttons and says why; `GET /resources/<id>` and its files list say `downloads` and `can_save`. Links the API signs for
an owner carry `save=1`, signed with the rest, so their download buttons keep working; through IIIF, owners save with
their token or the IIIF access cookie. The player streams what it plays, so someone determined can still record it:
the switch takes away the ways to save a copy, it isn't copy protection. Turning downloads off or on is audited as
`recording.access`.

## Files

A resource's supplementary files ([API](api.md#files)) follow its parts. On a public resource, transcripts, captions
Expand Down
7 changes: 5 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -765,8 +765,11 @@ gone) opens a neutral page with status 410 that says nothing about the asset.

`GET /resources/{rid}/access` says who may see an asset (members): `access` (`public`, `restricted` or
`private`), `open` (the parts a public asset opens to everyone: `media`, `transcript`, `index`), `featured`,
`inherited` (the access comes from the namespace) and the namespace's `default`. `PUT` changes any of them (owners);
`null` follows the namespace again. Changes are audited as `recording.access`, kept in the metadata history, and
`inherited` (the access comes from the namespace), the namespace's `default` and `downloads`. `PUT` changes any of
them (owners); `null` follows the namespace again. `downloads: false` keeps the asset's files (its own file, the PDF
made of it, its supplementary files) for its owners: everyone else gets 403 from `/media`, `/pdf` and `…/files/{id}/download`
but still plays it through `/audio`, sees its pages and reads its text ([Downloads](access.md#downloads)).
`GET /resources/{rid}` and `GET /resources/{rid}/files` say `downloads` and `can_save` (whether you may save them). Changes are audited as `recording.access`, kept in the metadata history, and
announced in the IIIF change feed. [Access](access.md) explains what each level lets people do.

`/resources/{rid}/permissions` lists the people given permission on the asset (owners), each with `expires_at` and
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Secret fields that refer to a Vaultwarden or Bitwarden vault, read by Lens and never shown to anyone, agents included | [test_secrets.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_secrets.py) `test_a_secret_field_reads_its_value_from_the_vault`, `test_the_vault_is_read_again_and_what_vanishes_stops_being_used`, `test_the_vault_is_set_up_and_checked_in_the_app`, `test_agents_refer_to_secrets_and_never_read_them`; [settings-secrets.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-secrets.test.tsx) `is picked from the vault's items, and only the reference is saved`, `shows its reference and whether it was read, never a value` |
| Roles per namespace: viewer, editor, owner, or custom roles made of permissions | [test_roles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_roles.py) `test_builtin_roles_keep_their_meaning`, `test_custom_roles_allow_exactly_their_permissions`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `person-add` |
| Access that ends on a date: a role in a namespace or on a collection, or permission on an asset, opens nothing after its end and lists show it as expired | [test_grant_ends.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_grant_ends.py) `test_asset_permission_holds_until_it_ends`, `test_namespace_role_holds_until_it_ends`, `test_collection_role_holds_until_it_ends`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_gives_a_role_until_a_date`; [access-until.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/access-until.test.ts) `sends the end of the day picked, and nothing for an empty field` |
| A downloads switch per asset: off, only its owners save its files (its own file, its PDF, supplementary files) while everyone who sees it still plays it and reads its text | [test_downloads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_downloads.py) `test_downloads_off_refuses_a_viewers_download_but_still_plays`, `test_downloads_off_keeps_iiif_playing`; [access-model.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/access-model.test.ts) `sends the downloads switch only when it changes, on by default`, `says why the files can't be saved, only when they can't` |
| Email invitations, and removing someone for good | [test_users.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_users.py) `test_invites_are_emailed_when_mail_is_set_up`, `test_admins_remove_people_for_good` |
| An audit log | [test_settings.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_settings.py) `test_settings_are_editable_and_secret`; [test_files.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_files.py) `test_change_a_file_and_the_audit_log` |
| Outside text reaches the model marked as data; text that reads like instructions is flagged, by patterns or an optional local classifier | [test_prompt_injection.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_prompt_injection.py) `test_what_tools_read_is_marked_as_data`, `test_a_result_that_reads_like_instructions_is_flagged`; [test_injection_model.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_injection_model.py) `test_until_the_model_is_here_the_patterns_stand_alone` |
Expand Down
21 changes: 20 additions & 1 deletion fastapi_backend/app/api/deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,26 @@ def who(self) -> acc.Who:

def signed(self) -> bool:
q = self.request.query_params
return security.verify_path(self.request.url.path, q.get("exp"), q.get("sig"), full=q.get("full") == "1")
return security.verify_path(self.request.url.path, q.get("exp"), q.get("sig"), full=q.get("full") == "1", save=q.get("save") == "1")

def owns(self, rec: dict[str, Any]) -> bool:
"""Whether this person acts as the recording's owner: the publish permission in its namespace, or admin of its
collection (admins have every permission)."""
if not self.user:
return False
coll = hierarchy.ROLES.get(self.collection_role(rec["space"], rec.get("collection")) or "", 0)
return self.can(rec["space"], "publish") or coll >= auth.ROLES[rl.minimum("publish")]

def saves(self, rec: dict[str, Any]) -> bool:
"""Whether this request may save the recording's files (docs/access.md#downloads): its downloads are on, or the
requester acts as its owner, or the link was signed for one (`save`)."""
q = self.request.query_params
return acc.saves(rec, self.owns(rec) or (q.get("save") == "1" and self.signed()))

def may_save(self, rec: dict[str, Any]) -> None:
"""403 unless this request may save the recording's files."""
if not self.saves(rec):
raise HTTPException(403, acc.DOWNLOADS_OFF)

def member(self, rec: dict[str, Any]) -> bool:
"""Whether this request is a member's: a role in the recording's namespace or on its collection, or a link
Expand Down
27 changes: 26 additions & 1 deletion fastapi_backend/app/api/iiif.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
from app.api.v1.routes.recordings import serve_audio
from app.api.v1.routes.video import serve_document
from app.domain import access as acc
from app.domain import auth, convert, documents, faces, iiif, iiif_auth, keyring, render, store, video
from app.domain import auth, convert, documents, faces, hierarchy, iiif, iiif_auth, keyring, render, store, video
from app.domain import files as filemod
from app.domain import metadata as md
from app.domain.store import DB, R
Expand Down Expand Up @@ -74,6 +74,27 @@ def _account_permitted(db: DB, acct: dict[str, Any], rid: int, space: int) -> bo
return acc.permitted(db, auth.roles(db, acct), acct["id"], rid, space)


def _owner(db: DB, perms: dict[int, frozenset[str]], account: int | None, rec: dict[str, Any]) -> bool:
if "publish" in perms.get(rec["space"], frozenset()):
return True
return bool(account) and hierarchy.roles_of(db, account).get(rec["space"], {}).get(rec.get("collection")) == "admin"


def _may_save(request: Request, db: DB, user: Principal | None, rid: int) -> None:
"""With the asset's downloads off (docs/access.md#downloads), its files (its own, the PDF made of it, the
supplementary ones) are for its owners: the requester's, or the IIIF access cookie's account's. 403 for anyone
else; playing it, its pages, captions and text stay as its access allows."""
rec = db.one("SELECT space, collection, downloads FROM $r", r=R("recording", rid)) or {}
if acc.downloads(rec):
return
if user and _owner(db, user.perms, user.id, rec):
return
acct, _ = iiif_auth.cookie_account(db, request.cookies.get(iiif_auth.COOKIE))
if acct and _owner(db, auth.permissions(db, acct), acct["id"], rec):
return
raise HTTPException(403, acc.DOWNLOADS_OFF)


def _member(request: Request, db: DB, user: Principal | None, space: int) -> bool:
"""A role in the namespace: the requester's, or the IIIF access cookie's account's."""
if auth.allows(_roles(user), space):
Expand Down Expand Up @@ -450,6 +471,7 @@ def iiif_audio(rid: int, request: Request, user: OptionalUser, db: Db, cfg: Cfg)
raise HTTPException(401, "sign in through the viewer to play this recording")
full = db.one("SELECT * FROM $r", r=R("recording", rid))
if full.get("source") in documents.KINDS: # a document or an image: its file, to save
_may_save(request, db, user, rid)
return serve_document(db, cfg, full, request)
return serve_audio(db, cfg, full, rid, request)

Expand All @@ -464,6 +486,7 @@ def iiif_pdf(rid: int, request: Request, user: OptionalUser, db: Db, cfg: Cfg) -
path = convert.rendition_path(cfg, rid)
if rec.get("source") != "document" or not path.is_file():
raise HTTPException(404, "not found")
_may_save(request, db, user, rid)
stem = pathlib.PurePosixPath(str((rec.get("remote") or {}).get("path") or rec.get("path") or "document")).stem or "document"
return stored_file(db, cfg, path, request, "application/pdf", f"{stem}.pdf", filemod.HEADERS)

Expand Down Expand Up @@ -533,6 +556,8 @@ def iiif_file(rid: int, name: str, request: Request, user: OptionalUser, db: Db,
raise HTTPException(401, "sign in through the viewer to open this file")
if vtt:
return Response(filemod.as_vtt(db, f), media_type="text/vtt; charset=utf-8")
if f["role"] != "thumbnail": # a thumbnail is how the manifest shows the asset; the rest are files to save
_may_save(request, db, user, rid)
path = filemod.path_of(cfg, f)
if not path.is_file():
raise HTTPException(404, "not found")
Expand Down
17 changes: 9 additions & 8 deletions fastapi_backend/app/api/media.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,35 +34,36 @@
)


def sign_url(url: str | None, full: bool = False) -> str | None:
def sign_url(url: str | None, full: bool = False, save: bool = False) -> str | None:
"""`full` for a link handed to a member of the recording's namespace: pictures as they are, where the namespace
pixelates faces for visitors (docs/video.md)."""
pixelates faces for visitors (docs/video.md). `save` for one handed to an owner of the asset: its files save even
with its downloads off (docs/access.md#downloads)."""
if not url:
return url
path, _, query = url.partition("?")
if not (MEDIA_RX.match(path) or NS_MEDIA_RX.match(path) or REPORT_RX.match(path)):
return url
params = dict(urllib.parse.parse_qsl(query))
for k in ("exp", "sig", "full"):
for k in ("exp", "sig", "full", "save"):
params.pop(k, None)
return sign_path(path, full=full, **cast(dict[str, Any], params))
return sign_path(path, full=full, save=save, **cast(dict[str, Any], params))


def sign_urls[T](obj: T, full: bool = False) -> T:
def sign_urls[T](obj: T, full: bool = False, save: bool = False) -> T:
"""Sign the media links in a JSON-like structure (in place for dicts and lists; returns it): the values of the
LINK_KEYS fields, at any depth. Other strings are left alone, however much they look like a link. `full` as in
sign_url: for a response to someone with a role in every recording's namespace (the API's own responses are)."""
if isinstance(obj, dict):
for k, v in obj.items():
if isinstance(v, str):
if k in LINK_KEYS:
obj[k] = sign_url(v, full)
obj[k] = sign_url(v, full, save)
elif isinstance(v, (dict, list)):
sign_urls(v, full)
sign_urls(v, full, save)
elif isinstance(obj, list):
for v in obj:
if isinstance(v, (dict, list)):
sign_urls(v, full)
sign_urls(v, full, save)
return obj


Expand Down
18 changes: 11 additions & 7 deletions fastapi_backend/app/api/v1/routes/files.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,9 @@ def _out(db: DB, rows: list[dict[str, Any]], a: dict[str, Any]) -> list[Resource
]


def _signed(out: Any) -> Any:
"""The response with its download links signed."""
return type(out).model_validate(sign_urls(out.model_dump(), full=True))
def _signed(out: Any, save: bool = False) -> Any:
"""The response with its download links signed (`save`: for an owner, so they save even with downloads off)."""
return type(out).model_validate(sign_urls(out.model_dump(), full=True, save=save))


def _file(db: DB, rid: int, fid: int) -> dict[str, Any]:
Expand All @@ -97,16 +97,19 @@ def _file(db: DB, rid: int, fid: int) -> dict[str, Any]:
@router.get("")
def list_files(rid: int, acl: Acl, db: Db, cfg: Cfg) -> ResourceFiles:
"""The resource's primary file (its audio or video, if it has one) and its supplementary files, with signed links to
download them."""
download them; `downloads` and `can_save` say whether its owners turned downloads off and whether you may save
them anyway (owners do)."""
rec = acl.recording(rid)
can_change = acl.rank_in(rec["space"], rec.get("collection")) >= auth.ROLES["editor"]
out = ResourceFiles(
primary=_primary(db, cfg, rid, rec),
files=_out(db, files.of(db, rid), acc.of(db, rid)),
max_mb=cfg["server"]["max_upload_mb"],
can_change=can_change,
downloads=acc.downloads(rec),
can_save=acl.saves(rec),
)
return _signed(out)
return _signed(out, save=acl.owns(rec))


async def _receive(request: Request, cfg: dict[str, Any]) -> pathlib.Path:
Expand Down Expand Up @@ -203,8 +206,9 @@ def delete_file(rid: int, fid: int, acl: Acl, user: Writer, db: Db, cfg: Cfg) ->

@router.get("/{fid}/download", response_class=FileResponse, responses={200: {"content": {"application/octet-stream": {}}}})
def download_file(rid: int, fid: int, request: Request, acl: Acl, db: Db, cfg: Cfg) -> Response:
"""The file as it was added, to save. Accepts a bearer token or a signed link (from the list)."""
acl.recording(rid)
"""The file as it was added, to save. Accepts a bearer token or a signed link (from the list). With the asset's
downloads off, 403 for anyone below its owners."""
acl.may_save(acl.recording(rid))
f = _file(db, rid, fid)
path = files.path_of(cfg, f)
if not path.is_file():
Expand Down
Loading
Loading