From 032d8e4845891a53e27e1d69df8cb578a0861814 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Mon, 19 Oct 2026 15:24:56 +0000 Subject: [PATCH] feat(library): show photos and videos on a map by where they were taken Library gets a Map view (beside Table and List, on a phone too): every asset whose file gave a GPS fix is a dot, with the Library's filters, framed to fit, on Leaflet (BSD-2) with tiles from the new Settings > Maps (OpenStreetMap by default, or none). A video's ISO 6709 location is now read at import beside a photo's EXIF GPS, and a photo anytopdf already placed counts too. GET /resources takes located and bbox (west,south,east,north) and gives each row its place, members only; MCP list_recordings takes the same. The threat model notes that the tile server sees the area looked at (N-MAP-TILES). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01QFUNzy1QZy5HhKAyJxTAYJ --- README.md | 22 ++- changelog.d/photo-map.md | 9 + docs/api.md | 7 +- docs/configuration.md | 19 +- docs/mcp.md | 4 +- docs/processing.md | 29 ++- docs/requirements.md | 1 + docs/technology-selection.md | 2 + fastapi_backend/app/api/mcp_tools.py | 16 +- .../app/api/v1/routes/recordings.py | 24 ++- fastapi_backend/app/domain/ingest.py | 30 ++- fastapi_backend/app/domain/library.py | 39 +++- fastapi_backend/app/domain/settings.py | 23 +++ fastapi_backend/app/domain/store.py | 3 + fastapi_backend/app/schemas/recordings.py | 31 ++- fastapi_backend/archive.example.yaml | 4 + fastapi_backend/tests/api/test_photo_map.py | 119 +++++++++++ .../__tests__/library-map.test.tsx | 140 +++++++++++++ nextjs-frontend/app/openapi-client/index.ts | 4 + nextjs-frontend/app/openapi-client/sdk.gen.ts | 17 ++ .../app/openapi-client/types.gen.ts | 62 +++++- .../components/library/library-screen.tsx | 46 +++-- .../components/library/map-model.ts | 48 +++++ .../components/library/places-map.tsx | 187 ++++++++++++++++++ .../components/settings/bodies.tsx | 7 + nextjs-frontend/components/settings/find.ts | 5 +- nextjs-frontend/components/settings/model.ts | 30 +++ .../components/settings/settings-shell.tsx | 2 + nextjs-frontend/openapi.json | 111 ++++++++++- nextjs-frontend/package.json | 2 + nextjs-frontend/pnpm-lock.yaml | 23 +++ threat-model.json | 19 +- threat-model.md | 3 + threat-model.yaml | 15 ++ 34 files changed, 1044 insertions(+), 59 deletions(-) create mode 100644 changelog.d/photo-map.md create mode 100644 fastapi_backend/tests/api/test_photo_map.py create mode 100644 nextjs-frontend/__tests__/library-map.test.tsx create mode 100644 nextjs-frontend/components/library/map-model.ts create mode 100644 nextjs-frontend/components/library/places-map.tsx diff --git a/README.md b/README.md index adfa09a8b..83b71fdb5 100644 --- a/README.md +++ b/README.md @@ -144,18 +144,20 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs, Office files, photos, JSON and zip or tar archives (photos and videos dated by when they were taken, from the file - itself, and photos placed by their GPS), and transcripts made elsewhere (SRT, WebVTT, text, Word, PDF), each line - credited to the speaker the file names; and, with an Apache Tika server (an opt-in extra), documents nothing else - here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking - the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off - by default), taking a video's own captions as its transcript when people wrote some (off by default too). Say what a - file or a page is as it comes in (a podcast, an interview, a scan), in Import, by API or to the assistant, and its - content type's pipeline runs from the start. A file, page or video the namespace already has links to the one there - instead of being imported again. Media browsers can't play, such as an AVI or HEVC from a phone, can get a copy they - can (off by default). A bulk import (a folder scan, an archive's files, a link list, a source's files) is one run: - the toast after it, and the Import screen's list of recent runs, undo it whole. + itself, and photos placed by their GPS on the Library's map, by API and MCP too), and transcripts made elsewhere + (SRT, WebVTT, text, Word, PDF), each line credited to the speaker the file names; and, with an Apache Tika server (an + opt-in extra), documents nothing else here can read; capture web pages one at a time, from a link list or a + bookmarks file, by API or by asking the assistant, from private networks you allow in Settings too; and import + videos and podcasts by their address (off by default), taking a video's own captions as its transcript when people + wrote some (off by default too). Say what a file or a page is as it comes in (a podcast, an interview, a scan), in + Import, by API or to the assistant, and its content type's pipeline runs from the start. A file, page or video the + namespace already has links to the one there instead of being imported again. Media browsers can't play, such as an + AVI or HEVC from a phone, can get a copy they can (off by default). A bulk import (a folder scan, an archive's + files, a link list, a source's files) is one run: the toast after it, and the Import screen's list of recent runs, + undo it whole. ([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika), [import webhooks](docs/api.md#import-webhooks), [playable copies](docs/video.md#playable-copies), + [where an asset was taken](docs/processing.md#where-an-asset-was-taken), [import runs](docs/api.md#import-runs); [checks](docs/requirements.md#bring-in-everything)) - **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a diff --git a/changelog.d/photo-map.md b/changelog.d/photo-map.md new file mode 100644 index 000000000..6e6298629 --- /dev/null +++ b/changelog.d/photo-map.md @@ -0,0 +1,9 @@ +- **Photos and videos on a map.** Library → **Map** (beside Table and List, on a phone too) shows each asset whose own + file said where it was taken as a dot, with the Library's filters, framed to show them all; a dot links to its + asset. Besides a photo's EXIF GPS fix, a video's location is now read as it arrives (the ISO 6709 tag iPhones and + Android phones write). Agents get the same through `GET /api/v1/resources?located=true&bbox=west,south,east,north`, + which also gives every row its `place`, and the MCP tool `list_recordings` (`located`, `bbox`); only members see + where an asset was taken. The tiles under the map come from Settings → **Maps** (OpenStreetMap's by default); each + viewer's browser fetches them, so the tile server sees the area looked at: point it at your own tile server, or leave + it empty for none ([Where an asset was taken](docs/processing.md#where-an-asset-was-taken), + [Maps](docs/configuration.md#maps)). Assets that arrived before keep what they had: nothing needs upgrading. diff --git a/docs/api.md b/docs/api.md index ec0d80a4b..1d5ccac87 100644 --- a/docs/api.md +++ b/docs/api.md @@ -612,6 +612,7 @@ GET /api/v1/resources/origins GET /api/v1/resources/languages GET /api/v1/resources/objects GET /api/v1/resources/nsfw +GET /api/v1/resources/map-tiles POST /api/v1/resources/tags POST /api/v1/resources/collection GET /api/v1/resources/{rid} @@ -680,13 +681,17 @@ repeat a parameter that takes several values (`?status=new&status=error`) to mat | `emotion` | emotions heard in them, ignoring case | | `edited_by` | `me`: assets you edited (corrected a line of the text, changed the catalogue record, or renamed) | | `sensitive` | where [NSFW classification](nsfw.md) is on: `only` the ones flagged NSFW, `exclude` them, or `include` them. Without it, flagged ones are left out where their namespace hides them (`display: hide`) | +| `located` | `true`: only assets known to have been taken somewhere (a `place`, [Where an asset was taken](processing.md#where-an-asset-was-taken)) | +| `bbox` | `west,south,east,north` in degrees (GeoJSON's order): only assets taken inside that box; west past east crosses the antimeridian | | `sort` | `date`, `title`, `duration`, `speakers`, `status` or `importance`; `-` in front for descending (default `-date`). Assets without the value come last either way | | `limit`, `offset` | one page (default 500 rows, at most 1000) | The body is the page's rows; the `X-Total-Count` header says how many assets match on all pages. Rows also say where each asset came from (`origin`, and `origin_name`: the source's name, or e.g. "Uploaded"), -its `language` when known and the collection it lives in (`collection`, `collection_name`). `GET /resources/{rid}` +its `language` when known, the collection it lives in (`collection`, `collection_name`) and where it was taken +(`place`: `{lat, lon}`, or null). `GET /resources/map-tiles` (anyone signed in) gives the tiles the web app's map draws +on: `url`, an XYZ address with `{z}`, `{x}` and `{y}` (null for none), and its `attribution` (Settings → Maps). `GET /resources/{rid}` adds `collection_path`: the collections from the top of the namespace down to its own, `[{id, name}]`. `POST /resources/collection {recordings, collection}` moves assets into a collection of their namespace (editors of each); an asset of another namespace is a 400 (move it to that namespace first). It answers how many `moved`; diff --git a/docs/configuration.md b/docs/configuration.md index 37aa26f41..2ef7417b3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -112,7 +112,7 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word -search and search by meaning), reports and graph, +search and search by meaning), reports and graph, [maps](#maps), video, OCR, faces and objects, [NSFW classification](nsfw.md), [enhancing images](enhance.md), workers, sign-in, components (and the edition, full or lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript @@ -290,6 +290,23 @@ resource are described. Each call is in the [activity ledger](activity.md) (`mod Agents read and change these with the MCP tools `picture_descriptions` and `set_picture_descriptions` (admins). +## Maps + +The Library's Map shows where photos and videos were taken +([Where an asset was taken](processing.md#where-an-asset-was-taken)) on tiles that each viewer's browser fetches +itself. Settings → Maps: + +| Setting | Default | | +|---|---|---| +| `maps.tile_url` | `https://tile.openstreetmap.org/{z}/{x}/{y}.png` | where the tiles come from: an http(s) address with `{z}`, `{x}` and `{y}` (and `{s}` for a subdomain, if the server has them), with no password in it. Empty draws no tiles, only the dots | +| `maps.attribution` | `© OpenStreetMap contributors` | the credit the tiles' terms ask to be shown on the map, one line | + +The tile server sees which tiles are asked for, so the area each person looks at (around where their photos were +taken) and the address they look from, as any web map's does; nothing else about the assets goes to it. To keep that +on your own hardware, run a tile server of your own and give its address, or leave the address empty. OpenStreetMap's +tiles are for light use under its [tile usage policy](https://operations.osmfoundation.org/policies/tiles/); a busy +install should use its own or a provider's. + ## Uploads Audio, video, documents (PDF, Office and OpenDocument files, text, Markdown, saved web pages and emails) and images diff --git a/docs/mcp.md b/docs/mcp.md index 374f861de..f06c04d5d 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -76,11 +76,11 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `classify` | which of some labels fits an asset or a text, how sure, and every label ranked: labels you give, or Lens's content types or the namespace's topics; the decision model answers, else the language model. Read only: it changes nothing ([Classifying](processing.md#classifying)) | | `list_namespaces` | the namespaces you can read, with their assets, hours, people and your role | | `budgets` | where spending stands: each namespace you can read this period against its budget, and for admins all of Lens and every other budget, each ok, near or over ([Asking the assistant or an agent](budgets.md#asking-the-assistant-or-an-agent)) | -| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) | +| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, whether they're flagged NSFW (`sensitive`), and where they were taken: each with its `place` (`lat`, `lon`) when its file had a GPS fix, `located` for only those and `bbox` (`west,south,east,north`) for those inside a box ([Where an asset was taken](processing.md#where-an-asset-was-taken)) | | `list_speakers` | the people of your namespaces, the ones with the most talk time first | | `list_unnamed_faces` | the faces nobody has named yet in your namespaces that recognise faces, the largest group first: a name to offer, how often each is seen and the assets it's in, with links ([Naming faces](video.md#naming-faces)) | | `name_face` | say who a face is: a name another face of its namespace has joins that person, any other becomes its own. Needs a write-scope token and editor access | -| `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW | +| `get_recording` | one asset: when, where it was taken (`place`), how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW | | `get_transcript` | an asset's lines in order (who, time, text, link, and the words transcription wasn't sure of as `unsure`), a page at a time, from a line or a time window | | `fetch` | an asset's whole text at once, as `[m:ss] Speaker: …` lines | | `cite` | a citation for a moment (a line or a second, and a few lines from there): the words, who said or wrote them, the asset and date, the time, the link, and all of it as Markdown | diff --git a/docs/processing.md b/docs/processing.md index 07fbbff68..e4e9054ec 100644 --- a/docs/processing.md +++ b/docs/processing.md @@ -96,9 +96,32 @@ to an import webhook), from the first of these that says it: Nothing is guessed: a clock that was never set (a year before 1971, such as QuickTime's 1904) or a date in the future counts as no date. A date an imported transcript gives wins over all three, as an email's or a document's own date does once it is read, and a file uploaded again into the namespace that has it keeps its asset's date. A file on a -[storage source](#storage-sources) is dated by the source (the date it lists, else when the file last changed there). A photo's GPS fix from its EXIF is kept as the asset's `place` (`{lat, lon}`, in degrees); -`GET /api/v1/recordings/{rid}` and the MCP tool `get_recording` give it to the namespace's members only, never to -visitors. It is read on this machine and nothing is sent anywhere. Assets that arrived before keep their dates. +[storage source](#storage-sources) is dated by the source (the date it lists, else when the file last changed there). +Assets that arrived before keep their dates. + +### Where an asset was taken + +As it arrives, an asset's own file can say where it was taken, and that is kept as its `place` (`{lat, lon}`, WGS 84 +degrees to six places): a photo's EXIF GPS fix (`GPSLatitude`, `GPSLongitude` and their `Ref`s), or a video's location +as ffprobe reads it, in the ISO 6709 form phones write (`+48.8582+002.2945/`: an iPhone's +`com.apple.quicktime.location.ISO6709`, an Android phone's `location`). A fix of 0, 0 is a device with no fix, and +counts as none. It is read on this machine with Pillow and ffprobe, which Lens has anyway, and nothing is sent +anywhere. A photo that arrived before this was kept, but that [anytopdf](configuration.md#anytopdf) read with its GPS +fix, is placed by that fix; other assets that arrived before have no place, and nothing about them changes. + +Only the namespace's members see where an asset was taken, never visitors (public pages, embeds and share links): + +- **The web app**: Library → **Map** (beside Table and List; on a phone, beside List) puts each asset with a place on a + map as a dot, with the Library's filters and tab, framed to show them all; a dot's popup links to the asset. It + draws the newest 1,000 that match and says when there are more. The tiles under the map are fetched by the browser + from Settings → Maps (`maps.tile_url`, OpenStreetMap's by default), so the tile server sees the area being looked at + and the address it is looked at from: point it at a tile server of your own, or leave it empty to draw only the + dots ([Configuration](configuration.md#maps)). +- **The API**: `GET /api/v1/resources?located=true` lists only assets with a place, and + `bbox=west,south,east,north` (degrees, in GeoJSON's order; west past east crosses the antimeridian) those taken + inside a box. Both combine with the list's other filters, and each row carries its `place`; so does + `GET /api/v1/resources/{rid}`. `GET /api/v1/resources/map-tiles` gives the tile address and its credit. +- **Agents**: the MCP tools `list_recordings` (with `located` and `bbox`) and `get_recording` give `place` too. ### Cleaning up transcripts diff --git a/docs/requirements.md b/docs/requirements.md index 251f5f896..6e6c3a116 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -172,6 +172,7 @@ request, to another check that proves the same thing; loosening a requirement or | Transcripts made elsewhere (text, Markdown, Word, PDF, SRT, WebVTT, JSON) imported with each line credited to the speaker the file names, WebVTT voice spans (``, ``) included | [test_importing.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_importing.py) `test_formats_and_speakers_by_name`, `test_webvtt_voice_spans_name_each_line` | | Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` | | Photos and videos dated by when they were taken, from the file itself (a photo's EXIF date and GPS fix as `place`, a video's creation time) when no name or source dates them; a date already set is kept and none is invented | [test_file_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_dates.py) `test_a_photo_is_dated_and_placed_by_its_exif`, `test_a_date_already_set_is_kept`, `test_a_video_is_dated_by_its_creation_time`, `test_a_scanned_video_is_dated_by_its_creation_time` | +| Assets placed by their file's GPS fix (a photo's EXIF, a video's ISO 6709 location) on the Library's map, listed by API (`located`, `bbox`) and MCP to members only, on tiles from a setting | [test_photo_map.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photo_map.py) `test_located_assets_are_listed_inside_a_box`, `test_an_agent_finds_what_was_taken_in_a_place`, `test_a_video_is_placed_by_its_location`, `test_the_map_tiles_are_a_setting`; [library-map.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/library-map.test.tsx) `placed assets are dots on the tiles from Settings, framed to show them all` | | An uploaded video, end to end | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) | | Apache Tika (opt-in) reads documents nothing else here can | [test_tika.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_tika.py) `test_off_by_default_nothing_changes`, `test_what_nothing_here_can_convert_tika_reads_as_text` | | Web pages captured one at a time, from a link list or a bookmarks file, by API or by asking; private networks only when allowed | [test_webcapture.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webcapture.py) `test_a_web_page_becomes_a_document`, `test_an_admin_allows_a_private_network_in_settings`, `test_private_addresses_are_refused`; [test_web_links.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_links.py) `test_one_or_several_addresses_as_a_list`, `test_a_bookmarks_file_becomes_pages_to_capture`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_captures_web_pages` | diff --git a/docs/technology-selection.md b/docs/technology-selection.md index 302bd819a..354085637 100644 --- a/docs/technology-selection.md +++ b/docs/technology-selection.md @@ -19,6 +19,8 @@ - [shadcn/ui](https://ui.shadcn.com/) and [Tailwind CSS](https://tailwindcss.com/) - [BlockSuite](https://blocksuite.io/): the notes editor; [@xyflow/react](https://reactflow.dev/): the canvas pipelines and workflows are drawn on +- [Leaflet](https://leafletjs.com/) (BSD-2-Clause): the Library's map of where assets were taken, on XYZ tiles from + Settings → Maps ([OpenStreetMap](https://www.openstreetmap.org/)'s by default), loaded only when the map is opened - Processing: ffmpeg, [SenseVoice](https://github.com/FunAudioLLM/SenseVoice) / [faster-whisper](https://github.com/SYSTRAN/faster-whisper) / [mlx-whisper](https://github.com/ml-explore/mlx-examples), [SpeechBrain](https://speechbrain.github.io/) ECAPA voiceprints, optional [pyannote](https://github.com/pyannote/pyannote-audio), Tesseract / Apple Vision / RapidOCR, OpenCV YuNet + SFace, diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 9d34e639d..24ece72c0 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -682,7 +682,8 @@ def budgets_tool(ctx: Context, namespace: str | None = None) -> dict[str, Any]: "list_recordings", "List assets", "Assets (audio, video, documents, images and text) you can read, newest first by default, with filters. `query` " - "matches words in the title, the namespace or a person's name; use search for what is in them.", + "matches words in the title, the namespace or a person's name; use search for what is in them. Each says where it " + "was taken (`place`: lat, lon) when its file had a GPS fix; `located` or `bbox` keeps to those, as on a map.", Arg("namespace", "string", "only this namespace"), Arg("query", "string", "words that must all appear in the title, the namespace or a person's name", max_length=200), Arg("date_from", "string", "dated on or after this day (YYYY-MM-DD)"), @@ -697,6 +698,13 @@ def budgets_tool(ctx: Context, namespace: str | None = None) -> dict[str, Any]: "are left out where their namespace hides them)", enum=("include", "only", "exclude"), ), + Arg("located", "boolean", "only assets known to have been taken somewhere (with a place)", default=False), + Arg( + "bbox", + "string", + "only assets taken inside this box: west,south,east,north in degrees, such as 2.2,48.8,2.4,48.9 (Paris)", + max_length=100, + ), Arg("sort", "string", "order: date, title or duration; - first for descending", default="-date", enum=SORTS), Arg("limit", "integer", "how many", default=20, minimum=1, maximum=100), Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000), @@ -711,11 +719,14 @@ def list_recordings( tag: str | None, media: str | None, sensitive: str | None, + located: bool, + bbox: str | None, sort: str, limit: int, offset: int, ) -> dict[str, Any]: spaces, within = ctx.acl.scope(namespace) + box = library.bbox(bbox) if bbox else None rows, total = library.list_recordings( ctx.db, spaces, @@ -731,6 +742,8 @@ def list_recordings( tags=[tag] if tag else None, cfg=ctx.cfg, sensitive=sensitive, + located=located, + box=box, ) out: dict[str, Any] = { "total": total, @@ -747,6 +760,7 @@ def list_recordings( "tags": r.get("tags") or None, "language": r.get("language"), "nsfw": True if (r.get("nsfw") or {}).get("flagged") else None, + "place": r.get("place"), "url": ctx.link(r["id"]), } for r in rows diff --git a/fastapi_backend/app/api/v1/routes/recordings.py b/fastapi_backend/app/api/v1/routes/recordings.py index 665c37db6..c825cd935 100644 --- a/fastapi_backend/app/api/v1/routes/recordings.py +++ b/fastapi_backend/app/api/v1/routes/recordings.py @@ -51,6 +51,7 @@ EvidencePart, JobQueued, LanguageCount, + MapTiles, MediaKind, NamespaceAccess, NsfwCounts, @@ -168,6 +169,13 @@ def list_recordings( description="with `field`: text that its value contains, the number, a date it starts with (1998, 1998-05), " "true/false, or one of its options", ), + located: bool = Query(False, description="only assets known to have been taken somewhere (a GPS fix: `place`)"), + bbox: str | None = Query( + None, + max_length=100, + description="only assets taken inside this box: west,south,east,north in degrees (GeoJSON's order); west past " + "east crosses the antimeridian", + ), sort: RecordingSort = Query( "-date", description="date, title, duration, speakers (number of people), status or importance; prefix - for descending" ), @@ -194,6 +202,7 @@ def list_recordings( raise HTTPException(404, "not found") by_field = fieldmod.filter_condition(f, value) with domain_errors(): + box = library.bbox(bbox) if bbox else None rows, total = library.list_recordings( db, spaces, @@ -227,6 +236,8 @@ def list_recordings( topics=topic, content_types=content_type, emotions=emotion, + located=located, + box=box, ) response.headers["X-Total-Count"] = str(total) for r in rows: @@ -264,6 +275,14 @@ def list_languages(acl: Acl, user: CurrentUser, db: Db, ns: str | None = Query(N return [LanguageCount.model_validate(x) for x in library.language_counts(db, spaces, within)] +@router.get("/map-tiles") +def get_map_tiles(user: CurrentUser, cfg: Cfg) -> MapTiles: + """The tiles the web app draws its map of where assets were taken on (`GET /resources?located=true` lists them): + Settings → Maps, OpenStreetMap's unless an admin changed it.""" + m = cfg.get("maps") or {} + return MapTiles(url=m.get("tile_url") or None, attribution=m.get("attribution") or None) + + @router.get("/objects") def list_objects(acl: Acl, user: CurrentUser, db: Db, ns: str | None = Query(None, description="one namespace")) -> list[ObjectCount]: """The kinds of object seen in the assets you can read (or one namespace's), with how many assets each is @@ -342,8 +361,9 @@ def get_recording(rid: int, acl: Acl, db: Db, cfg: Cfg) -> Recording: d["converted_from"] = _attached_to(acl, db, r.get("converted_from")) if r.get("enrichment") and acl.member(r): # where a photo was taken: members only d["enrichment"] = {k: v for k, v in r["enrichment"].items() if k != "photo_fp"} - if r.get("place") and acl.member(r): # its file's GPS fix, likewise - d["place"] = r["place"] + place = library.place_of({"place": r.get("place"), "_gps": (r.get("enrichment") or {}).get("place")}) + if place and acl.member(r): # where it was taken, likewise + d["place"] = place d["nsfw"] = _nsfw(db, cfg, r, rid) d["enhanced"] = enhance.brief(r.get("enhanced"), rid) waiting = r.get("awaits_pdf") or None diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py index b3fb3e6d6..3a9e26f5d 100644 --- a/fastapi_backend/app/domain/ingest.py +++ b/fastapi_backend/app/domain/ingest.py @@ -164,22 +164,40 @@ def _exif(path): when = when or _taken_when(exif.get(ExifTags.Base.DateTime)) lat = _degrees(gps.get(ExifTags.GPS.GPSLatitude), gps.get(ExifTags.GPS.GPSLatitudeRef)) lon = _degrees(gps.get(ExifTags.GPS.GPSLongitude), gps.get(ExifTags.GPS.GPSLongitudeRef)) + return when, _point(lat, lon) + + +def _point(lat, lon): + """A GPS fix as Lens keeps it ({lat, lon}, to six places: about 10 cm), or None for one that isn't a place on Earth + or is 0, 0 (what a device writes when it has no fix).""" ok = lat is not None and lon is not None and math.isfinite(lat + lon) and abs(lat) <= 90 and abs(lon) <= 180 - place = {"lat": round(lat, 6), "lon": round(lon, 6)} if ok and (lat, lon) != (0, 0) else None # 0, 0: no fix - return when, place + return {"lat": round(lat, 6), "lon": round(lon, 6)} if ok and (lat, lon) != (0, 0) else None + + +ISO6709_RX = re.compile(r"^\s*([+-]\d+(?:\.\d+)?)([+-]\d+(?:\.\d+)?)") +LOCATION_TAGS = ("com.apple.quicktime.location.ISO6709", "location", "location-eng") + + +def _iso6709(s): + """Where an ISO 6709 string such as "+48.8582+002.2945+035.000/" (a phone's video) says, in degrees; None if it + isn't one. Only the decimal-degree form phones write is read.""" + m = ISO6709_RX.match(str(s or "")) + return _point(float(m.group(1)), float(m.group(2))) if m else None def taken(path): """When and where a photo, a video or a recording was taken, as its own file says: a photo's EXIF (DateTimeOriginal, else DateTime, and its GPS fix), a video's or a recording's creation time (QuickTime's creationdate, which keeps - the local time, else creation_time). (ISO 8601 or None, {lat, lon} or None): nothing is guessed. Read it before - the file is encrypted.""" + the local time, else creation_time) and its location (ISO 6709, as phones write it). (ISO 8601 or None, {lat, lon} + or None): nothing is guessed. Read it before the file is encrypted.""" ext = pathlib.Path(path).suffix.lower() if ext in store.IMAGE_EXT: return _exif(path) if ext in store.MEDIA_EXT: - tags = (_ffprobe(path, "format_tags=creation_time,com.apple.quicktime.creationdate").get("format") or {}).get("tags") or {} - return _taken_when(tags.get("com.apple.quicktime.creationdate")) or _taken_when(tags.get("creation_time")), None + names = ",".join(("creation_time", "com.apple.quicktime.creationdate", *LOCATION_TAGS)) + tags = (_ffprobe(path, f"format_tags={names}").get("format") or {}).get("tags") or {} + when = _taken_when(tags.get("com.apple.quicktime.creationdate")) or _taken_when(tags.get("creation_time")) + return when, next((p for p in map(_iso6709, (tags.get(t) for t in LOCATION_TAGS)) if p), None) return None, None diff --git a/fastapi_backend/app/domain/library.py b/fastapi_backend/app/domain/library.py index 517d72cac..fb141627c 100644 --- a/fastapi_backend/app/domain/library.py +++ b/fastapi_backend/app/domain/library.py @@ -26,8 +26,10 @@ } FIELDS = ( "record::id(id) AS id, title, recorded_at, duration_ms, status, error, source, stats, summary, space, collection, media, " - "access, access_parts, featured, tags, language, path, remote, engine, nsfw" + "access, access_parts, featured, tags, language, path, remote, engine, nsfw, place, enrichment.place AS _gps" ) +# where an asset was taken: its file's own GPS fix (ingest.taken), else the one anytopdf read from a photo (enrich.py) +LAT, LON = "(place.lat ?? enrichment.place.lat)", "(place.lon ?? enrichment.place.lon)" # where a recording came from (origin_of): a connected source (source:), or one of these ORIGINS = { "upload": "Uploaded", @@ -192,6 +194,26 @@ def check_origins(origins): raise ValueError(f"origin is one of {', '.join(ORIGINS)} or source:") +def bbox(text): + """A bounding box given as "west,south,east,north" in degrees (GeoJSON's order, RFC 7946), as those four numbers. West + past east crosses the antimeridian (170,-50,-170,-30 is the Pacific around Fiji). ValueError for anything else.""" + try: + w, s, e, n = (float(x) for x in str(text).split(",")) + except ValueError: + raise ValueError("bbox is west,south,east,north in degrees, such as 2.2,48.8,2.4,48.9") from None + if not all(-180 <= x <= 180 for x in (w, e)) or not -90 <= s <= n <= 90: + raise ValueError("bbox is west,south,east,north: longitudes from -180 to 180, latitudes from -90 to 90, south first") + return w, s, e, n + + +def place_of(rec): + """Where an asset was taken ({lat, lon}), from a row that selected FIELDS; None when it isn't known.""" + for p in (rec.get("place"), rec.get("_gps")): + if isinstance(p, dict) and p.get("lat") is not None and p.get("lon") is not None: + return {"lat": p["lat"], "lon": p["lon"]} + return None + + def in_scope(spaces, within=None): """The WHERE clause, with its parameters, for the recordings of these namespaces and of the collections in `within` ({namespace: collection ids}: namespaces someone sees only some collections of).""" @@ -256,6 +278,8 @@ def where( content_types=None, emotions=None, only=None, + located=False, + box=None, ): """The WHERE clause and its parameters for these filters. Filters combine with AND, the values of one filter with OR. @@ -273,7 +297,9 @@ def where( without it (and with cfg), flagged ones are left out where their namespace hides them. faces: face identity ids, seen in it (faces.py). entities: entity ids, mentioned in it. topics: topic ids, that it was accepted as about (topics.py). content_types: subtype keys, its content type (content_types.py). emotions: - emotions heard in it (ignoring case). only: recording ids (such as those another search found). + emotions heard in it (ignoring case). only: recording ids (such as those another search found). located: only + those known to have been taken somewhere (a GPS fix: LAT, LON). box: only those taken inside this bounding box + (west, south, east, north, as bbox() gives it). """ base, p = in_scope(spaces, within) w = [base] @@ -405,6 +431,12 @@ def where( if collections is not None: p["cols"] = sorted(collections) w.append("collection IN $cols") + if located or box: + w.append(f"{LAT} != NONE AND {LON} != NONE") + if box: + p["bw"], p["bs"], p["be"], p["bn"] = box + w.append(f"{LAT} >= $bs AND {LAT} <= $bn") + w.append(f"({LON} >= $bw {'AND' if box[0] <= box[2] else 'OR'} {LON} <= $be)") # west past east: across 180° if field is not None: cond, value = field w.append(cond) @@ -481,7 +513,8 @@ def summaries(db, rows, cfg=None): if origins: r["origin"] = came[r["id"]] r["origin_name"] = called[r["origin"]] - for k in ("_k", "_none", "access_parts", "path", "remote", "engine", "nsfw"): + r["place"] = place_of(r) + for k in ("_k", "_none", "access_parts", "path", "remote", "engine", "nsfw", "_gps"): r.pop(k, None) if flags.get(r["id"]): r["nsfw"] = flags[r["id"]] diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index a2033fa74..01abfb1fe 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -32,6 +32,7 @@ "search": None, "embeddings": None, "reports": None, + "maps": None, "workers": None, "iiif": None, "ai": None, @@ -521,6 +522,8 @@ def _check(section, key, value, default): ): raise ValueError("iiif.viewers is a list of {name, url} with an http(s) URL; the URL may use {manifest} and {content_state}") return value + if section == "maps": + return _map_setting(key, value) if (section, key) == ("transcribe", "cleanup"): from . import audio_cleanup @@ -1808,6 +1811,26 @@ def num(n): return out +TILE_URL = re.compile(r"^https?://[^\s]{1,500}$") + + +def _map_setting(key, value): + """The map's tiles: an http(s) address with {z}, {x} and {y} in it (and {s} for a subdomain, if it likes), or empty + for none; and the credit shown on the map, one line.""" + value = (value or "").strip() if isinstance(value, str) or value is None else value + if not isinstance(value, str): + raise ValueError(f"maps.{key} is text") + if key == "tile_url" and value: + u = urllib.parse.urlsplit(value) + if not (TILE_URL.match(value) and u.hostname and u.username is None and u.password is None) or not all( + f"{{{k}}}" in value for k in "zxy" + ): + raise ValueError("maps.tile_url is an http(s) address with {z}, {x} and {y} in it, or empty for no tiles") + if key == "attribution" and (len(value) > 300 or "\n" in value): + raise ValueError("maps.attribution is one line of text") + return value + + def _upload_setting(key, value): if key == "extensions": ok = isinstance(value, list) and value and all(isinstance(x, str) for x in value) diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 6dd5a337f..c351f60e2 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -608,6 +608,9 @@ def labels(): # or open up arbitrary folders on the server. Local folders can only be watched inside local_roots. "sources": {"rclone": None, "local_roots": [], "check_seconds": 15, "cache_dir": None}, "reports": {"audio": "link"}, + # the map of where assets were taken (the Library's Map): the tiles drawn under it, fetched by each viewer's browser + # from tile_url ({z}/{x}/{y}), which so learns the area looked at (threat-model.md); empty draws no tiles + "maps": {"tile_url": "https://tile.openstreetmap.org/{z}/{x}/{y}.png", "attribution": "© OpenStreetMap contributors"}, # the read-only SMB share of the file tree (smb_share.py, docs/smb.md): off until an admin turns it on, then # served by `lens worker` to people signing in with a file access key, from allowed_networks only. bind and port # are startup settings only: in Docker it listens on 0.0.0.0:4445 inside the container, published where the web diff --git a/fastapi_backend/app/schemas/recordings.py b/fastapi_backend/app/schemas/recordings.py index 8c5062437..f359c404e 100644 --- a/fastapi_backend/app/schemas/recordings.py +++ b/fastapi_backend/app/schemas/recordings.py @@ -98,6 +98,23 @@ class RecordingNsfwUpdate(RequestModel): flagged: bool | None = Field(description="true: NSFW; false: not NSFW; null: let the score decide again") +class GeoPoint(ResponseModel): + """Where an asset was taken, from its own file (a photo's EXIF GPS fix, a video's ISO 6709 location), in WGS 84 + degrees.""" + + lat: float = Field(description="latitude, north positive") + lon: float = Field(description="longitude, east positive") + + +class MapTiles(ResponseModel): + """The tiles a map of where assets were taken is drawn on (Settings → Maps).""" + + url: str | None = Field( + None, description="where each viewer's browser fetches them, with {z}, {x} and {y} (XYZ tiles); none: draw no tiles" + ) + attribution: str | None = Field(None, description="the credit the tiles' terms ask to be shown on the map") + + class RecordingSummary(ResponseModel): id: int title: str | None = None @@ -131,6 +148,9 @@ class RecordingSummary(ResponseModel): None, description="your role on it: through its namespace or its collection (an admin of the collection is an owner)" ) nsfw: NsfwState | None = Field(None, description="NSFW classification, where it's on in its namespace and it was classified") + place: GeoPoint | None = Field( + None, description="where it was taken: its file's GPS fix, or the one anytopdf read from a photo; none when not known" + ) class OriginCount(ResponseModel): @@ -301,13 +321,6 @@ class EnrichedPlace(ResponseModel): said: str | None = Field(None, description="named in the text: the words that name it") -class GeoPoint(ResponseModel): - """Where a photo was taken, from its own file (its EXIF GPS fix), in WGS 84 degrees.""" - - lat: float = Field(description="latitude, north positive") - lon: float = Field(description="longitude, east positive") - - class EnrichedDate(ResponseModel): """A date or time the text gives.""" @@ -384,7 +397,9 @@ class Recording(ResponseModel): description="when it was made: a date in its file's name, else what its file says (a photo's EXIF, a video's " "creation time), else its file's last change; a source's or a document's own date wins", ) - place: GeoPoint | None = Field(None, description="a photo: where it was taken, from its EXIF GPS (members of its namespace only)") + place: GeoPoint | None = Field( + None, description="where it was taken, from its file: a photo's EXIF GPS, a video's location (members of its namespace only)" + ) nsfw: NsfwState | None = Field(None, description="NSFW classification, where it's on in its namespace and it was classified") audio_cleanup: AudioCleanup | None = Field(None, description="how its clean_audio step cleaned up its sound before transcription") enhanced: Enhanced | None = Field(None, description="an image the enhance step looked at: what it found, and its enhanced copy") diff --git a/fastapi_backend/archive.example.yaml b/fastapi_backend/archive.example.yaml index 10934a639..01ea6d2c2 100644 --- a/fastapi_backend/archive.example.yaml +++ b/fastapi_backend/archive.example.yaml @@ -144,3 +144,7 @@ video: # everything here except the model paths can reports: audio: link # link (relative path) | embed (inline, large) | none + +maps: # the Library's map of where photos and videos were taken; can also be changed in the app + tile_url: https://tile.openstreetmap.org/{z}/{x}/{y}.png # fetched by each viewer's browser, so it sees the area looked at; empty: no tiles + attribution: © OpenStreetMap contributors # the credit the tiles' terms ask for diff --git a/fastapi_backend/tests/api/test_photo_map.py b/fastapi_backend/tests/api/test_photo_map.py new file mode 100644 index 000000000..1c1cc04b4 --- /dev/null +++ b/fastapi_backend/tests/api/test_photo_map.py @@ -0,0 +1,119 @@ +"""Assets on a map (docs/processing.md, Where an asset was taken): those whose file had a GPS fix are listed with their +`place`, inside a bounding box if asked, through the API and MCP; a video's location is read too; and the map's tiles +are a setting.""" + +from __future__ import annotations + +import pytest + +from app.domain import ingest, store +from tests.api.test_documents import _upload +from tests.api.test_file_dates import MODIFIED, _photo, _video +from tests.api.test_mcp import tool, tool_error +from tests.api.test_oauth import _bearer, _grant +from tests.helpers import login, make_user +from tests.video_helpers import needs_ffmpeg + +R = store.R +EIFFEL = {"lat": 48.858233, "lon": 2.294567} +PARIS = "2.2,48.8,2.4,48.9" + + +@pytest.fixture +def app(cfg, db): + from app.main import create_app + + return create_app(cfg, db, background=False) + + +@pytest.fixture +def env(client, db): + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + make_user(db, "root@x.io", "admin password 1", admin=True) + return {"he": login(client, "ed@x.io", "editor password 1"), "ha": login(client, "root@x.io", "admin password 1")} + + +def _note(db, cfg, title, **fields): + """An asset of 'pods' with these fields set (where it was taken).""" + rid = ingest.import_text(db, cfg, "pods", f"{title}, a note.", title=title) + if fields: + db.q("UPDATE $r MERGE $d", r=R("recording", rid), d=fields) + return rid + + +def _listed(client, h, **params): + r = client.get("/api/v1/resources", params=params, headers=h) + assert r.status_code == 200, r.text + return {x["title"]: x["place"] for x in r.json()} + + +def test_located_assets_are_listed_inside_a_box(client, env, db, cfg): + he = env["he"] + _upload(client, he, _photo(), "eiffel.jpg", modified=MODIFIED) # its EXIF GPS fix, read as it arrives + _upload(client, he, _photo(gps=False), "nowhere.jpg", modified=MODIFIED) + _note(db, cfg, "fiji", place={"lat": -17.7134, "lon": 178.065}) + _note(db, cfg, "samoa", place={"lat": -13.759, "lon": -172.1046}) + # a photo analysed with anytopdf before its file's place was kept: the GPS fix anytopdf read places it + _note(db, cfg, "tokyo", enrichment={"by": "anytopdf", "place": {"name": "Tokyo", "lat": 35.6895, "lon": 139.6917, "from": "gps"}}) + _note(db, cfg, "plain") + + every = _listed(client, he) + assert every["eiffel"] == EIFFEL and every["nowhere"] is None and every["plain"] is None + assert every["tokyo"] == {"lat": 35.6895, "lon": 139.6917} + assert set(_listed(client, he, located="true")) == {"eiffel", "fiji", "samoa", "tokyo"} + assert _listed(client, he, bbox=PARIS) == {"eiffel": EIFFEL} + # west past east crosses the antimeridian: Fiji and Samoa, not Paris or Tokyo + assert set(_listed(client, he, bbox="170,-30,-170,0")) == {"fiji", "samoa"} + # filters combine as ever: with words in the title, with a namespace + assert set(_listed(client, he, located="true", q="tok")) == {"tokyo"} + assert _listed(client, he, bbox=PARIS, ns="pods") == {"eiffel": EIFFEL} + for bad in ("2,48,3", "a,b,c,d", "2,49,3,48", "200,0,10,10"): # three numbers, words, south past north, off the Earth + r = client.get("/api/v1/resources", params={"bbox": bad}, headers=he) + assert r.status_code == 400 and "bbox" in r.text, (bad, r.text) + # one asset says it too, for its members + tokyo = db.values("SELECT VALUE record::id(id) FROM recording WHERE title = 'tokyo'")[0] + assert client.get(f"/api/v1/resources/{tokyo}", headers=he).json()["place"] == {"lat": 35.6895, "lon": 139.6917} + + +def test_an_agent_finds_what_was_taken_in_a_place(client, env, db, cfg): + _note(db, cfg, "louvre", place={"lat": 48.8606, "lon": 2.3376}) + _note(db, cfg, "rome", place={"lat": 41.9028, "lon": 12.4964}) + _note(db, cfg, "plain") + _, tokens = _grant(client, env["he"], scope="read") + got = tool(client, _bearer(tokens), "list_recordings", bbox=PARIS) + assert got["total"] == 1 and got["recordings"][0]["title"] == "louvre" + assert got["recordings"][0]["place"] == {"lat": 48.8606, "lon": 2.3376} + got = tool(client, _bearer(tokens), "list_recordings", located=True) + assert {r["title"] for r in got["recordings"]} == {"louvre", "rome"} + assert "bbox" in tool_error(client, _bearer(tokens), "list_recordings", bbox="north of here") + + +@needs_ffmpeg +def test_a_video_is_placed_by_its_location(client, env, db, folder): + # what an Android phone writes (ffmpeg's location tag), and what an iPhone does (QuickTime's ISO 6709 key) + clip = _video(folder / "clip.mp4", "location=+48.8582+002.2945/") + assert ingest.taken(clip)[1] == {"lat": 48.8582, "lon": 2.2945} + mov = _video(folder / "phone.mov", "com.apple.quicktime.location.ISO6709=-33.8568+151.2153+004.000/") + assert ingest.taken(mov)[1] == {"lat": -33.8568, "lon": 151.2153} + # no fix (0, 0) and none at all are no place + assert ingest.taken(_video(folder / "zero.mp4", "location=+00.0000+000.0000/"))[1] is None + assert ingest.taken(_video(folder / "plain.mp4"))[1] is None + rid = _upload(client, env["he"], clip.read_bytes(), "clip.mp4", modified=MODIFIED)["recording"] + assert db.one("SELECT VALUE place FROM $r", r=R("recording", rid)) == {"lat": 48.8582, "lon": 2.2945} + + +def test_the_map_tiles_are_a_setting(client, env, new_client): + he, ha = env["he"], env["ha"] + tiles = client.get("/api/v1/resources/map-tiles", headers=he).json() + assert tiles == {"url": "https://tile.openstreetmap.org/{z}/{x}/{y}.png", "attribution": "© OpenStreetMap contributors"} + assert new_client().get("/api/v1/resources/map-tiles").status_code == 401 # signed in only + # an admin points the map at their own tile server, or draws none + own = "https://tiles.home.lan/{z}/{x}/{y}.png" + assert client.put("/api/v1/settings/maps", json={"tile_url": own, "attribution": "Home"}, headers=ha).status_code == 200 + assert client.get("/api/v1/resources/map-tiles", headers=he).json() == {"url": own, "attribution": "Home"} + assert client.put("/api/v1/settings/maps", json={"tile_url": ""}, headers=ha).status_code == 200 + assert client.get("/api/v1/resources/map-tiles", headers=he).json()["url"] is None + for bad in ("https://tiles.home.lan/tile.png", "ftp://tiles.home.lan/{z}/{x}/{y}.png", "https://me:pw@t.lan/{z}/{x}/{y}.png"): + r = client.put("/api/v1/settings/maps", json={"tile_url": bad}, headers=ha) + assert r.status_code == 400 and "maps.tile_url" in r.text, bad + assert client.put("/api/v1/settings/maps", json={"tile_url": own}, headers=he).status_code == 403 # admins only diff --git a/nextjs-frontend/__tests__/library-map.test.tsx b/nextjs-frontend/__tests__/library-map.test.tsx new file mode 100644 index 000000000..eecde48c3 --- /dev/null +++ b/nextjs-frontend/__tests__/library-map.test.tsx @@ -0,0 +1,140 @@ +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { render, screen, waitFor } from "@testing-library/react"; +import "@testing-library/jest-dom"; +import type { ReactNode } from "react"; + +import { Resources } from "@/app/openapi-client"; +import type { RecordingSummary } from "@/app/openapi-client/types.gen"; +import { boundsOf, mapCount, mapQuery, placed, placeText } from "@/components/library/map-model"; +import { PlacesMap } from "@/components/library/places-map"; +import { crossErrors } from "@/components/settings/model"; + +jest.mock("@/app/openapi-client", () => ({ + Resources: { getMapTiles: jest.fn(), listRecordings: jest.fn() }, +})); +jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) })); +const push = jest.fn(); +jest.mock("next/navigation", () => ({ useRouter: () => ({ push }) })); + +// Leaflet draws into the page with SVG and canvas jsdom lacks: what the map asks of it is recorded instead +const calls: { tiles: unknown[][]; dots: unknown[][]; fit: unknown[][] } = { tiles: [], dots: [], fit: [] }; +jest.mock("leaflet", () => { + const layer = () => { + const l = { + addTo: () => l, + bindTooltip: () => l, + bindPopup: () => l, + clearLayers: () => l, + }; + return l; + }; + const map = () => { + const m = { + setView: () => m, + fitBounds: (...a: unknown[]) => calls.fit.push(a), + remove: () => undefined, + }; + return m; + }; + return { + map, + layerGroup: layer, + tileLayer: (...a: unknown[]) => (calls.tiles.push(a), layer()), + circleMarker: (...a: unknown[]) => (calls.dots.push(a), layer()), + }; +}); + +const ok = (data: unknown, total?: number) => + Promise.resolve({ + data, + response: { ok: true, status: 200, headers: new Headers(total != null ? { "x-total-count": String(total) } : {}) }, + }); +const m = (f: unknown) => f as jest.Mock; +const row = (id: number, title: string, place: { lat: number; lon: number } | null) => + ({ + id, + title, + place, + space: 1, + namespace: "pods", + media_kind: "image", + recorded_at: "2019-08-17T09:41:05", + }) as RecordingSummary; +const EIFFEL = row(1, "Eiffel Tower", { lat: 48.8582, lon: 2.2945 }); +const OPERA = row(2, "Sydney Opera House", { lat: -33.8568, lon: 151.2153 }); + +function wrap(ui: ReactNode) { + const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render({ui}); +} + +beforeEach(() => { + calls.tiles = []; + calls.dots = []; + calls.fit = []; +}); + +test("the map's model: placed rows, the box around them, the query and the words", () => { + expect(placed([EIFFEL, row(3, "Nowhere", null), OPERA]).map((r) => r.id)).toEqual([1, 2]); + expect(boundsOf([EIFFEL.place!, OPERA.place!])).toEqual([ + [-33.8568, 2.2945], + [48.8582, 151.2153], + ]); + expect(boundsOf([])).toBeNull(); + // the Library's filters, only assets with a place, as many as the map draws + expect(mapQuery({ ns: "pods", q: "tower", sort: "-date" })).toEqual({ + ns: "pods", + q: "tower", + sort: "-date", + located: true, + limit: 1000, + offset: 0, + }); + expect(placeText(OPERA.place!)).toBe("33.8568° S, 151.2153° E"); + expect(mapCount(2, 2)).toBe("2 placed assets"); + expect(mapCount(1000, 1500)).toBe("The newest 1,000 of 1,500 placed assets · narrow the filters to see the rest"); + expect(mapCount(0, 0)).toMatch(/No assets/); +}); + +test("placed assets are dots on the tiles from Settings, framed to show them all", async () => { + m(Resources.getMapTiles).mockReturnValue( + ok({ url: "https://tile.openstreetmap.org/{z}/{x}/{y}.png", attribution: "© OpenStreetMap " }), + ); + m(Resources.listRecordings).mockReturnValue(ok([EIFFEL, OPERA], 2)); + wrap(); + await waitFor(() => expect(calls.dots).toHaveLength(2)); + expect(m(Resources.listRecordings).mock.calls[0][0].query).toMatchObject({ ns: "pods", located: true, limit: 1000 }); + // the tile server's address, with the credit as text (Leaflet reads it as HTML) + expect(calls.tiles[0][0]).toBe("https://tile.openstreetmap.org/{z}/{x}/{y}.png"); + expect(calls.tiles[0][1]).toMatchObject({ attribution: "© OpenStreetMap <contributors>" }); + expect(calls.dots.map((d) => d[0])).toEqual([ + [48.8582, 2.2945], + [-33.8568, 151.2153], + ]); + expect(calls.fit[0][0]).toEqual([ + [-33.8568, 2.2945], + [48.8582, 151.2153], + ]); + expect(screen.getByText(/2 placed assets/)).toHaveTextContent( + "Map tiles from tile.openstreetmap.org, which sees the area you look at", + ); + // each one, for a screen reader, as a link to it + expect(screen.getByRole("link", { name: /Eiffel Tower, 48.8582° N/ })).toHaveAttribute("href", "/resources/1"); +}); + +test("with no tiles set only the dots are drawn, and nothing placed says why", async () => { + m(Resources.getMapTiles).mockReturnValue(ok({ url: null, attribution: null })); + m(Resources.listRecordings).mockReturnValue(ok([], 0)); + wrap(); + expect(await screen.findByText(/None of these assets say where they were taken/)).toBeInTheDocument(); + expect(screen.getByText(/No map tiles: Settings → Maps/)).toBeInTheDocument(); + await waitFor(() => expect(calls.fit).toHaveLength(0)); + expect(calls.tiles).toHaveLength(0); +}); + +test("Settings → Maps takes a tile address with {z}, {x} and {y}, or none", () => { + expect(crossErrors({ "maps.tile_url": "https://tiles.home.lan/{z}/{x}/{y}.png" })["maps.tile_url"]).toBeUndefined(); + expect(crossErrors({ "maps.tile_url": null })["maps.tile_url"]).toBeUndefined(); + expect(crossErrors({ "maps.tile_url": "https://tiles.home.lan/tile.png" })["maps.tile_url"]).toMatch(/\{z\}/); + expect(crossErrors({ "maps.tile_url": "tiles.home.lan/{z}/{x}/{y}" })["maps.tile_url"]).toMatch(/http/); +}); diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index aa5b52e48..906887a94 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -1510,6 +1510,9 @@ export type { GetJobLogResponses, GetJobResponse, GetJobResponses, + GetMapTilesData, + GetMapTilesResponse, + GetMapTilesResponses, GetMediaData, GetMediaError, GetMediaErrors, @@ -2570,6 +2573,7 @@ export type { MakeSigninLinkResponses, ManifestCheck, ManifestChecked, + MapTiles, MarkActivitiesData, MarkActivitiesError, MarkActivitiesErrors, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 3fe58460f..8a47cb19b 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -711,6 +711,8 @@ import type { GetJobLogErrors, GetJobLogResponses, GetJobResponses, + GetMapTilesData, + GetMapTilesResponses, GetMediaData, GetMediaErrors, GetMediaResponses, @@ -6044,6 +6046,21 @@ export class Resources { }); } + /** + * Get Map Tiles + * + * The tiles the web app draws its map of where assets were taken on (`GET /resources?located=true` lists them): + * Settings → Maps, OpenStreetMap's unless an admin changed it. + */ + public static getMapTiles( + options?: Options, + ): RequestResult { + return (options?.client ?? client).get({ + url: "/api/v1/resources/map-tiles", + ...options, + }); + } + /** * List Objects * diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 9804fa9f5..a798a6f02 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -9810,6 +9810,10 @@ export type FindItem = { * NSFW classification, where it's on in its namespace and it was classified */ nsfw?: NsfwState | null; + /** + * where it was taken: its file's GPS fix, or the one anytopdf read from a photo; none when not known + */ + place?: GeoPoint | null; /** * Why * @@ -10137,7 +10141,8 @@ export type ForgotPasswordRequest = { /** * GeoPoint * - * Where a photo was taken, from its own file (its EXIF GPS fix), in WGS 84 degrees. + * Where an asset was taken, from its own file (a photo's EXIF GPS fix, a video's ISO 6709 location), in WGS 84 + * degrees. */ export type GeoPoint = { /** @@ -13644,6 +13649,27 @@ export type ManifestChecked = { [key: string]: unknown; }; +/** + * MapTiles + * + * The tiles a map of where assets were taken is drawn on (Settings → Maps). + */ +export type MapTiles = { + /** + * Url + * + * where each viewer's browser fetches them, with {z}, {x} and {y} (XYZ tiles); none: draw no tiles + */ + url?: string | null; + /** + * Attribution + * + * the credit the tiles' terms ask to be shown on the map + */ + attribution?: string | null; + [key: string]: unknown; +}; + /** * MarkIn * @@ -21585,7 +21611,7 @@ export type Recording = { */ recorded_at?: string | null; /** - * a photo: where it was taken, from its EXIF GPS (members of its namespace only) + * where it was taken, from its file: a photo's EXIF GPS, a video's location (members of its namespace only) */ place?: GeoPoint | null; /** @@ -22043,6 +22069,10 @@ export type RecordingSummary = { * NSFW classification, where it's on in its namespace and it was classified */ nsfw?: NsfwState | null; + /** + * where it was taken: its file's GPS fix, or the one anytopdf read from a photo; none when not known + */ + place?: GeoPoint | null; [key: string]: unknown; }; @@ -38059,6 +38089,18 @@ export type ListRecordingsData = { * with `field`: text that its value contains, the number, a date it starts with (1998, 1998-05), true/false, or one of its options */ value?: string | null; + /** + * Located + * + * only assets known to have been taken somewhere (a GPS fix: `place`) + */ + located?: boolean; + /** + * Bbox + * + * only assets taken inside this box: west,south,east,north in degrees (GeoJSON's order); west past east crosses the antimeridian + */ + bbox?: string | null; /** * Sort * @@ -38242,6 +38284,22 @@ export type ListLanguagesResponses = { export type ListLanguagesResponse = ListLanguagesResponses[keyof ListLanguagesResponses]; +export type GetMapTilesData = { + body?: never; + path?: never; + query?: never; + url: "/api/v1/resources/map-tiles"; +}; + +export type GetMapTilesResponses = { + /** + * Successful Response + */ + 200: MapTiles; +}; + +export type GetMapTilesResponse = GetMapTilesResponses[keyof GetMapTilesResponses]; + export type ListObjectsData = { body?: never; path?: never; diff --git a/nextjs-frontend/components/library/library-screen.tsx b/nextjs-frontend/components/library/library-screen.tsx index 3214d4c32..2528bd7f1 100644 --- a/nextjs-frontend/components/library/library-screen.tsx +++ b/nextjs-frontend/components/library/library-screen.tsx @@ -1,6 +1,6 @@ "use client"; -import { ArrowUp, Eye, List, Table2, Upload } from "lucide-react"; +import { ArrowUp, Eye, List, Map as MapIcon, Table2, Upload } from "lucide-react"; import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import type { RecordingSummary, SavedView } from "@/app/openapi-client/types.gen"; @@ -14,6 +14,7 @@ import { CollectionsDialog, PlaceDialog } from "@/components/library/collections import { FiltersBar } from "@/components/library/filters-bar"; import { LibraryEmpty } from "@/components/library/library-empty"; import { LibraryTabs } from "@/components/library/library-tabs"; +import { PlacesMap } from "@/components/library/places-map"; import { NO_FILTERS, activeFilterCount, @@ -62,6 +63,7 @@ import { atLeast } from "@/lib/roles"; import { cn } from "@/lib/utils"; const VIEW_KEY = "lens.library.view"; +type Layout = "table" | "list" | "map"; /** A value that changes at most once per `ms`: live progress is announced politely, not on every poll. */ function useThrottled(value: T, ms: number): T { @@ -99,7 +101,9 @@ export function LibraryScreen({ initial, }: { initial?: { namespace: string; collection: number | null; sensitive?: SensitiveFilter } } = {}) { const { namespace, namespaces, isPartial, roleIn, can, me, setNamespace } = useArchive(); - const [layout, setLayout] = useState<"table" | "list">("table"); + // the rows as a table or a list, or where they were taken on a map; a phone shows cards or the map + const [layout, setLayout] = useState("table"); + const rowsLayout = useRef<"table" | "list">("table"); const [filters, setFilters] = useState(NO_FILTERS); const [view, setView] = useState("all"); const [sort, setSort] = useState<{ key: SortKey; dir: SortDir }>({ @@ -142,13 +146,17 @@ export function LibraryScreen({ useEffect(() => { try { const v = localStorage.getItem(VIEW_KEY); - if (v === "list" || v === "table") setLayout(v); + if (v === "list" || v === "table") rowsLayout.current = v; + if (v === "list" || v === "table" || v === "map") setLayout(v); } catch { /* storage unavailable */ } }, []); - const chooseLayout = (v: string) => { - setLayout(v as "table" | "list"); + const chooseLayout = (picked: string) => { + // a phone's List: the table or list last chosen on a wider screen + const v = (picked === "rows" ? rowsLayout.current : picked) as Layout; + if (v !== "map") rowsLayout.current = v; + setLayout(v); try { localStorage.setItem(VIEW_KEY, v); } catch { @@ -452,16 +460,20 @@ export function LibraryScreen({ {announced} - {!narrow && ( - }, - { value: "list", label: "List", icon: }, - ]} - /> - )} + }] + : [ + { value: "table", label: "Table", icon: }, + { value: "list", label: "List", icon: }, + ]), + { value: "map", label: "Map", icon: }, + ]} + /> {isPartial(namespace) && ( @@ -622,6 +634,8 @@ export function LibraryScreen({ > Try fewer filters. + ) : layout === "map" ? ( + ) : narrow ? ( ) : layout === "table" ? ( @@ -632,7 +646,7 @@ export function LibraryScreen({ )} - {loaded && !empty && matching > 0 && (lib.recordings.hasNextPage || filtering) && ( + {loaded && !empty && layout !== "map" && matching > 0 && (lib.recordings.hasNextPage || filtering) && (
{filtering diff --git a/nextjs-frontend/components/library/map-model.ts b/nextjs-frontend/components/library/map-model.ts new file mode 100644 index 000000000..55375d6ea --- /dev/null +++ b/nextjs-frontend/components/library/map-model.ts @@ -0,0 +1,48 @@ +/** + * The Library's Map: which assets go on it (those whose file said where they were taken, `place`), the box that frames + * them, and the query that asks for them with the Library's own filters. + */ +import type { GeoPoint, RecordingSummary } from "@/app/openapi-client/types.gen"; +import type { LibraryQuery } from "@/components/library/model"; + +/** Assets drawn on the map at most: the newest this many that match. */ +export const MAP_LIMIT = 1000; + +export type Placed = RecordingSummary & { place: GeoPoint }; + +/** The list endpoint's query for the map: the Library's filters, sort and namespace, only assets with a place. */ +export function mapQuery(q: LibraryQuery): LibraryQuery { + return { ...q, located: true, limit: MAP_LIMIT, offset: 0 }; +} + +/** The rows that have a place, in the order given. */ +export function placed(rows: readonly RecordingSummary[]): Placed[] { + return rows.filter( + (r): r is Placed => r.place != null && Number.isFinite(r.place.lat) && Number.isFinite(r.place.lon), + ); +} + +/** The south-west and north-east corners ([lat, lon]) of the box around these points; null for none. */ +export function boundsOf(points: readonly GeoPoint[]): [[number, number], [number, number]] | null { + if (!points.length) return null; + const lats = points.map((p) => p.lat); + const lons = points.map((p) => p.lon); + return [ + [Math.min(...lats), Math.min(...lons)], + [Math.max(...lats), Math.max(...lons)], + ]; +} + +/** "48.8582° N, 2.2945° E": a place as people read it, to four places (about 10 m). */ +export function placeText(p: GeoPoint): string { + const f = (v: number, pos: string, neg: string) => `${Math.abs(v).toFixed(4)}° ${v < 0 ? neg : pos}`; + return `${f(p.lat, "N", "S")}, ${f(p.lon, "E", "W")}`; +} + +/** What the count line under the map says: how many are drawn, and how many more match when the cap cut them. */ +export function mapCount(shown: number, total: number): string { + const n = (x: number) => x.toLocaleString("en"); + if (!total) return "No assets here say where they were taken"; + if (total > shown) return `The newest ${n(shown)} of ${n(total)} placed assets · narrow the filters to see the rest`; + return total === 1 ? "1 placed asset" : `${n(total)} placed assets`; +} diff --git a/nextjs-frontend/components/library/places-map.tsx b/nextjs-frontend/components/library/places-map.tsx new file mode 100644 index 000000000..55eec32f5 --- /dev/null +++ b/nextjs-frontend/components/library/places-map.tsx @@ -0,0 +1,187 @@ +"use client"; + +import "leaflet/dist/leaflet.css"; + +import { keepPreviousData, useQuery } from "@tanstack/react-query"; +import type * as Leaflet from "leaflet"; +import { MapPinOff } from "lucide-react"; +import { useRouter } from "next/navigation"; +import { useEffect, useMemo, useRef, useState } from "react"; + +import { Resources } from "@/app/openapi-client"; +import type { RecordingSummary } from "@/app/openapi-client/types.gen"; +import { boundsOf, mapCount, mapQuery, placed, placeText, type Placed } from "@/components/library/map-model"; +import type { LibraryQuery } from "@/components/library/model"; +import { Button } from "@/components/ui/button"; +import { EmptyState, Skeleton } from "@/components/ui/states"; +import { data, page, useApiClient } from "@/lib/api/browser"; +import { shortDate } from "@/lib/format"; + +/** Text for Leaflet's attribution, which it reads as HTML: the admin's words, shown as words. */ +function escapeHtml(s: string): string { + return s.replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`); +} + +/** The tile server's host, which sees the area looked at (threat-model.md). */ +function tileHost(url: string): string { + try { + return new URL(url.replace(/\{[a-z]+\}/g, "a")).host; + } catch { + return url; + } +} + +/** A dot's popup: the asset's title (a link to it), its date and where. Built as nodes, so a title stays text. */ +function popup(r: Placed, open: (r: Placed) => void): HTMLElement { + const box = document.createElement("div"); + const a = document.createElement("a"); + a.href = `/resources/${r.id}`; + a.textContent = r.title || "Untitled"; + a.style.fontWeight = "700"; // Leaflet's popups are light in either theme, with colours of their own + a.addEventListener("click", (e) => { + if (e.metaKey || e.ctrlKey || e.shiftKey || e.button !== 0) return; + e.preventDefault(); + open(r); + }); + const sub = document.createElement("div"); + sub.style.color = "#555"; + sub.textContent = [r.recorded_at ? shortDate(r.recorded_at) : null, r.namespace, placeText(r.place)] + .filter(Boolean) + .join(" · "); + box.append(a, sub); + return box; +} + +/** + * The Library as a map: each asset that says where it was taken (a photo's EXIF GPS fix, a video's location) as a dot, + * with the Library's filters, framed to show them all. The tiles under it come from Settings → Maps; the browser + * fetches them, so the tile server sees the area looked at, which the line under the map says. + */ +export function PlacesMap({ query, onOpen }: { query: LibraryQuery; onOpen?: (r: RecordingSummary) => void }) { + const client = useApiClient(); + const router = useRouter(); + const tiles = useQuery({ + queryKey: ["map-tiles"], + queryFn: () => data(Resources.getMapTiles({ client })), + staleTime: 5 * 60_000, + }); + const q = useMemo(() => mapQuery(query), [query]); + const found = useQuery({ + queryKey: ["recordings", "map", q], + queryFn: () => page(Resources.listRecordings({ client, query: q })), + placeholderData: keepPreviousData, + }); + const points = useMemo(() => placed(found.data?.items ?? []), [found.data]); + const el = useRef(null); + const lib = useRef<{ L: typeof Leaflet; map: Leaflet.Map; dots: Leaflet.LayerGroup } | null>(null); + const [ready, setReady] = useState(false); + // a popup's link opens the asset in the app (the popups are made once per dot, so they call the latest of this) + const openRef = useRef<(r: Placed) => void>(() => {}); + useEffect(() => { + openRef.current = (r: Placed) => { + onOpen?.(r); + router.push(`/resources/${r.id}`); + }; + }, [onOpen, router]); + + const url = tiles.data?.url ?? null; + const attribution = tiles.data?.attribution ?? null; + // Leaflet reads `window` as it loads, so it's loaded here, in the browser, once the tiles are known + useEffect(() => { + if (!tiles.isSuccess || !el.current) return; + let gone = false; + let made: Leaflet.Map | null = null; + void import("leaflet").then((mod) => { + const L = (mod as { default?: typeof Leaflet }).default ?? (mod as typeof Leaflet); + if (gone || !el.current) return; + made = L.map(el.current, { worldCopyJump: true }).setView([20, 0], 2); + if (url) + L.tileLayer(url, { maxZoom: 19, attribution: attribution ? escapeHtml(attribution) : undefined }).addTo(made); + lib.current = { L, map: made, dots: L.layerGroup().addTo(made) }; + setReady(true); + }); + return () => { + gone = true; + made?.remove(); + lib.current = null; + setReady(false); + }; + }, [tiles.isSuccess, url, attribution]); + + // The dots, and the view framed around them, each time the filters bring others + useEffect(() => { + const m = lib.current; + if (!ready || !m) return; + m.dots.clearLayers(); + for (const r of points) { + m.L.circleMarker([r.place.lat, r.place.lon], { + radius: 7, + weight: 2, + fillOpacity: 0.9, + className: "fill-blue stroke-background", + }) + .bindTooltip(r.title || "Untitled") + .bindPopup(() => popup(r, (x) => openRef.current(x))) + .addTo(m.dots); + } + const b = boundsOf(points.map((r) => r.place)); + if (b) m.map.fitBounds(b, { padding: [32, 32], maxZoom: 15 }); + }, [ready, points]); + + if (found.isError || tiles.isError) { + return ( + { + void tiles.refetch(); + void found.refetch(); + }} + > + Try again + + } + > + {((found.error ?? tiles.error) as Error).message} + + ); + } + const total = found.data?.total ?? 0; + return ( +
+
+ {!tiles.isSuccess && } +
+ {found.isSuccess && !total && ( +
+
+ + + None of these assets say where they were taken. Photos with a GPS fix in their EXIF, and videos with a + location, show up here as they arrive. + +
+
+ )} +
+

+ {found.isSuccess ? mapCount(points.length, total) : "Loading…"} + {" · "} + {url ? `Map tiles from ${tileHost(url)}, which sees the area you look at` : "No map tiles: Settings → Maps"} +

+ +
+ ); +} diff --git a/nextjs-frontend/components/settings/bodies.tsx b/nextjs-frontend/components/settings/bodies.tsx index 33ebb2ff9..9aff71e5c 100644 --- a/nextjs-frontend/components/settings/bodies.tsx +++ b/nextjs-frontend/components/settings/bodies.tsx @@ -328,6 +328,13 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) {
); + case "maps": + return ( + <> + + + + ); case "video": return ( <> diff --git a/nextjs-frontend/components/settings/find.ts b/nextjs-frontend/components/settings/find.ts index cd830b59e..5b75c5d48 100644 --- a/nextjs-frontend/components/settings/find.ts +++ b/nextjs-frontend/components/settings/find.ts @@ -16,7 +16,10 @@ import { export const SETTINGS_GROUPS: { label: string; sections: SectionId[] }[] = [ { label: "AI and search", sections: ["llm", "local-model", "ai", "sandbox", "search", "analysis"] }, { label: "Speech and voices", sections: ["transcription", "speaker-separation", "voice-ids", "speech-providers"] }, - { label: "Media and documents", sections: ["uploads", "documents", "video", "nsfw", "enhance", "reports", "iiif"] }, + { + label: "Media and documents", + sections: ["uploads", "documents", "video", "nsfw", "enhance", "reports", "maps", "iiif"], + }, { label: "Chat and email", sections: ["telegram", "slack", "bridge", "mailbox", "mail", "notifications"] }, { label: "Security and access", diff --git a/nextjs-frontend/components/settings/model.ts b/nextjs-frontend/components/settings/model.ts index 51541f5db..2fd8c1511 100644 --- a/nextjs-frontend/components/settings/model.ts +++ b/nextjs-frontend/components/settings/model.ts @@ -70,6 +70,7 @@ export type SectionId = | "sandbox" | "search" | "reports" + | "maps" | "video" | "nsfw" | "enhance" @@ -198,6 +199,13 @@ export const SECTIONS: SectionSpec[] = [ backend: ["reports", "graph"], description: "What reports carry, and how big the knowledge graph gets.", }, + { + id: "maps", + label: "Maps", + backend: ["maps"], + description: + "The tiles under the Library’s map of where photos and videos were taken. Each viewer’s browser fetches them, so the tile server sees the area looked at.", + }, { id: "video", label: "Video, OCR, faces and objects", @@ -4140,6 +4148,25 @@ export const FIELDS: FieldSpec[] = [ mono: true, hint: "One per line, like 192.168.1.0/24: web pages, calendar feeds and videos there can be imported. Public addresses always can, and networks set on the server stay", }, + // Maps + { + section: "maps", + key: "tile_url", + label: "Map tiles", + kind: "text", + mono: true, + nullable: true, + placeholder: "No tiles: only the dots", + hint: "An address with {z}, {x} and {y}, such as your own tile server’s. It sees the area people look at. Empty draws no tiles", + }, + { + section: "maps", + key: "attribution", + label: "Credit shown on the map", + kind: "text", + nullable: true, + hint: "What the tiles’ terms ask to be shown, such as © OpenStreetMap contributors", + }, // IIIF & metadata { section: "iiif", @@ -4505,6 +4532,9 @@ export function crossErrors(values: Record): Record tiles.includes(k)))) + e["maps.tile_url"] = "Use an http(s) address with {z}, {x} and {y}, or leave it empty for no tiles"; const appUrl = values["notifications.app_url"] as string | null | undefined; if (appUrl && !URL_RX.test(appUrl)) e["notifications.app_url"] = "Use an http(s) address"; const nets = values["notifications.networks"] as string[] | undefined; diff --git a/nextjs-frontend/components/settings/settings-shell.tsx b/nextjs-frontend/components/settings/settings-shell.tsx index 4841e0267..2a3e04bb1 100644 --- a/nextjs-frontend/components/settings/settings-shell.tsx +++ b/nextjs-frontend/components/settings/settings-shell.tsx @@ -46,6 +46,7 @@ import { PackageCheck, RadioTower, Lock, + Map as MapIcon, RefreshCw, Network, ScrollText, @@ -92,6 +93,7 @@ const ICON: Record = { sandbox: SquareTerminal, search: Search, reports: ChartNoAxesColumn, + maps: MapIcon, video: Clapperboard, nsfw: EyeOff, enhance: WandSparkles, diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 485046418..dfeae5b12 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -10230,6 +10230,37 @@ }, "description": "with `field`: text that its value contains, the number, a date it starts with (1998, 1998-05), true/false, or one of its options" }, + { + "name": "located", + "in": "query", + "required": false, + "schema": { + "type": "boolean", + "description": "only assets known to have been taken somewhere (a GPS fix: `place`)", + "default": false, + "title": "Located" + }, + "description": "only assets known to have been taken somewhere (a GPS fix: `place`)" + }, + { + "name": "bbox", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "maxLength": 100 + }, + { + "type": "null" + } + ], + "description": "only assets taken inside this box: west,south,east,north in degrees (GeoJSON's order); west past east crosses the antimeridian", + "title": "Bbox" + }, + "description": "only assets taken inside this box: west,south,east,north in degrees (GeoJSON's order); west past east crosses the antimeridian" + }, { "name": "sort", "in": "query", @@ -10536,6 +10567,28 @@ } } }, + "/api/v1/resources/map-tiles": { + "get": { + "tags": [ + "resources" + ], + "summary": "Get Map Tiles", + "description": "The tiles the web app draws its map of where assets were taken on (`GET /resources?located=true` lists them):\nSettings \u2192 Maps, OpenStreetMap's unless an admin changed it.", + "operationId": "get_map_tiles", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/MapTiles" + } + } + } + } + } + } + }, "/api/v1/resources/objects": { "get": { "tags": [ @@ -55075,6 +55128,17 @@ ], "description": "NSFW classification, where it's on in its namespace and it was classified" }, + "place": { + "anyOf": [ + { + "$ref": "#/components/schemas/GeoPoint" + }, + { + "type": "null" + } + ], + "description": "where it was taken: its file's GPS fix, or the one anytopdf read from a photo; none when not known" + }, "why": { "items": { "$ref": "#/components/schemas/FindReason" @@ -55810,7 +55874,7 @@ "lon" ], "title": "GeoPoint", - "description": "Where a photo was taken, from its own file (its EXIF GPS fix), in WGS 84 degrees." + "description": "Where an asset was taken, from its own file (a photo's EXIF GPS fix, a video's ISO 6709 location), in WGS 84\ndegrees." }, "GoogleClient": { "properties": { @@ -61752,6 +61816,38 @@ ], "title": "ManifestChecked" }, + "MapTiles": { + "properties": { + "url": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Url", + "description": "where each viewer's browser fetches them, with {z}, {x} and {y} (XYZ tiles); none: draw no tiles" + }, + "attribution": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Attribution", + "description": "the credit the tiles' terms ask to be shown on the map" + } + }, + "additionalProperties": true, + "type": "object", + "title": "MapTiles", + "description": "The tiles a map of where assets were taken is drawn on (Settings \u2192 Maps)." + }, "MarkIn": { "properties": { "reviewed": { @@ -75183,7 +75279,7 @@ "type": "null" } ], - "description": "a photo: where it was taken, from its EXIF GPS (members of its namespace only)" + "description": "where it was taken, from its file: a photo's EXIF GPS, a video's location (members of its namespace only)" }, "nsfw": { "anyOf": [ @@ -75939,6 +76035,17 @@ } ], "description": "NSFW classification, where it's on in its namespace and it was classified" + }, + "place": { + "anyOf": [ + { + "$ref": "#/components/schemas/GeoPoint" + }, + { + "type": "null" + } + ], + "description": "where it was taken: its file's GPS fix, or the one anytopdf read from a photo; none when not known" } }, "additionalProperties": true, diff --git a/nextjs-frontend/package.json b/nextjs-frontend/package.json index f9871ff6e..b7085cd92 100644 --- a/nextjs-frontend/package.json +++ b/nextjs-frontend/package.json @@ -37,6 +37,7 @@ "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "dotenv": "^18.0.7", + "leaflet": "^1.9.4", "lucide-react": "^1.57.0", "next": "16.4.0", "next-auth": "5.0.0-beta.32", @@ -59,6 +60,7 @@ "@testing-library/jest-dom": "^7.0.1", "@testing-library/react": "^16.3.3", "@types/jest": "^30.0.0", + "@types/leaflet": "^1.9.22", "@types/node": "^22", "@types/nprogress": "^0.2.3", "@types/react": "19.3.0", diff --git a/nextjs-frontend/pnpm-lock.yaml b/nextjs-frontend/pnpm-lock.yaml index 99163e6f2..d9878928d 100644 --- a/nextjs-frontend/pnpm-lock.yaml +++ b/nextjs-frontend/pnpm-lock.yaml @@ -101,6 +101,9 @@ importers: dotenv: specifier: ^18.0.7 version: 18.0.7 + leaflet: + specifier: ^1.9.4 + version: 1.9.4 lucide-react: specifier: ^1.57.0 version: 1.57.0(@types/react@19.3.0)(react@19.3.0) @@ -162,6 +165,9 @@ importers: '@types/jest': specifier: ^30.0.0 version: 30.0.0 + '@types/leaflet': + specifier: ^1.9.22 + version: 1.9.22 '@types/node': specifier: ^22 version: 22.20.5 @@ -1758,6 +1764,9 @@ packages: resolution: {integrity: sha512-FdKaw3dPgcCvqU9w8SGgqx8PrxFF2EKySRhEj1nc45HNe/2cBLff3pF52C67hzeUbNhq49t7XRIIibBd41vcIg==} deprecated: This is a stub types definition. flexsearch provides its own type definitions, so you do not need this installed. + '@types/geojson@7946.0.16': + resolution: {integrity: sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg==} + '@types/hast@3.0.5': resolution: {integrity: sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==} @@ -1785,6 +1794,9 @@ packages: '@types/katex@0.16.8': resolution: {integrity: sha512-trgaNyfU+Xh2Tc+ABIb44a5AYUpicB3uwirOioeOkNPPbmgRNtcWyDeeFRzjPZENO9Vq8gvVqfhaaXWLlevVwg==} + '@types/leaflet@1.9.22': + resolution: {integrity: sha512-h3lhECYEKDasG7LFHu+GiHqAvsgLuQvlJvVZzJDGONo3sEL+wUOqSFLnwkZlK0qVxnxbuGFW8iBlJNYs5wgndA==} + '@types/lodash.ismatch@4.4.9': resolution: {integrity: sha512-qWihnStOPKH8urljLGm6ZOEdN/5Bt4vxKR81tL3L4ArUNLvcf9RW3QSnPs21eix5BiqioSWq4aAXD4Iep+d0fw==} @@ -3516,6 +3528,9 @@ packages: resolution: {integrity: sha512-MbjN408fEndfiQXbFQ1vnd+1NoLDsnQW41410oQBXiyXDMYH5z505juWa4KUE1LqxRC7DgOgZDbKLxHIwm27hA==} engines: {node: '>=0.10'} + leaflet@1.9.4: + resolution: {integrity: sha512-nxS1ynzJOmOlHp+iL3FyWqK89GtNL8U8rvlMOsQdTTssxZwCXh8N2NB3GDQOL+YR3XnWyZAxwQixURb+FA74PA==} + leven@3.1.0: resolution: {integrity: sha512-qsda+H8jTaUaN/x5vzW2rzc+8Rw4TAQ/4KjB46IwK5VH+IlVeeeje/EoZRpiXvIqjFgK84QffqPztGI3VBLG1A==} engines: {node: '>=6'} @@ -6857,6 +6872,8 @@ snapshots: dependencies: flexsearch: 0.7.43 + '@types/geojson@7946.0.16': {} + '@types/hast@3.0.5': dependencies: '@types/unist': 3.0.3 @@ -6888,6 +6905,10 @@ snapshots: '@types/katex@0.16.8': {} + '@types/leaflet@1.9.22': + dependencies: + '@types/geojson': 7946.0.16 + '@types/lodash.ismatch@4.4.9': dependencies: '@types/lodash': 4.17.25 @@ -8993,6 +9014,8 @@ snapshots: dependencies: language-subtag-registry: 0.3.23 + leaflet@1.9.4: {} + leven@3.1.0: {} levn@0.4.1: diff --git a/threat-model.json b/threat-model.json index 4f478c851..35af3b0bb 100644 --- a/threat-model.json +++ b/threat-model.json @@ -58,7 +58,8 @@ "lens help tool", "microsandbox program requests", "POST /circles and /circles/shares, MCP save_circle and share_with_circle, the assistant's and automations' share_with_circle (circles and sharing with them)", - "POST /circles/when-free, MCP when_free, the assistant's when_free (when people in your circles are free)" + "POST /circles/when-free, MCP when_free, the assistant's when_free (when people in your circles are free)", + "the Library's map: GET /resources?located=true&bbox=..., GET /resources/map-tiles, MCP list_recordings located and bbox, maps.tile_url" ], "touches": [ "filesystem" @@ -481,6 +482,15 @@ "provenance": "documented", "source": "maintainer ruling, 2026-10" }, + { + "entry_point": "the Library's map: GET /resources?located=true&bbox=..., GET /resources/map-tiles, MCP list_recordings located and bbox, maps.tile_url", + "parameter": "a bounding box (four numbers); the tile address and its credit (admins)", + "attacker_controllable": "yes", + "component": "authz", + "caller_must_enforce": "none; a place is listed only to members of its asset's namespace or collection, as GET /resources/{rid} already gives it; tiles are fetched by each viewer's browser, never by Lens, so the tile server sees the area looked at (N-MAP-TILES); Leaflet shows the credit as HTML, so the web app escapes it", + "provenance": "documented", + "source": "docs/processing.md \"the tile server sees the area being looked at\"" + }, { "entry_point": "the assistant's web_search and web_read", "parameter": "a search query and a page address, both chosen by the model; the results and pages that come back", @@ -2011,6 +2021,13 @@ "provenance": "documented", "source": "docs/nsfw.md \"display preferences, not access control\"" }, + { + "property": "N-MAP-TILES", + "reason": "privacy from the map's tile server: each member's browser fetches the Library map's tiles from maps.tile_url (OpenStreetMap's by default), so the tile server learns the tiles asked for (the area around where their assets were taken), the viewer's address and Lens's origin (Referer); no asset ids, titles or coordinates go to it. An empty maps.tile_url fetches nothing; a tile server of the operator's own keeps it on their hardware", + "false_friend": true, + "provenance": "documented", + "source": "docs/configuration.md \"The tile server sees which tiles are asked for\"" + }, { "property": "N-RDF-XML", "reason": "XML parsing in RDF import; RDF/XML is refused, so XXE and billion-laughs do not apply there; RDF import only, not docx", diff --git a/threat-model.md b/threat-model.md index d1e5467d0..32aef7a4d 100644 --- a/threat-model.md +++ b/threat-model.md @@ -213,6 +213,7 @@ install, no personal content) are off by default and there is no update check *( | Default outbound calls (exceptions to the rule above) | `components.auto` (on by default) runs pip installs and fetches hashed model files; llama.cpp and cloudflared come from GitHub "latest" when those features are on; `hf:` models from Hugging Face | *(maintainer, 2026-10)* | | Other outbound calls | none found in Lens code outside the configured destinations | *(assumption, Q19)* | | Optional ML extras | some download weights on first use on their own (Hugging Face, ModelScope) | *(inferred, Q19)* | +| Calls from the browser | the Library's Map fetches its tiles from `maps.tile_url` (OpenStreetMap's by default) while a member has it open (N-MAP-TILES) | *(documented, docs/configuration.md "The tile server sees which tiles are asked for")* | | Env proxies | plain `urllib` callers honour `HTTP(S)_PROXY`; the guarded openers ignore it | *(assumption, Q19)* | | Child processes | rclone, ffmpeg/ffprobe, LibreOffice, Chromium, Tesseract, poppler tools, antiword/catdoc, cloudflared, llama-server, `unshare` + Python for code tools, uv/pip; all as argument lists | *(assumption, Q20)* | | Listening sockets | API and web app; a `netguard` proxy on loopback during captures; MQTT 1883 and syslog 5514 on `sensors.bind` (default `0.0.0.0`) when sensors are on, and MQTT over WebSockets on 9883 when `sensors.mqtt_web` is on too; the file shares' S3 server on 9555 and WebDAV server on 9556 (`shares.bind`) when file shares are on; llama-server on its port (0.0.0.0 in containers, with its own API key) | *(assumption, Q19)* | @@ -302,6 +303,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | email replies (`responder.py`; `POST /recordings/{id}/reply` and `/reply/send`, MCP `draft_reply` and `send_reply`, a pipeline's `reply` step) | an imported email's headers and text (From, Reply-To, Message-ID, References, the body), read to draft a reply; the reply sent, or kept in Gmail's drafts, through the Gmail connection the email came in on | yes, anyone who can send mail to a watched mailbox; From and Reply-To are forgeable | data, x-prompt-injection, size | drafting needs the right to process the email; the email reaches the model marked untrusted and the system prompt says not to follow it; only the email's own namespace is searched for excerpts; a web address in the draft that neither the email nor the excerpts hold is broken up, and a citation to an excerpt the model wasn't given is taken out; nothing is sent on its own: sending (or keeping it in Gmail's drafts) is an admin's own pick, signed in to Lens, after an Undo window, on a Gmail connection an admin let send (write), checked when it goes; an API key, an app or an agent can only ask (MCP `send_reply`), which waits in Approvals showing all of the message and sends the text that was approved, never a later edit; a draft with a [to confirm: …] mark left isn't sent; the recipient (the address Reply-To names) is shown and editable before anything goes; the reply goes without its citations; each send is in the audit log | *(documented, docs/email-replies.md "Lens never sends a reply on its own")* | | `POST /imports/web`, extension `http` tools, workflow HTTP call nodes, notification targets | URL | yes: editors (web, tools), owners (targets), admins (workflow nodes, host written out) | resource-name, x-network-destination | none | *(maintainer, 2026-10)* | | the assistant's `web_search` and `web_read` (off until an admin picks a provider, `web.search`) | a search query and a page address, both chosen by the model; the results and pages that come back | yes: anyone who may use the assistant, and indirectly whatever it read, through the model; results and pages: whoever runs the site | resource-name, x-network-destination, x-prompt-injection, size | none; queries go only to the admin's provider; pages are fetched through netguard like web import, 10 MB at most (P-SSRF-GUARD); once the answer has read outside text, a query or address neither the person nor a tool gave it asks first (P-INJECTION-EGRESS); results and pages reach the model marked as data; the ledger keeps the provider or host, never the query or page | *(documented, docs/assistant.md "a search or a page read sends words out of Lens")* | +| the Library's map (`GET /resources?located=true&bbox=…`, `GET /resources/map-tiles`, MCP `list_recordings` `located` and `bbox`) and `maps.tile_url` | a bounding box (four numbers); the tile address and its credit (admins) | bbox: yes, any member, parsed as four numbers in range (400 otherwise) and bound as query parameters; tile address: admins only (Settings → Maps) | data, x-network-destination | none; a place is listed only to members of its asset's namespace or collection, as `GET /resources/{rid}` already gives it; tiles are fetched by each viewer's browser, never by Lens, so the tile server sees the area looked at (N-MAP-TILES); Leaflet shows the credit as HTML, so the web app escapes it | *(documented, docs/processing.md "the tile server sees the area being looked at")* | | the assistant's connection tools (`connection_tools.py`: `connections`, `search_mail`, `read_mail`, `calendar_events`, `list_files`, `find_files`, `read_file`, `import_from_connection`, `send_mail`, `add_event`, `write_file`) | a connection's id or name, a Gmail search, a message id, a time span, a path; the mail, events and files that come back; the message, event or text file the model would send or write | yes: admins who use the assistant, and indirectly whatever it read, through the model; mail and events: anyone who can email the account or invite it; files: whoever writes to the storage | resource-name, x-prompt-injection, size | none — admins only, and only the tools for the connections there are; reading changes nothing in the account; a file is read up to 20 MB and a message's or file's text up to 11,000 characters; mail, events and files reach the model marked as data, and once read, tools that could carry it out of Lens ask first (P-INJECTION-EGRESS); bringing anything into a namespace waits for the person's yes; sending mail, saving a draft, adding an event and writing a file always wait for the person's yes, even in a setup conversation, with all of it shown (and a note when the answer read outside text), and the model is told never to act on what it read; sending and adding work only on a Gmail or Google Calendar connection an admin let do so (write, which asks Google for gmail.compose or calendar.events), checked again when approved; at most 50 addresses a field, 50,000 characters a message and 200,000 a file; a file is replaced only when asked and approved, and Google Drive connected read only is never written; each change is in the audit log; the ledger keeps what was called, never what came back | *(documented, docs/assistant.md "Mail, calendars and files")* | | event proposals from dates said (`POST /dates/propose`, MCP `propose_event`; `event_proposals.py`) | a recording, a line and which of its dates; the fields to use instead of the prefill (title, start, end, where, description, attendees, invite, connection, calendar); the line's text and an email's addresses that fill the rest | yes: admins with a write token or session; the line's text and addresses: whoever wrote the asset (an email's sender, a speaker) | data, x-prompt-injection, size | none; admins only, for an asset they can read, and only with a Google Calendar connection an admin let add events; the proposal is an approval of add_event with all of the event shown (the line, the time read from it, who is invited and that they aren't emailed unless invite is true), and nothing is added or sent until the person approves it signed in to Lens: an API key, an app, MCP and the assistant can only decline it; declining does nothing; the connection is checked again when approved; no model is called; at most 20 addresses are prefilled and 50 given; each proposal and each addition is in the audit log (calendar.propose, calendar.add) | *(documented, docs/processing.md "Propose an event")* | | admin settings: model, embeddings, speech, decisions, Fedora, bridge, Telegram bot, Slack app tokens, assistant mailbox, telemetry, IIIF import, rclone remotes, IMAP, Gmail (a Google refresh token or a Workspace service account key), iCal, news feeds, MQTT bridge, OIDC issuer, web search (a SearXNG address, a Brave Search or Tavily key), a LanguageTool server for the clean step's grammar fixes | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* | @@ -676,6 +678,7 @@ Exported by phase 3.6 (backtest) and de-identified. Closing rows carry their sta | N-PUBLIC | Privacy of a public asset's open parts: anyone may use them | Open parts only; closed parts and restricted or private assets are P-ACCESS-PARTS | correctness-only | no | *(documented, docs/access.md "the parts of a public asset anyone may use")* | | N-DESCRIPTION | Hiding an asset's description when its parts are closed | Anyone who may see the page | correctness-only | no | *(documented, docs/access.md "never a closed part")* | | N-NSFW | Access control from NSFW classification. Blurring and hiding flagged resources are display preferences of the web app, and the score can be wrong | Anyone who may read a resource still opens it, sees its pictures (`sensitive=include`, signed frame links, public pages, IIIF, embeds) and reads its classification | correctness-only | yes: "hidden" reads as restricted | *(documented, docs/nsfw.md "display preferences, not access control")* | +| N-MAP-TILES | Privacy from the map's tile server. Each member's browser fetches the Library map's tiles from `maps.tile_url` (OpenStreetMap's by default), so the tile server learns the tiles asked for (the area around where their assets were taken), the viewer's address and Lens's origin | No asset ids, titles or coordinates go to it. An empty `maps.tile_url` fetches nothing; a tile server of the operator's own keeps it on their hardware | correctness-only | yes: "data stays on your hardware" reads as covering the map | *(documented, docs/configuration.md "The tile server sees which tiles are asked for")* | | N-RDF-XML | XML parsing in RDF import. RDF/XML is refused, so XXE and billion-laughs do not apply there | RDF import only, not docx | security-critical | no | *(documented, rdf_import.py docstring "RDF/XML isn't read")* | The "False friend?" column carries the false friends. One more has no row: a read-scope token still reads everything diff --git a/threat-model.yaml b/threat-model.yaml index ac53939ef..9734c7cf5 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -98,6 +98,11 @@ host_side_effects: components: [web, identity, authz, outbound, ingest, query, assistant, optional] conditions: "none found in Lens code outside the configured destinations" provenance: {kind: assumption, question_id: Q19} + - effect: x-browser-map-tiles + stance: conditional + components: [web] + conditions: "the Library's Map fetches its tiles from maps.tile_url (OpenStreetMap's by default) in the member's browser while it is open; empty fetches nothing (N-MAP-TILES)" + provenance: {kind: documented, source: 'docs/configuration.md "The tile server sees which tiles are asked for"'} - effect: x-ml-extra-weight-downloads stance: conditional components: [ingest] @@ -391,6 +396,15 @@ entry_points: control_kinds: [resource-name, x-network-destination] caller_must_enforce: "none — P-SSRF-GUARD" provenance: {kind: maintainer, date: "2026-10"} + - id: "the Library's map: GET /resources?located=true&bbox=..., GET /resources/map-tiles, MCP list_recordings located and bbox, maps.tile_url" + component: authz + parameters: + - name: "a bounding box (four numbers); the tile address and its credit (admins)" + attacker_controllable: true + x-condition: "bbox: any member, parsed as four numbers in range (400 otherwise) and bound as query parameters; tile address: admins only (Settings, Maps)" + control_kinds: [data, x-network-destination] + caller_must_enforce: "none; a place is listed only to members of its asset's namespace or collection, as GET /resources/{rid} already gives it; tiles are fetched by each viewer's browser, never by Lens, so the tile server sees the area looked at (N-MAP-TILES); Leaflet shows the credit as HTML, so the web app escapes it" + provenance: {kind: documented, source: 'docs/processing.md "the tile server sees the area being looked at"'} - id: "the assistant's web_search and web_read" component: assistant parameters: @@ -2301,6 +2315,7 @@ properties_disclaimed: - {id: N-PUBLIC, components: [authz], tier: correctness-only, false_friend: false, conditions: "privacy of a public asset's open parts: anyone may use them; open parts only; closed parts and restricted or private assets are P-ACCESS-PARTS", provenance: {kind: documented, source: 'docs/access.md "the parts of a public asset anyone may use"'}} - {id: N-DESCRIPTION, components: [authz], tier: correctness-only, false_friend: false, conditions: "hiding an asset's description when its parts are closed; anyone who may see the page", provenance: {kind: documented, source: 'docs/access.md "never a closed part"'}} - {id: N-NSFW, components: [web, authz], tier: correctness-only, false_friend: true, conditions: "access control from NSFW classification; blurring and hiding flagged resources are display preferences of the web app, and the score can be wrong; anyone who may read a resource still opens it, sees its pictures (sensitive=include, signed frame links, public pages, IIIF, embeds) and reads its classification", x-false-friend-note: "'hidden' reads as restricted", provenance: {kind: documented, source: 'docs/nsfw.md "display preferences, not access control"'}} + - {id: N-MAP-TILES, components: [web], tier: correctness-only, false_friend: true, conditions: "privacy from the map's tile server: each member's browser fetches the Library map's tiles from maps.tile_url (OpenStreetMap's by default), so the tile server learns the tiles asked for (the area around where their assets were taken), the viewer's address and Lens's origin (Referer); no asset ids, titles or coordinates go to it. An empty maps.tile_url fetches nothing; a tile server of the operator's own keeps it on their hardware", x-false-friend-note: "'data stays on your hardware' reads as covering the map", provenance: {kind: documented, source: 'docs/configuration.md "The tile server sees which tiles are asked for"'}} - {id: N-RDF-XML, components: [query], tier: security-critical, false_friend: false, conditions: "XML parsing in RDF import; RDF/XML is refused, so XXE and billion-laughs do not apply there; RDF import only, not docx", provenance: {kind: documented, source: 'rdf_import.py docstring "RDF/XML isn''t read"'}} x-false-friends-without-row: