diff --git a/README.md b/README.md index f2d512c4..2ef73bc4 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p ([Web clipper](docs/web-clipper.md), [setting it up](docs/web-clipper.md#setting-it-up), [the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser)) - **Understand it.** Transcription, which can expect the names your namespace knows (off by default), people - recognised by voice (an owner can keep voices to each recording instead; long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own + recognised by voice (an owner can keep voices to each recording instead; long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default; the ones nobody has named yet are named in a row, the most seen first, with Undo), NSFW pictures flagged by a model on your own server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities and topics in a knowledge graph you can explore, query with Cypher or SPARQL, and see as it was at any version, where every source, sensor, person and agent is a node linked to what it @@ -192,7 +192,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p itself shows (its captions menu, full screen, picture in picture), in the workspace and on public pages. ([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages), [similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md), - [topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md), + [topics](docs/topics.md), [video](docs/video.md), [naming faces](docs/video.md#naming-faces), [NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings), [cleaning up audio](docs/processing.md#cleaning-up-audio), [your own names](docs/processing.md#your-own-names-in-transcription), diff --git a/changelog.d/unnamed-faces.md b/changelog.d/unnamed-faces.md new file mode 100644 index 00000000..bdd6c40a --- /dev/null +++ b/changelog.d/unnamed-faces.md @@ -0,0 +1,7 @@ +- **Name the unnamed faces in a row.** Where a namespace recognises faces, Settings → People has an Unnamed faces tab + (and a video's People tab links to it): the faces nobody has named yet, the largest group first, each with a few + crops and the assets it's in. The first name field has focus, filled with the name Lens offers when it has one (the + voice the face is linked to, or a named face suggested for it); Enter names it and moves on to the next, with Undo. + A name another face already has joins that person, the same merge as "Same person". Agents do it with + `GET /api/v1/namespaces//faces/unnamed` and `POST /api/v1/faces//name`, or the MCP tools + `list_unnamed_faces` and `name_face` ([Naming faces](docs/video.md#naming-faces)). diff --git a/docs/api.md b/docs/api.md index f0f3e853..4ce54d24 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1550,10 +1550,12 @@ GET /api/v1/resources/{rid}/frames/{name} PATCH /api/v1/resources/{rid}/ocr/{span} DELETE /api/v1/resources/{rid}/faces/{track} GET /api/v1/namespaces/{name}/faces +GET /api/v1/namespaces/{name}/faces/unnamed DELETE /api/v1/namespaces/{name}/faces PUT /api/v1/namespaces/{name}/faces/mode POST /api/v1/faces/merges/{mid}/undo POST /api/v1/faces/{fid} +POST /api/v1/faces/{fid}/name DELETE /api/v1/faces/{fid} POST /api/v1/faces/{fid}/merge POST /api/v1/faces/{fid}/speaker @@ -1567,6 +1569,14 @@ say them for a resource. `GET /resources/{rid}/frames/{name}` (frames, keyframes image's pages and thumbnails) serves a request without a role in the namespace the faces found on it pixelated, where the namespace says so; a role, or a link the API signed for a member (`full=1`), gets the picture as it is. +`GET /namespaces/{name}/faces/unnamed?limit=` lists the faces nobody has named yet (recognize mode; none otherwise), +the largest group first: in the most assets, then the longest on screen (a page counts five seconds). Each has a few +signed crops (`thumbnails`, from different assets first), the assets it's in (`recordings`, where it's seen longest +first), how often it's seen, and `guess`, a name to offer: the voice it's linked to, else a named face or voice +suggested for it. `POST /faces/{fid}/name {name}` (editors) names one: a name another face of the namespace has (any +case) joins it to that face, a merge whose id comes back as `merge` to undo with `POST /faces/merges/{mid}/undo`; any +other name becomes its own, undone with `POST /faces/{fid} {"name": ""}` ([Naming faces](video.md#naming-faces)). + ### Objects The `objects` step finds the people, vehicles, animals and everyday things (the 80 kinds of the COCO dataset) on a diff --git a/docs/mcp.md b/docs/mcp.md index b8703177..03ee8900 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -78,6 +78,8 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `budgets` | where spending stands: each namespace you can read this period against its budget, and for admins all of Lens and every other budget, each ok, near or over ([Asking the assistant or an agent](budgets.md#asking-the-assistant-or-an-agent)) | | `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) | | `list_speakers` | the people of your namespaces, the ones with the most talk time first | +| `list_unnamed_faces` | the faces nobody has named yet in your namespaces that recognise faces, the largest group first: a name to offer, how often each is seen and the assets it's in, with links ([Naming faces](video.md#naming-faces)) | +| `name_face` | say who a face is: a name another face of its namespace has joins that person, any other becomes its own. Needs a write-scope token and editor access | | `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW | | `get_transcript` | an asset's lines in order (who, time, text, link, and the words transcription wasn't sure of as `unsure`), a page at a time, from a line or a time window | | `fetch` | an asset's whole text at once, as `[m:ss] Speaker: …` lines | diff --git a/docs/requirements.md b/docs/requirements.md index f60e85a1..de011c29 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -206,6 +206,7 @@ request, to another check that proves the same thing; loosening a requirement or | An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` | | Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` | | Faces, off by default | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_people_on_screen`, `test_shots_text_search_chat_and_iiif`; [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_video.py) `test_face_modes` | +| The faces nobody has named yet, the largest group first with crops, the assets they're in and a name to offer, named one after another (Enter for the next), a name someone has joining that person, with Undo; through the API, MCP and the People page | [test_unnamed_faces.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_unnamed_faces.py) `test_unnamed_faces_largest_first_named_in_a_row_with_undo`, `test_only_where_faces_are_recognised`, `test_agents_list_and_name_faces`; jest [name-faces.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/name-faces.test.tsx) `focuses the largest group first; Enter names it and moves on to the next, with Undo`, `joins a face to the person who has the name, and Undo takes the merge back` | | NSFW pictures flagged by a model on your own server, off by default, per namespace | [test_nsfw.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_nsfw.py) `test_off_by_default_nothing_is_fetched_or_classified`, `test_a_namespace_classifies_flags_blurs_and_hides`; [nsfw.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/nsfw.test.tsx) | | Low quality images enhanced before they're read, off by default | [test_enhance.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_enhance.py) `test_off_by_default_the_original_is_read`, `test_a_low_quality_image_is_read_from_its_enhanced_copy`, `test_a_good_image_is_left_as_it_is`, `test_agents_and_automations_enhance_too`, `test_the_copy_is_encrypted_where_the_files_are`; [enhance.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/enhance.test.tsx) | | Entities and topics in a knowledge graph to explore and query with Cypher or SPARQL | [test_graph.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph.py) `test_query`; [test_rdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rdf.py) `test_sparql`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `graph-find` | diff --git a/docs/video.md b/docs/video.md index ef1cbefb..7bb48232 100644 --- a/docs/video.md +++ b/docs/video.md @@ -16,7 +16,8 @@ soundtrack, and three more steps look at the picture. Each step skips itself for - Where a namespace allows it, faces are grouped per recording and matched against that namespace's people, the way voices are: auto-match, review, or new. - When a face is on screen while one speaker talks, the app suggests they're the same person. - - Faces can be renamed, merged (with undo), linked to a person in People, marked "not a face" or deleted. + - Faces can be renamed, merged (with undo), linked to a person in People, marked "not a face" or deleted, and the + ones nobody has named yet are [named in a row](#naming-faces). - Engines: OpenCV's YuNet + SFace (`pip install -e ".[faces]"`, then point `video.yunet_model` and `video.sface_model` at the ONNX files from the OpenCV Zoo), InsightFace, whose pretrained models are licensed for non-commercial research only, or `anytopdf`: [anytopdf](configuration.md#anytopdf)'s plugins find faces with @@ -66,6 +67,23 @@ recording's media, even when its transcript came with it: Recordings transcribed before the setting was on get their copy when their transcribe step runs again. +## Naming faces + +In recognize mode, the faces nobody has named yet wait in **Settings → People → Unnamed faces** (the tab shows only +where faces are recognised; a video's People tab links to it while any are unnamed). The largest group comes first: +the face in the most assets, then the longest on screen. Each shows a few crops, how often it's seen and the assets +it's in. + +- The first face's name field has focus, filled with the name Lens offers when it has one (the voice the face is + linked to, or a named face or voice suggested for it) and selected, so typing replaces it. +- Enter names it and moves on to the next. The notice that follows has Undo. +- A name another face of the namespace already has (any case) joins that person: it's the same merge as "Same + person", and Undo takes it back. Any other name becomes the face's own, everywhere in the namespace. +- Naming needs the curate permission (editors); members who can't see the reason on the button. + +Agents do the same with the MCP tools `list_unnamed_faces` and `name_face`, or `GET +/api/v1/namespaces//faces/unnamed` and `POST /api/v1/faces//name` ([API](api.md#video)). + ## The video recording page Video recordings open in their own layout. diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index b606c999..60ef186e 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -909,6 +909,60 @@ def list_speakers(ctx: Context, namespace: str | None, query: str | None, limit: return {"total": len(rows), "speakers": rows[:limit]} +@tool( + "list_unnamed_faces", + "List unnamed faces", + "The faces nobody has named yet in the namespaces you have a role in that recognise faces, the largest group " + "first: each face's id, what it's called until named, a name to offer (`guess`: the voice it's linked to or one " + "suggested), how often it's seen and the assets it's in, with links. Name them with name_face.", + Arg("namespace", "string", "only this namespace"), + Arg("limit", "integer", "how many", default=50, minimum=1, maximum=500), +) +def list_unnamed_faces(ctx: Context, namespace: str | None, limit: int) -> dict[str, Any]: + from app.domain import faces + + spaces = [ctx.acl.namespace(namespace)] if namespace else ctx.acl.spaces() + names = store.space_names(ctx.db) + rows = [ + store.clean( + { + "face_id": g["id"], + "label": g["label"], + "guess": g["guess"], + "namespace": names.get(sid), + "appearances": g["appearances"], + "screen_minutes": round(g["screen_ms"] / 60000, 1) if g["screen_ms"] else None, + "pages": g["pages"] or None, + "assets": [{"recording_id": r["id"], "title": r["title"], "url": ctx.link(r["id"])} for r in g["recordings"][:10]], + "more_assets": max(0, len(g["recordings"]) - 10) or None, + } + ) + for sid in spaces + if faces.mode(ctx.db, sid) == "recognize" + for g in faces.unnamed(ctx.db, sid, thumbs=0) + ] + return {"total": len(rows), "faces": rows[:limit]} + + +@tool( + "name_face", + "Name a face", + "Say who a face is (ids from list_unnamed_faces): a name another face of its namespace has joins it to that " + "person (a merge, which the web app can undo), any other name becomes its own. Needs a token with the write scope " + "and editor access.", + Arg("face_id", "integer", "the face", required=True), + Arg("name", "string", "who it is", required=True, max_length=80), + writes=True, +) +def name_face(ctx: Context, face_id: int, name: str) -> dict[str, Any]: + from app.api.v1.routes import video as video_routes + from app.schemas.video import FaceName + + _writer(ctx, "name faces") + out = video_routes.name_face(face_id, FaceName(name=name), ctx.user, ctx.acl, ctx.db) + return store.clean({"face_id": face_id, "name": out.name, "joined_face_id": out.into, "merge_id": out.merge}) + + # ---------- reading ---------- @tool( "get_recording", diff --git a/fastapi_backend/app/api/v1/routes/video.py b/fastapi_backend/app/api/v1/routes/video.py index 319fb691..9152bf2d 100644 --- a/fastapi_backend/app/api/v1/routes/video.py +++ b/fastapi_backend/app/api/v1/routes/video.py @@ -12,7 +12,7 @@ import re from typing import Any -from fastapi import APIRouter, HTTPException, Request +from fastapi import APIRouter, HTTPException, Query, Request from fastapi.responses import FileResponse, Response from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Writer, domain_errors @@ -27,12 +27,15 @@ FaceDismiss, FaceMerge, FaceMerged, + FaceName, + FaceNamed, FaceRename, FacesMode, FacesModeSet, FaceSpeaker, NamespaceFaces, OcrFix, + UnnamedFaces, ) router = APIRouter(tags=["video"]) @@ -190,6 +193,21 @@ def get_namespace_faces(name: str, user: CurrentUser, acl: Acl, db: Db) -> Names return NamespaceFaces.model_validate(sign_urls(out, full=True)) +@router.get("/namespaces/{name}/faces/unnamed") +def get_unnamed_faces(name: str, user: CurrentUser, acl: Acl, db: Db, limit: int = Query(100, ge=1, le=500)) -> UnnamedFaces: + """The faces nobody has named yet, grouped across the namespace (recognize mode), the largest group first: each + with a few signed crops, the assets it's in and a name to offer. Name them with POST /faces/{fid}/name.""" + sid = acl.namespace(name) + mode = facemod.mode(db, sid) + groups = facemod.unnamed(db, sid) if mode == "recognize" else [] + for g in groups: + g["thumbnails"] = [ + {"recording": t["recording"], "cover_url": f"{API}/recordings/{t['recording']}/frames/{t['file']}"} for t in g["thumbnails"] + ] + out = {"mode": mode, "total": len(groups), "groups": groups[:limit]} + return UnnamedFaces.model_validate(sign_urls(out, full=True)) + + @router.put("/namespaces/{name}/faces/mode") def set_namespace_faces_mode(name: str, body: FacesMode, user: Writer, acl: Acl, db: Db, cfg: Cfg) -> FacesModeSet: """off, detect or recognize (which needs a purpose), and whether faces are pixelated for visitors. Owners only. @@ -242,6 +260,20 @@ def rename_face(fid: int, body: FaceRename, user: Writer, acl: Acl, db: Db) -> O return Ok() +@router.post("/faces/{fid}/name") +def name_face(fid: int, body: FaceName, user: Writer, acl: Acl, db: Db) -> FaceNamed: + """Name a face: a name another face of the namespace has joins it to that person (a merge: undo it with POST + /faces/merges/{merge}/undo); any other name is its own (undo: POST /faces/{fid} with no name).""" + _face_space(db, acl, fid, "curate") + with domain_errors(): + out = facemod.name_face(db, fid, body.name, user.email) + if out["merge"]: + auth.audit(db, user.as_audit(), "face.merge", f"face:{fid}", {"into": out["into"]}) + else: + auth.audit(db, user.as_audit(), "face.rename", f"face:{fid}") + return FaceNamed(**out) + + @router.post("/faces/{fid}/merge") def merge_face(fid: int, body: FaceMerge, user: Writer, acl: Acl, db: Db) -> FaceMerged: """Merge this face into another (the same person).""" diff --git a/fastapi_backend/app/domain/faces.py b/fastapi_backend/app/domain/faces.py index fad6effa..c3f974d6 100644 --- a/fastapi_backend/app/domain/faces.py +++ b/fastapi_backend/app/domain/faces.py @@ -369,10 +369,97 @@ def list_faces(db, sid): return sorted(out, key=lambda x: (-x["screen_ms"], x["id"])) +THUMBS = 4 # crops shown for each face waiting for a name + + +def unnamed(db, sid, thumbs=THUMBS): + """The faces of a namespace nobody has named yet, the largest group first (in the most assets, then the longest on + screen, a page counting PAGE_SECONDS): each with a few crops (from different assets first), the assets it's in and + `guess`, a name to offer: the voice it's linked to, else the named face or voice suggested for it.""" + rows = [ + r + for r in db.rows("SELECT record::id(id) AS id, label, name, speaker FROM face WHERE space = $s", s=sid) + if not (r.get("name") or "").strip() + ] + if not rows: + return [] + tracks = defaultdict(list) + for t in db.rows( + "SELECT face, recording, screen_ms, paged, cover FROM face_track WHERE space = $s AND face IN $f", s=sid, f=[r["id"] for r in rows] + ): + tracks[t["face"]].append(t) + rids = {t["recording"] for ts in tracks.values() for t in ts} + titles = { + r["id"]: r.get("title") + for r in db.rows("SELECT record::id(id) AS id, title FROM recording WHERE id IN $r", r=[R("recording", x) for x in rids]) + } + named = {r["id"]: r["name"] for r in db.rows("SELECT record::id(id) AS id, name FROM face WHERE space = $s AND name != NONE", s=sid)} + voices = { + r["id"]: r["name"] for r in db.rows("SELECT record::id(id) AS id, name FROM speaker WHERE space = $s AND name != NONE", s=sid) + } + offered = defaultdict(list) # face -> [(score, name)], named faces and voices only + for g in db.rows("SELECT kind, face, candidate, speaker, score FROM face_suggestion WHERE space = $s", s=sid): + name = named.get(g.get("candidate")) if g["kind"] == "face" else voices.get(g.get("speaker")) + if name: + offered[g["face"]].append((g.get("score") or 0, name)) + out = [] + for r in rows: + ts = tracks.get(r["id"]) + if not ts: # every track of it was removed as not a face + continue + ms = defaultdict(int) + for t in ts: + ms[t["recording"]] += (t.get("screen_ms") or 0) * (PAGE_SECONDS * 1000 if t.get("paged") else 1) + order = sorted(ms, key=lambda x: (-ms[x], x)) + rank = {x: i for i, x in enumerate(order)} + ts = sorted(ts, key=lambda t: (rank[t["recording"]], -(t.get("screen_ms") or 0))) + seen, first, rest = set(), [], [] + for t in ts: + if t.get("cover"): + (rest if t["recording"] in seen else first).append({"recording": t["recording"], "file": t["cover"]}) + seen.add(t["recording"]) + out.append( + { + "id": r["id"], + "label": r["label"], + "guess": voices.get(r.get("speaker")) or max(offered.get(r["id"]) or [(0, None)])[1], + "speaker": r.get("speaker"), + "appearances": len(ts), + "screen_ms": sum(0 if t.get("paged") else t.get("screen_ms") or 0 for t in ts), + "pages": sum(t.get("screen_ms") or 0 for t in ts if t.get("paged")), + "weight": sum(ms.values()), + "thumbnails": (first + rest)[:thumbs], + "recordings": [{"id": x, "title": titles.get(x)} for x in order], + } + ) + out.sort(key=lambda x: (-len(x["recordings"]), -x["weight"], x["id"])) + for x in out: + del x["weight"] + return out + + def rename(db, fid, name): db.q("UPDATE $r SET name = $n", r=R("face", fid), n=(name or "").strip()[:80] or None) +def name_face(db, fid, name, user=None): + """Name a face. A name another face of the namespace already has joins it to that person (a merge, which undo + takes back); any other name is the face's own (rename). -> {"name", "into": the face joined, "merge": its id}""" + name = (name or "").strip()[:80] + f = db.one("SELECT space FROM $r", r=R("face", fid)) + if not f or not name: + raise ValueError("say the name" if f else "no such face") + same = sorted( + r["id"] + for r in db.rows("SELECT record::id(id) AS id, name FROM face WHERE space = $s AND name != NONE", s=f["space"]) + if r["id"] != fid and r["name"].strip().casefold() == name.casefold() + ) + if same: + return {"name": name, "into": same[0], "merge": merge(db, fid, same[0], user)} + rename(db, fid, name) + return {"name": name, "into": None, "merge": None} + + def link_speaker(db, fid, speaker): f = db.one("SELECT space FROM $r", r=R("face", fid)) if speaker is not None: diff --git a/fastapi_backend/app/schemas/video.py b/fastapi_backend/app/schemas/video.py index 4b1cce11..65dea3be 100644 --- a/fastapi_backend/app/schemas/video.py +++ b/fastapi_backend/app/schemas/video.py @@ -42,6 +42,44 @@ class FaceRename(RequestModel): name: str = "" +class FaceName(RequestModel): + name: str = Field(description="who it is; a name another face of the namespace has joins it to that person") + + +class FaceNamed(Ok): + name: str + into: int | None = Field(None, description="the face of the same name it joined, if any") + merge: int | None = Field(None, description="that merge, to undo with POST /faces/merges/{merge}/undo") + + +class FaceThumb(ResponseModel): + recording: int + cover_url: str = Field(description="a signed link to the crop") + + +class FaceAsset(ResponseModel): + id: int + title: str | None = None + + +class UnnamedFace(ResponseModel): + id: int + label: str = Field(description="what it's called until it's named, e.g. Face 3") + guess: str | None = Field(None, description="a name to offer: the voice it's linked to, else one suggested for it") + speaker: int | None = Field(None, description="the voice it's linked to") + appearances: int + screen_ms: int = Field(description="time on screen in videos") + pages: int = Field(description="pages of documents and images it's on") + thumbnails: list[FaceThumb] + recordings: list[FaceAsset] = Field(description="the assets it's in, where it's seen longest first") + + +class UnnamedFaces(ResponseModel): + mode: str = Field(description="off, detect or recognize; faces are grouped across the namespace only in recognize") + total: int + groups: list[UnnamedFace] = Field(description="the largest group first") + + class FaceMerge(RequestModel): into: int = 0 diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index 3386019f..ac5b5800 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -74,6 +74,7 @@ def test_clients_are_sent_to_sign_in(client, app, env): # every tool only reads, except proposing an event, suggesting a topic and asking for a graph change (which need # the write scope), and the rest below assert [t["name"] for t in tools if not t["annotations"]["readOnlyHint"]] == [ + "name_face", "propose_event", "suggest_topic", "propose_graph_change", diff --git a/fastapi_backend/tests/api/test_unnamed_faces.py b/fastapi_backend/tests/api/test_unnamed_faces.py new file mode 100644 index 00000000..76fe909c --- /dev/null +++ b/fastapi_backend/tests/api/test_unnamed_faces.py @@ -0,0 +1,141 @@ +"""Naming faces in bulk (docs/video.md#naming-faces): a namespace's unnamed face groups, the largest first, each with +crops, the assets it's in and a name to offer; naming one makes it that person, or joins the face that has the name, +with undo; through the API and MCP.""" + +from __future__ import annotations + +import pytest + +from app.domain import faces, store +from tests.api.test_mcp import rpc, tool +from tests.helpers import login, make_user, seed + +R = store.R + + +@pytest.fixture +def env(client, db, cfg, folder): + """In pods (recognising faces): a face in both episodes, one on two pages of ep1 that looks like Mara, one briefly + in ep1 linked to Alice's voice, Mara herself, and a face whose only track was removed as not a face.""" + ep1, ep2, _ = seed(db, cfg, folder) + pods = store.ns_id(db, "pods") + faces.set_mode(db, pods, "recognize", "Name the hosts", "root@x.io") + big, paged, brief, mara, gone = (faces.new_face(db, pods) for _ in range(5)) + faces.rename(db, mara, "Mara") + alice = next(s["id"] for s in faces.spk.list_speakers(db, pods) if s["display"] == "Alice") + db.q("UPDATE $r SET speaker = $s", r=R("face", brief), s=alice) + db.q("CREATE face_suggestion CONTENT $d", d={"kind": "face", "face": paged, "candidate": mara, "score": 0.61, "space": pods}) + + def track(face, rid, ms, cover, paged=None): + return store.clean( + { + "recording": rid, + "space": pods, + "local": "P1", + "face": face, + "spans": [[0, ms]], + "screen_ms": ms, + "first_ms": 0, + "cover": cover, + "paged": paged, + } + ) + + db.q( + "INSERT INTO face_track $rows", + rows=[ + track(big, ep1, 3000, "face-1.jpg"), + track(big, ep2, 9000, "face-1.jpg"), + track(paged, ep1, 2, "face-3.jpg", paged=True), # two pages: as long as ten seconds on screen + track(brief, ep1, 1000, "face-2.jpg"), + track(mara, ep2, 4000, "face-2.jpg"), + ], + ) + make_user(db, "root@x.io", "root password 1", admin=True) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + return { + "ep1": ep1, + "ep2": ep2, + "pods": pods, + "big": big, + "paged": paged, + "brief": brief, + "mara": mara, + "gone": gone, + "h": login(client, "root@x.io", "root password 1"), + "hv": login(client, "vi@x.io", "viewer password 1"), + "he": login(client, "ed@x.io", "editor password 1"), + } + + +def name_face(client, h, **args): + """name_face over MCP (its `name` argument would clash with the tool helpers' own): (result, error).""" + res = rpc(client, h, "tools/call", {"name": "name_face", "arguments": args})["result"] + return (None, res["content"][0]["text"]) if res["isError"] else (res["structuredContent"], None) + + +def unnamed(client, h): + r = client.get("/api/v1/namespaces/pods/faces/unnamed", headers=h) + assert r.status_code == 200, r.text + return r.json() + + +def test_unnamed_faces_largest_first_named_in_a_row_with_undo(client, db, env): + hv, he, ep1, ep2 = env["hv"], env["he"], env["ep1"], env["ep2"] + got = unnamed(client, hv) + assert got["mode"] == "recognize" and got["total"] == 3 # not Mara (named), nor a face with no tracks left + big, paged, brief = got["groups"] + assert [g["id"] for g in got["groups"]] == [env["big"], env["paged"], env["brief"]] # most assets, then longest + assert [g["guess"] for g in got["groups"]] == [None, "Mara", "Alice"] # a suggested face, a linked voice + assert [r["id"] for r in big["recordings"]] == [ep2, ep1] and big["recordings"][0]["title"] # longest seen first + assert [t["recording"] for t in big["thumbnails"]] == [ep2, ep1] + assert all("sig=" in t["cover_url"] for t in big["thumbnails"]) # an can't send a bearer token + assert (big["appearances"], big["screen_ms"], paged["pages"], paged["screen_ms"]) == (2, 12000, 2, 0) + assert client.get("/api/v1/namespaces/pods/faces/unnamed?limit=1", headers=hv).json()["total"] == 3 + + # a new name: the face becomes that person; Undo is the old rename with no name + assert client.post(f"/api/v1/faces/{env['big']}/name", headers=hv, json={"name": "Ann"}).status_code == 403 + assert client.post(f"/api/v1/faces/{env['big']}/name", headers=he, json={"name": " "}).status_code == 400 + r = client.post(f"/api/v1/faces/{env['big']}/name", headers=he, json={"name": " Ann "}) + assert r.json() == {"ok": True, "name": "Ann", "into": None, "merge": None} + assert [g["id"] for g in unnamed(client, hv)["groups"]] == [env["paged"], env["brief"]] + assert {f["display"] for f in client.get("/api/v1/namespaces/pods/faces", headers=hv).json()["faces"]} >= {"Ann", "Mara"} + assert client.post(f"/api/v1/faces/{env['big']}", headers=he, json={"name": ""}).status_code == 200 + assert unnamed(client, hv)["total"] == 3 + + # a name another face has (any case): it joins that person, and undoing the merge brings it back + r = client.post(f"/api/v1/faces/{env['paged']}/name", headers=he, json={"name": "mara"}).json() + assert (r["into"], r["name"]) == (env["mara"], "mara") and r["merge"] + assert db.values("SELECT VALUE face FROM face_track WHERE recording = $r AND paged = true", r=ep1) == [env["mara"]] + assert [g["id"] for g in unnamed(client, hv)["groups"]] == [env["big"], env["brief"]] + assert client.post(f"/api/v1/faces/merges/{r['merge']}/undo", headers=he).status_code == 200 + assert [g["id"] for g in unnamed(client, hv)["groups"]] == [env["big"], env["paged"], env["brief"]] + + actions = [a["action"] for a in client.get("/api/v1/audit", headers=env["h"]).json()] + assert {"face.rename", "face.merge", "face.merge.undo"} <= set(actions) + + +def test_only_where_faces_are_recognised(client, db, env): + faces.set_mode(db, env["pods"], "detect") + assert unnamed(client, env["hv"]) == {"mode": "detect", "total": 0, "groups": []} # no identities to name + assert client.get("/api/v1/namespaces/calls/faces/unnamed", headers=env["hv"]).status_code == 404 # not theirs + + +def test_agents_list_and_name_faces(client, db, env): + hv, he = env["hv"], env["he"] + got = tool(client, hv, "list_unnamed_faces", namespace="pods") + first = got["faces"][0] + assert got["total"] == 3 and first["face_id"] == env["big"] and first["namespace"] == "pods" + assert first["assets"][0] == { + "recording_id": env["ep2"], + "title": first["assets"][0]["title"], + "url": f"http://localhost:3000/resources/{env['ep2']}", + } + assert (first["appearances"], first["screen_minutes"]) == (2, 0.2) + assert [f["guess"] for f in got["faces"][1:]] == ["Mara", "Alice"] + assert "curate" in name_face(client, hv, face_id=env["brief"], name="Alice")[1] # viewers can't + assert name_face(client, he, face_id=env["brief"], name="Alice")[0] == {"face_id": env["brief"], "name": "Alice"} + joined, _ = name_face(client, he, face_id=env["paged"], name="Mara") + assert joined["joined_face_id"] == env["mara"] and joined["merge_id"] + assert [f["face_id"] for f in tool(client, he, "list_unnamed_faces")["faces"]] == [env["big"]] diff --git a/nextjs-frontend/__tests__/name-faces.test.tsx b/nextjs-frontend/__tests__/name-faces.test.tsx new file mode 100644 index 00000000..2c66db78 --- /dev/null +++ b/nextjs-frontend/__tests__/name-faces.test.tsx @@ -0,0 +1,143 @@ +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { fireEvent, render, screen, waitFor, within } from "@testing-library/react"; +import "@testing-library/jest-dom"; +import type { ReactNode } from "react"; + +import { Video } from "@/app/openapi-client"; +import type { UnnamedFace } from "@/app/openapi-client/types.gen"; +import { NameFaces } from "@/components/speakers/name-faces"; +import { ToastProvider } from "@/components/ui/toast"; +import { TooltipProvider } from "@/components/ui/tooltip"; + +jest.mock("@/app/openapi-client", () => ({ + Video: { getUnnamedFaces: jest.fn(), nameFace: jest.fn(), renameFace: jest.fn(), undoFaceMerge: jest.fn() }, +})); +jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) })); +let role = "editor"; +jest.mock("@/lib/hooks/session", () => ({ + useArchive: () => ({ can: (r: string) => r === "viewer" || role === "editor" }), + needRole: (r: string, ns: string) => `Needs ${r} access to ${ns}`, +})); + +const ok = (data: unknown) => Promise.resolve({ data, response: { ok: true, status: 200 } }); +const m = (f: unknown) => f as jest.Mock; + +function face(id: number, over: Partial = {}): UnnamedFace { + return { + id, + label: `Face ${id}`, + appearances: 1, + screen_ms: 12_000, + pages: 0, + thumbnails: [{ recording: 7, cover_url: `/api/v1/recordings/7/frames/face-${id}.jpg?sig=x` }], + recordings: [{ id: 7, title: "Weekly review" }], + ...over, + }; +} + +let groups: UnnamedFace[] = []; +let mode = "recognize"; +function serve() { + m(Video.getUnnamedFaces).mockImplementation(() => ok({ mode, total: groups.length, groups })); + m(Video.nameFace).mockImplementation(({ path, body }: { path: { fid: number }; body: { name: string } }) => { + groups = groups.filter((g) => g.id !== path.fid); + return ok({ + ok: true, + name: body.name, + into: body.name === "Mara" ? 9 : null, + merge: body.name === "Mara" ? 31 : null, + }); + }); + m(Video.renameFace).mockImplementation(() => ok({ ok: true })); + m(Video.undoFaceMerge).mockImplementation(() => ok({ ok: true })); +} + +function wrap(ui: ReactNode) { + const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render( + + + {ui} + + , + ); +} + +beforeEach(() => { + jest.clearAllMocks(); + role = "editor"; + mode = "recognize"; + groups = [ + face(3, { + appearances: 2, + thumbnails: [ + { recording: 7, cover_url: "/a.jpg?sig=x" }, + { recording: 8, cover_url: "/b.jpg?sig=x" }, + ], + recordings: [ + { id: 7, title: "Weekly review" }, + { id: 8, title: "Board deck" }, + ], + }), + face(5, { guess: "Mara", screen_ms: 0, pages: 2 }), + face(6), + ]; + serve(); +}); + +describe("naming faces", () => { + it("focuses the largest group first; Enter names it and moves on to the next, with Undo", async () => { + wrap(); + const first = await screen.findByRole("textbox", { name: "Name for Face 3" }); + await waitFor(() => expect(first).toHaveFocus()); + const row = screen.getByRole("listitem", { name: "Face 3" }); + expect(within(row).getByText("2 assets · 12 s on screen")).toBeInTheDocument(); + expect(within(row).getByRole("link", { name: "Board deck" })).toHaveAttribute("href", "/resources/8"); + expect(row.querySelectorAll("img")).toHaveLength(2); + expect(screen.getByText(/3 faces without a name/)).toBeInTheDocument(); + + fireEvent.change(first, { target: { value: "Ann" } }); + fireEvent.submit(first.closest("form")!); + await waitFor(() => + expect(Video.nameFace).toHaveBeenCalledWith(expect.objectContaining({ path: { fid: 3 }, body: { name: "Ann" } })), + ); + const next = await screen.findByRole("textbox", { name: "Name for Face 5" }); + await waitFor(() => expect(next).toHaveFocus()); // focus moves on to the next one + expect(next).toHaveValue("Mara"); // the name Lens offers, ready for Enter + expect(screen.getByRole("listitem", { name: "Face 5" })).toHaveTextContent("1 asset · 2 pages"); + expect(screen.queryByRole("listitem", { name: "Face 3" })).not.toBeInTheDocument(); + + fireEvent.click(await screen.findByRole("button", { name: "Undo" })); // Face 3 is Ann: undo renames it back + await waitFor(() => + expect(Video.renameFace).toHaveBeenCalledWith(expect.objectContaining({ path: { fid: 3 }, body: { name: "" } })), + ); + }); + + it("joins a face to the person who has the name, and Undo takes the merge back", async () => { + groups = groups.slice(1); + wrap(); + const input = await screen.findByRole("textbox", { name: "Name for Face 5" }); + fireEvent.submit(input.closest("form")!); + expect(await screen.findByText("Face 5 joined Mara")).toBeInTheDocument(); + fireEvent.click(screen.getByRole("button", { name: "Undo" })); + await waitFor(() => + expect(Video.undoFaceMerge).toHaveBeenCalledWith(expect.objectContaining({ path: { mid: 31 } })), + ); + }); + + it("says why when faces aren't recognised, nobody is waiting, or the person can't name them", async () => { + mode = "detect"; + groups = []; + const { unmount } = wrap(); + expect(await screen.findByText("Faces aren’t recognised in pods")).toBeInTheDocument(); + unmount(); + mode = "recognize"; + const empty = wrap(); + expect(await screen.findByText("Every face has a name")).toBeInTheDocument(); + empty.unmount(); + role = "viewer"; + groups = [face(6)]; + wrap(); + expect(await screen.findByRole("textbox", { name: "Name for Face 6" })).toBeDisabled(); + }); +}); diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index d525ce1b..aa5b52e4 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -1238,14 +1238,18 @@ export type { ExternalStartErrors, ExternalStartResponse, ExternalStartResponses, + FaceAsset, FaceDismiss, FaceMerge, FaceMerged, + FaceName, + FaceNamed, FaceRename, FacesMode, FacesModeSet, FaceSpeaker, FacetCount, + FaceThumb, FedoraStatus, FedoraSyncResult, Feed, @@ -1728,6 +1732,11 @@ export type { GetTopicErrors, GetTopicResponse, GetTopicResponses, + GetUnnamedFacesData, + GetUnnamedFacesError, + GetUnnamedFacesErrors, + GetUnnamedFacesResponse, + GetUnnamedFacesResponses, GetUpdatesData, GetUpdatesResponse, GetUpdatesResponses, @@ -2708,6 +2717,11 @@ export type { MuteActivitiesResponses, MuteIn, MutesOut, + NameFaceData, + NameFaceError, + NameFaceErrors, + NameFaceResponse, + NameFaceResponses, Namespace, NamespaceAccess, NamespaceAssistant, @@ -4249,6 +4263,8 @@ export type { UnlockVaultErrors, UnlockVaultResponse, UnlockVaultResponses, + UnnamedFace, + UnnamedFaces, UnpinPageData, UnpinPageError, UnpinPageErrors, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index b95f6f8b..ba22e00f 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -846,6 +846,9 @@ import type { GetTopicData, GetTopicErrors, GetTopicResponses, + GetUnnamedFacesData, + GetUnnamedFacesErrors, + GetUnnamedFacesResponses, GetUpdatesData, GetUpdatesResponses, GetUploadData, @@ -1349,6 +1352,9 @@ import type { MuteActivitiesData, MuteActivitiesErrors, MuteActivitiesResponses, + NameFaceData, + NameFaceErrors, + NameFaceResponses, NewImportHookTokenData, NewImportHookTokenErrors, NewImportHookTokenResponses, @@ -9647,6 +9653,21 @@ export class Video { }); } + /** + * Get Unnamed Faces + * + * The faces nobody has named yet, grouped across the namespace (recognize mode), the largest group first: each + * with a few signed crops, the assets it's in and a name to offer. Name them with POST /faces/{fid}/name. + */ + public static getUnnamedFaces( + options: Options, + ): RequestResult { + return (options.client ?? client).get({ + url: "/api/v1/namespaces/{name}/faces/unnamed", + ...options, + }); + } + /** * Set Namespace Faces Mode * @@ -9708,6 +9729,25 @@ export class Video { }); } + /** + * Name Face + * + * Name a face: a name another face of the namespace has joins it to that person (a merge: undo it with POST + * /faces/merges/{merge}/undo); any other name is its own (undo: POST /faces/{fid} with no name). + */ + public static nameFace( + options: Options, + ): RequestResult { + return (options.client ?? client).post({ + url: "/api/v1/faces/{fid}/name", + ...options, + headers: { + "Content-Type": "application/json", + ...options.headers, + }, + }); + } + /** * Merge Face * diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 50aefc63..d835614c 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -8685,6 +8685,21 @@ export type ExternalStart = { next?: string; }; +/** + * FaceAsset + */ +export type FaceAsset = { + /** + * Id + */ + id: number; + /** + * Title + */ + title?: string | null; + [key: string]: unknown; +}; + /** * FaceDismiss */ @@ -8724,6 +8739,45 @@ export type FaceMerged = { [key: string]: unknown; }; +/** + * FaceName + */ +export type FaceName = { + /** + * Name + * + * who it is; a name another face of the namespace has joins it to that person + */ + name: string; +}; + +/** + * FaceNamed + */ +export type FaceNamed = { + /** + * Ok + */ + ok?: boolean; + /** + * Name + */ + name: string; + /** + * Into + * + * the face of the same name it joined, if any + */ + into?: number | null; + /** + * Merge + * + * that merge, to undo with POST /faces/merges/{merge}/undo + */ + merge?: number | null; + [key: string]: unknown; +}; + /** * FaceRename */ @@ -8744,6 +8798,23 @@ export type FaceSpeaker = { speaker?: number | null; }; +/** + * FaceThumb + */ +export type FaceThumb = { + /** + * Recording + */ + recording: number; + /** + * Cover Url + * + * a signed link to the crop + */ + cover_url: string; + [key: string]: unknown; +}; + /** * FacesMode */ @@ -29681,6 +29752,84 @@ export type UnitCounts = { [key: string]: unknown; }; +/** + * UnnamedFace + */ +export type UnnamedFace = { + /** + * Id + */ + id: number; + /** + * Label + * + * what it's called until it's named, e.g. Face 3 + */ + label: string; + /** + * Guess + * + * a name to offer: the voice it's linked to, else one suggested for it + */ + guess?: string | null; + /** + * Speaker + * + * the voice it's linked to + */ + speaker?: number | null; + /** + * Appearances + */ + appearances: number; + /** + * Screen Ms + * + * time on screen in videos + */ + screen_ms: number; + /** + * Pages + * + * pages of documents and images it's on + */ + pages: number; + /** + * Thumbnails + */ + thumbnails: Array; + /** + * Recordings + * + * the assets it's in, where it's seen longest first + */ + recordings: Array; + [key: string]: unknown; +}; + +/** + * UnnamedFaces + */ +export type UnnamedFaces = { + /** + * Mode + * + * off, detect or recognize; faces are grouped across the namespace only in recognize + */ + mode: string; + /** + * Total + */ + total: number; + /** + * Groups + * + * the largest group first + */ + groups: Array; + [key: string]: unknown; +}; + /** * UpdateStatus */ @@ -45345,6 +45494,41 @@ export type GetNamespaceFacesResponses = { export type GetNamespaceFacesResponse = GetNamespaceFacesResponses[keyof GetNamespaceFacesResponses]; +export type GetUnnamedFacesData = { + body?: never; + path: { + /** + * Name + */ + name: string; + }; + query?: { + /** + * Limit + */ + limit?: number; + }; + url: "/api/v1/namespaces/{name}/faces/unnamed"; +}; + +export type GetUnnamedFacesErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type GetUnnamedFacesError = GetUnnamedFacesErrors[keyof GetUnnamedFacesErrors]; + +export type GetUnnamedFacesResponses = { + /** + * Successful Response + */ + 200: UnnamedFaces; +}; + +export type GetUnnamedFacesResponse = GetUnnamedFacesResponses[keyof GetUnnamedFacesResponses]; + export type SetNamespaceFacesModeData = { body: FacesMode; path: { @@ -45465,6 +45649,36 @@ export type RenameFaceResponses = { export type RenameFaceResponse = RenameFaceResponses[keyof RenameFaceResponses]; +export type NameFaceData = { + body: FaceName; + path: { + /** + * Fid + */ + fid: number; + }; + query?: never; + url: "/api/v1/faces/{fid}/name"; +}; + +export type NameFaceErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type NameFaceError = NameFaceErrors[keyof NameFaceErrors]; + +export type NameFaceResponses = { + /** + * Successful Response + */ + 200: FaceNamed; +}; + +export type NameFaceResponse = NameFaceResponses[keyof NameFaceResponses]; + export type MergeFaceData = { body: FaceMerge; path: { diff --git a/nextjs-frontend/components/recording/video/people-tab.tsx b/nextjs-frontend/components/recording/video/people-tab.tsx index fa22fd3f..48f30ace 100644 --- a/nextjs-frontend/components/recording/video/people-tab.tsx +++ b/nextjs-frontend/components/recording/video/people-tab.tsx @@ -27,6 +27,7 @@ import { cn } from "@/lib/utils"; type NsFace = { id: number; + name?: string | null; display?: string; speaker?: number | null; speaker_name?: string | null; @@ -44,6 +45,7 @@ export function PeopleTab({ noFaces, onPage }: { noFaces?: boolean; onPage?: (pa const nsFaces = useNamespaceFaces(ns, model.facesMode === "recognize"); const color = useFaceColors(); const byId = new Map(((nsFaces.data?.faces ?? []) as NsFace[]).map((f) => [f.id, f])); + const unnamed = [...byId.values()].filter((f) => !f.name).length; if (model.facesMode === "off") return ( @@ -99,6 +101,14 @@ export function PeopleTab({ noFaces, onPage }: { noFaces?: boolean; onPage?: (pa /> ))} + {model.facesMode === "recognize" && unnamed > 0 && ns && ( + + Name the {unnamed === 1 ? "unnamed face" : `${unnamed} unnamed faces`} in {ns}, the most seen first + + )} ); } diff --git a/nextjs-frontend/components/speakers/name-faces.tsx b/nextjs-frontend/components/speakers/name-faces.tsx new file mode 100644 index 00000000..deda6007 --- /dev/null +++ b/nextjs-frontend/components/speakers/name-faces.tsx @@ -0,0 +1,240 @@ +"use client"; + +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { ScanFace } from "lucide-react"; +import Link from "next/link"; +import { useEffect, useRef, useState, type FormEvent } from "react"; + +import { Video } from "@/app/openapi-client"; +import type { FaceNamed, UnnamedFace } from "@/app/openapi-client/types.gen"; +import { useBlur } from "@/components/nsfw/nsfw"; +import { screenTime } from "@/components/recording/video/model"; +import { Banner } from "@/components/ui/banner"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/field"; +import { EmptyState, SkeletonRows } from "@/components/ui/states"; +import { useToast } from "@/components/ui/toast"; +import { ApiError, data, useApiClient } from "@/lib/api/browser"; +import { plural } from "@/lib/format"; +import { needRole, useArchive } from "@/lib/hooks/session"; +import { cn } from "@/lib/utils"; + +/** The namespace's faces nobody has named yet, the largest group first (only for people with a role in it). Under + * ["faces", ns], so whatever changes the namespace's faces refreshes it too. */ +export function useUnnamedFaces(ns: string | null | undefined) { + const client = useApiClient(); + const { can } = useArchive(); + return useQuery({ + queryKey: ["faces", ns, "unnamed"], + queryFn: () => data(Video.getUnnamedFaces({ client, path: { name: ns as string }, query: { limit: 500 } })), + enabled: Boolean(ns) && can("viewer", ns), + staleTime: 30_000, + }); +} + +/** + * Name the namespace's unnamed faces one after another: the first one's name field has focus (with the name Lens + * would offer, selected so typing replaces it), Enter names it and moves on to the next, and the notice that follows + * has Undo. A name another face has joins that person (a merge); any other name becomes the face's own. + */ +export function NameFaces({ ns }: { ns: string }) { + const client = useApiClient(); + const qc = useQueryClient(); + const toast = useToast(); + const { can } = useArchive(); + const q = useUnnamedFaces(ns); + const canName = can("editor", ns); + const [named, setNamed] = useState>(new Set()); // named here, until the list comes back without them + const [active, setActive] = useState(null); + const groups = (q.data?.groups ?? []).filter((g) => !named.has(g.id)); + const current = active != null && groups.some((g) => g.id === active) ? active : (groups[0]?.id ?? null); + const refresh = () => qc.invalidateQueries({ queryKey: ["faces", ns] }); + const fail = (title: string) => (e: unknown) => + toast({ title, body: e instanceof ApiError ? e.message : "Please try again.", tone: "red" }); + const show = (fid: number, on: boolean) => + setNamed((s) => { + const next = new Set(s); + if (on) next.delete(fid); + else next.add(fid); + return next; + }); + + const undo = useMutation({ + mutationFn: ({ g, r }: { g: UnnamedFace; r: FaceNamed }) => + r.merge != null + ? data(Video.undoFaceMerge({ client, path: { mid: r.merge } })) + : data(Video.renameFace({ client, path: { fid: g.id }, body: { name: "" } })), + onSuccess: (_r, { g }) => { + show(g.id, true); + setActive(g.id); + void refresh(); + }, + onError: fail("Couldn’t undo it"), + }); + const name = useMutation({ + mutationFn: ({ g, value }: { g: UnnamedFace; value: string; next: number | null }) => + data(Video.nameFace({ client, path: { fid: g.id }, body: { name: value } })), + onSuccess: (r, { g, next }) => { + show(g.id, false); + setActive(next); + void refresh(); + toast({ + title: r.merge != null ? `${g.label} joined ${r.name}` : `${g.label} is ${r.name}`, + tone: "green", + action: { label: "Undo", onClick: () => undo.mutate({ g, r }) }, + }); + }, + onError: fail("Couldn’t name it"), + }); + + if (q.isLoading) return ; + if (q.isError) + return ( + q.refetch()}> + Try again + + } + > + {q.error.message} + + ); + if (q.data?.mode !== "recognize") + return ( + } + title={`Faces aren’t recognised in ${ns}`} + className="rounded-lg border border-border" + > + Faces are grouped across a namespace, to be named, only where an owner has turned face recognition on (in a + video’s People tab). Nothing changes until they do. + + ); + if (!groups.length) + return ( + } title="Every face has a name" className="rounded-lg border border-border"> + Faces found from now on that aren’t anyone {ns} knows wait here for a name. + + ); + return ( +
+

+ {plural(groups.length, "face")} without a name, the most seen first. Type who it is and press Enter for the + next; a name someone already has joins that person. +

+
    + {groups.map((g, i) => ( + setActive(g.id)} + onName={(value) => name.mutate({ g, value, next: (groups[i + 1] ?? groups[i - 1])?.id ?? null })} + /> + ))} +
+
+ ); +} + +function FaceGroup({ + g, + active, + disabled, + pending, + onFocus, + onName, +}: { + g: UnnamedFace; + active: boolean; + /** Why this person can't name faces, when they can't. */ + disabled?: string; + pending: boolean; + onFocus: () => void; + onName: (name: string) => void; +}) { + const blur = useBlur(); + const [value, setValue] = useState(g.guess ?? ""); + const input = useRef(null); + useEffect(() => { + if (active && !disabled) input.current?.focus(); + }, [active, disabled]); + const submit = (e: FormEvent) => { + e.preventDefault(); + if (value.trim()) onName(value.trim()); + }; + const seen = [ + plural(g.recordings.length, "asset"), + g.screen_ms ? `${screenTime(g.screen_ms)} on screen` : null, + g.pages ? plural(g.pages, "page") : null, + ] + .filter(Boolean) + .join(" · "); + return ( +
  • +
    + {g.thumbnails.map((t, k) => ( + // eslint-disable-next-line @next/next/no-img-element + + ))} +
    +
    +
    {g.label}
    +
    {seen}
    +
    + {g.recordings.slice(0, 3).map((r, k) => ( + + {k > 0 && · } + + {r.title || `Asset ${r.id}`} + + + ))} + {g.recordings.length > 3 && · {g.recordings.length - 3} more} +
    +
    +
    + setValue(e.target.value)} + onFocus={(e) => { + e.target.select(); + onFocus(); + }} + maxLength={80} + placeholder="Who is it?" + aria-label={`Name for ${g.label}`} + disabled={Boolean(disabled) || pending} + className="h-9 min-w-0 flex-1 text-[13.5px] sm:w-[220px]" + /> + +
    +
  • + ); +} diff --git a/nextjs-frontend/components/speakers/speakers-page.tsx b/nextjs-frontend/components/speakers/speakers-page.tsx index bae5faf6..0cf36ca7 100644 --- a/nextjs-frontend/components/speakers/speakers-page.tsx +++ b/nextjs-frontend/components/speakers/speakers-page.tsx @@ -6,6 +6,7 @@ import { useMemo } from "react"; import { useRecordingIndex } from "@/components/search/data"; import { MergeHistory, OtherNamespaces } from "@/components/speakers/history"; +import { NameFaces, useUnnamedFaces } from "@/components/speakers/name-faces"; import { useNamespaceSpeakers } from "@/components/speakers/parts"; import { lastHeard, reviewPairs } from "@/components/speakers/derive"; import { Registry } from "@/components/speakers/registry"; @@ -19,10 +20,11 @@ import { Tabs } from "@/components/ui/tabs"; import { plural } from "@/lib/format"; import { useArchive } from "@/lib/hooks/session"; -const TABS = ["registry", "review", "history", "links"] as const; +const TABS = ["registry", "review", "faces", "history", "links"] as const; type Tab = (typeof TABS)[number]; -/** SP1–SP3: the speakers of one namespace — registry, review queue, merge history, links to other namespaces. */ +/** SP1–SP3: the speakers of one namespace — registry, review queue, merge history, links to other namespaces — and, + * where it recognises faces, the faces waiting for a name. */ export function SpeakersPage() { const params = useSearchParams(); const router = useRouter(); @@ -36,6 +38,8 @@ export function SpeakersPage() { ? (params.get("tab") as Tab) : "registry"; const dir = useNamespaceSpeakers(ns); + const unnamed = useUnnamedFaces(ns); + const faces = unnamed.data?.mode === "recognize" || tab === "faces"; // the tab shows where faces are recognised const index = useRecordingIndex(); const set = (k: string, v: string | null) => { @@ -102,6 +106,9 @@ export function SpeakersPage() { label: "Review queue", count: dir.data ? pairs.length : undefined, }, + ...(faces + ? [{ value: "faces", label: "Unnamed faces", count: unnamed.data ? unnamed.data.total : undefined }] + : []), { value: "history", label: "Merge history" }, { value: "links", @@ -146,6 +153,7 @@ export function SpeakersPage() { )} {dir.data && ns && tab === "review" && } + {ns && tab === "faces" && } {dir.data && ns && tab === "history" && } {dir.data && ns && tab === "links" && } diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index 0f49f556..96473a88 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -22535,6 +22535,61 @@ } } }, + "/api/v1/namespaces/{name}/faces/unnamed": { + "get": { + "tags": [ + "video" + ], + "summary": "Get Unnamed Faces", + "description": "The faces nobody has named yet, grouped across the namespace (recognize mode), the largest group first: each\nwith a few signed crops, the assets it's in and a name to offer. Name them with POST /faces/{fid}/name.", + "operationId": "get_unnamed_faces", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string", + "title": "Name" + } + }, + { + "name": "limit", + "in": "query", + "required": false, + "schema": { + "type": "integer", + "maximum": 500, + "minimum": 1, + "default": 100, + "title": "Limit" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/UnnamedFaces" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/namespaces/{name}/faces/mode": { "put": { "tags": [ @@ -22723,6 +22778,59 @@ } } }, + "/api/v1/faces/{fid}/name": { + "post": { + "tags": [ + "video" + ], + "summary": "Name Face", + "description": "Name a face: a name another face of the namespace has joins it to that person (a merge: undo it with POST\n/faces/merges/{merge}/undo); any other name is its own (undo: POST /faces/{fid} with no name).", + "operationId": "name_face", + "parameters": [ + { + "name": "fid", + "in": "path", + "required": true, + "schema": { + "type": "integer", + "title": "Fid" + } + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FaceName" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/FaceNamed" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/faces/{fid}/merge": { "post": { "tags": [ @@ -52836,6 +52944,31 @@ "type": "object", "title": "ExternalStart" }, + "FaceAsset": { + "properties": { + "id": { + "type": "integer", + "title": "Id" + }, + "title": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Title" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id" + ], + "title": "FaceAsset" + }, "FaceDismiss": { "properties": { "kind": { @@ -52884,6 +53017,64 @@ ], "title": "FaceMerged" }, + "FaceName": { + "properties": { + "name": { + "type": "string", + "title": "Name", + "description": "who it is; a name another face of the namespace has joins it to that person" + } + }, + "additionalProperties": false, + "type": "object", + "required": [ + "name" + ], + "title": "FaceName" + }, + "FaceNamed": { + "properties": { + "ok": { + "type": "boolean", + "title": "Ok", + "default": true + }, + "name": { + "type": "string", + "title": "Name" + }, + "into": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Into", + "description": "the face of the same name it joined, if any" + }, + "merge": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Merge", + "description": "that merge, to undo with POST /faces/merges/{merge}/undo" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "name" + ], + "title": "FaceNamed" + }, "FaceRename": { "properties": { "name": { @@ -52914,6 +53105,26 @@ "type": "object", "title": "FaceSpeaker" }, + "FaceThumb": { + "properties": { + "recording": { + "type": "integer", + "title": "Recording" + }, + "cover_url": { + "type": "string", + "title": "Cover Url", + "description": "a signed link to the crop" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "recording", + "cover_url" + ], + "title": "FaceThumb" + }, "FacesMode": { "properties": { "mode": { @@ -88958,6 +89169,113 @@ ], "title": "UnitCounts" }, + "UnnamedFace": { + "properties": { + "id": { + "type": "integer", + "title": "Id" + }, + "label": { + "type": "string", + "title": "Label", + "description": "what it's called until it's named, e.g. Face 3" + }, + "guess": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Guess", + "description": "a name to offer: the voice it's linked to, else one suggested for it" + }, + "speaker": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Speaker", + "description": "the voice it's linked to" + }, + "appearances": { + "type": "integer", + "title": "Appearances" + }, + "screen_ms": { + "type": "integer", + "title": "Screen Ms", + "description": "time on screen in videos" + }, + "pages": { + "type": "integer", + "title": "Pages", + "description": "pages of documents and images it's on" + }, + "thumbnails": { + "items": { + "$ref": "#/components/schemas/FaceThumb" + }, + "type": "array", + "title": "Thumbnails" + }, + "recordings": { + "items": { + "$ref": "#/components/schemas/FaceAsset" + }, + "type": "array", + "title": "Recordings", + "description": "the assets it's in, where it's seen longest first" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "label", + "appearances", + "screen_ms", + "pages", + "thumbnails", + "recordings" + ], + "title": "UnnamedFace" + }, + "UnnamedFaces": { + "properties": { + "mode": { + "type": "string", + "title": "Mode", + "description": "off, detect or recognize; faces are grouped across the namespace only in recognize" + }, + "total": { + "type": "integer", + "title": "Total" + }, + "groups": { + "items": { + "$ref": "#/components/schemas/UnnamedFace" + }, + "type": "array", + "title": "Groups", + "description": "the largest group first" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "mode", + "total", + "groups" + ], + "title": "UnnamedFaces" + }, "UpdateStatus": { "properties": { "current": {