Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,8 +286,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[prompt injection](docs/assistant.md#third-party-content-and-prompt-injection), [HTTPS](docs/https.md),
[remote access](docs/remote-access.md); [checks](docs/requirements.md#stay-private-and-safe))
- **Use open standards.** RDF and Dublin Core for every record, OAuth for apps and OpenID Connect sign-in, MCP, MQTT,
IIIF for publishing, OpenTelemetry and Prometheus for monitoring, and an optional Fedora mirror.
([Linked data](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md);
IIIF for publishing, OpenTelemetry and Prometheus for monitoring, an optional Fedora mirror, and a CSV catalogue of
the assets you select in the Library.
([Linked data](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md),
[catalogue](docs/processing.md#finding-by-any-attribute);
[checks](docs/requirements.md#use-open-standards))

## Light by default
Expand Down
3 changes: 3 additions & 0 deletions changelog.d/catalogue-csv.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
- **A catalogue of what you select.** Library › Export › Catalogue of all selected (.csv) saves one spreadsheet row per
asset (title, namespace, collection, kind, date, length, people, tags, language, status, access, origin and a link),
with formula-like cells defused ([Finding by any attribute](docs/processing.md#finding-by-any-attribute)).
6 changes: 6 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,12 @@ first; otherwise the newest. Nothing here calls a model, apart from `text` searc
The Library list (`GET /resources`) takes the same attributes as ids: `face`, `entity`, `topic` (as it is, without its
narrower topics), `content_type` and `emotion`, besides its other filters ([API](api.md#resources)).

**A catalogue to keep.** Select assets in the Library and choose Export › Catalogue of all selected (.csv): one
spreadsheet row per asset with its id, title, namespace, collection, kind, date, length, words, the people named in
it, tags, language, status, access, where it came from and a link to its page. It is made in the browser from the rows
already listed, so it costs nothing; a cell that would start a spreadsheet formula gets a leading apostrophe. Agents
get the same rows as JSON from `GET /resources`.

Planned, not built yet: these filters in the Library's filter bar and in saved collections, finding by what a photo
shows (`GET /search/photos`) inside the same filter, and ranking by more than the best matching moment.

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,7 @@ request, to another check that proves the same thing; loosening a requirement or
| IIIF for publishing | [test_iiif.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_iiif.py) `test_manifests_collections_search_state_discovery`, `test_authorization_flow` |
| OpenTelemetry and Prometheus for monitoring | [test_telemetry.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_telemetry.py) `test_traces_and_metrics_go_to_the_endpoint`; [test_prometheus.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_prometheus.py) `test_what_it_measures` |
| An optional Fedora mirror | [test_fedora.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_fedora.py) `test_off_by_default`, `test_sync_sends_changes_and_deletes` |
| A CSV catalogue of the assets selected in the Library, one row each with a link to its page, safe to open in a spreadsheet | [library-catalogue.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/library-catalogue.test.ts) `has a row per asset with its people, tags and a link to its page`, `quotes what needs it and never lets a cell run as a formula` |

## Light by default

Expand Down
65 changes: 65 additions & 0 deletions nextjs-frontend/__tests__/library-catalogue.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import type { RecordingSummary } from "@/app/openapi-client/types.gen";
import { catalogueRows, CATALOGUE_COLUMNS } from "@/components/library/model";
import { csvCell, toCsvText } from "@/lib/csv";

const row = (r: Partial<RecordingSummary>): RecordingSummary =>
({ id: 1, space: 1, media_kind: "audio", ...r }) as RecordingSummary;

describe("the library's catalogue CSV", () => {
it("has a row per asset with its people, tags and a link to its page", () => {
const rows = catalogueRows(
[
row({
id: 7,
title: "Call with Siobhan, part 2",
namespace: "family",
collection_name: "Calls",
recorded_at: "2026-09-30T10:00:00Z",
duration_ms: 3_725_000,
words: 512,
speakers: "Siobhan, Speaker 2, Aoife",
tags: ["school", "trip"],
language: "en",
status: "analyzed",
access: "private",
origin_name: "Phone",
}),
row({ id: 8, title: "Scan", media_kind: "document", pages: 3 }),
],
"https://lens.example/",
);
expect(rows[0]).toEqual([...CATALOGUE_COLUMNS]);
expect(rows[1]).toEqual([
"7",
"Call with Siobhan, part 2",
"family",
"Calls",
"audio",
"2026-09-30T10:00:00Z",
"1:02:05",
"512",
"Siobhan; Aoife",
"school; trip",
"en",
"analyzed",
"private",
"Phone",
"https://lens.example/resources/7",
]);
expect(rows[2].slice(0, 7)).toEqual(["8", "Scan", "", "", "document", "", "3 pages"]);
expect(toCsvText(rows).split("\n")[1]).toContain('"Call with Siobhan, part 2"');
});

it("quotes what needs it and never lets a cell run as a formula", () => {
expect(csvCell('say "hi"')).toBe('"say ""hi"""');
expect(csvCell("=HYPERLINK(1)")).toBe("'=HYPERLINK(1)");
expect(csvCell("@sum, x")).toBe('"\'@sum, x"');
expect(csvCell("-3")).toBe("-3");
expect(
toCsvText([
["a", "b"],
["1", "2"],
]),
).toBe("a,b\n1,2\n");
});
});
7 changes: 2 additions & 5 deletions nextjs-frontend/components/admin/audit-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
* filtering, and CSV export. The backend records who, what and on what; secrets are never in it.
*/
import { FIELDS, SECTIONS } from "@/components/settings/model";
import { toCsvText } from "@/lib/csv";

export type AuditEntry = {
at: string;
Expand Down Expand Up @@ -160,10 +161,6 @@ export function filterEntries(entries: AuditEntry[], f: Filters, now = Date.now(
);
}

function csvCell(v: string): string {
return /[",\n]/.test(v) ? `"${v.replace(/"/g, '""')}"` : v;
}

export function toCsv(entries: AuditEntry[], look: Lookup): string {
const rows = [["time", "person", "email", "action", "target", "details"]];
for (const e of entries)
Expand All @@ -175,5 +172,5 @@ export function toCsv(entries: AuditEntry[], look: Lookup): string {
targetText(e.target, e.action, look),
detailText(e, look),
]);
return rows.map((r) => r.map(csvCell).join(",")).join("\n") + "\n";
return toCsvText(rows);
}
16 changes: 14 additions & 2 deletions nextjs-frontend/components/library/actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ import { useSession } from "next-auth/react";
import { useCallback } from "react";

import { Jobs, Resources } from "@/app/openapi-client";
import type { RecordingSummary } from "@/app/openapi-client/types.gen";
import { useToast } from "@/components/ui/toast";
import { deletedToast, hasSound, movedToast } from "@/components/library/model";
import { catalogueRows, deletedToast, hasSound, movedToast } from "@/components/library/model";
import { ApiError, data, useApiClient } from "@/lib/api/browser";
import { saveText, toCsvText } from "@/lib/csv";
import { plural } from "@/lib/format";

export const EXPORT_FORMATS = [
Expand Down Expand Up @@ -157,6 +159,16 @@ export function useRecordingActions() {
[toast, token],
);

/** The selected rows as one catalogue CSV, made here from what the list already has: nothing more is fetched. */
const exportCatalogue = useCallback(
(rows: RecordingSummary[]) => {
if (!rows.length) return;
saveText(toCsvText(catalogueRows(rows, window.location.origin)), "lens-catalogue.csv");
toast({ title: `Saved a catalogue of ${plural(rows.length, "asset")}`, tone: "green" });
},
[toast],
);

/** Delete recordings one by one (owners). Returns the ids that went. */
const deleteMany = useCallback(
async (recs: readonly { id: number; title?: string | null }[], onProgress?: (n: number) => void) => {
Expand Down Expand Up @@ -238,5 +250,5 @@ export function useRecordingActions() {
[client, qc, refresh, toast],
);

return { retryJob, reprocess, exportMany, deleteMany, moveMany, retag };
return { retryJob, reprocess, exportMany, exportCatalogue, deleteMany, moveMany, retag };
}
11 changes: 10 additions & 1 deletion nextjs-frontend/components/library/bulk-bar.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { hasSound, tagsFromText, tagsOn } from "@/components/library/model";
import { Button } from "@/components/ui/button";
import { Dialog } from "@/components/ui/dialog";
import { Checkbox, Field, Input, Select } from "@/components/ui/field";
import { Menu, MenuContent, MenuItem, MenuLabel, MenuTrigger } from "@/components/ui/menu";
import { Menu, MenuContent, MenuItem, MenuLabel, MenuSeparator, MenuTrigger } from "@/components/ui/menu";
import { Tooltip } from "@/components/ui/tooltip";
import { count, plural } from "@/lib/format";
import { cn } from "@/lib/utils";
Expand Down Expand Up @@ -62,6 +62,7 @@ export function BulkBar({
moveTargets,
onReprocess,
onExport,
onCatalogue,
onMove,
onPlace,
placeReason,
Expand All @@ -79,6 +80,8 @@ export function BulkBar({
moveTargets: string[];
onReprocess: () => void;
onExport: (fmt: ExportFormat) => void;
/** Save the selection as one catalogue spreadsheet (CSV). */
onCatalogue?: () => void;
onMove: () => void;
/** File them in a collection of their namespace. */
onPlace?: () => void;
Expand Down Expand Up @@ -145,6 +148,12 @@ export function BulkBar({
{f.label}
</MenuItem>
))}
{onCatalogue && (
<>
<MenuSeparator />
<MenuItem onSelect={onCatalogue}>Catalogue of all selected (.csv)</MenuItem>
</>
)}
</MenuContent>
</Menu>
<BarButton icon={Trash2} label="Delete" danger onClick={onDelete} disabledReason={deleteReason} />
Expand Down
1 change: 1 addition & 0 deletions nextjs-frontend/components/library/library-screen.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -660,6 +660,7 @@ export function LibraryScreen({
moveTargets={targets}
onReprocess={() => setReprocessOpen(true)}
onExport={(fmt) => actions.exportMany([...selected], fmt)}
onCatalogue={() => actions.exportCatalogue(selectedRows)}
onMove={() => setMoveOpen(true)}
onPlace={() => setPlaceOpen(true)}
placeReason={placeReason}
Expand Down
49 changes: 49 additions & 0 deletions nextjs-frontend/components/library/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -510,3 +510,52 @@ export function tagsOn(rows: readonly { tags?: string[] | null }[]): { tag: stri
}
return [...by.values()].sort((a, b) => b.count - a.count || a.tag.localeCompare(b.tag));
}

/** The catalogue columns of a CSV of library rows, in order. */
export const CATALOGUE_COLUMNS = [
"id",
"title",
"namespace",
"collection",
"kind",
"recorded",
"length",
"words",
"people",
"tags",
"language",
"status",
"access",
"origin",
"link",
] as const;

/**
* Library rows as a catalogue, one row per asset, for a spreadsheet: what the table shows, with people and tags joined
* by semicolons and a link to each asset's page on `origin` (this Lens's address).
*/
export function catalogueRows(rows: RecordingSummary[], origin: string): string[][] {
const out: string[][] = [[...CATALOGUE_COLUMNS]];
for (const r of rows)
out.push([
String(r.id),
r.title ?? "",
r.namespace ?? "",
r.collection_name ?? "",
r.media_kind,
r.recorded_at ?? "",
lengthText(r) === "—" ? "" : lengthText(r),
r.words != null ? String(r.words) : "",
speakerList(r.speakers)
.filter((s) => !s.unnamed)
.map((s) => s.name)
.join("; "),
(r.tags ?? []).join("; "),
r.language ?? "",
r.status ?? "",
r.access ?? "",
r.origin_name ?? r.origin ?? "",
`${origin.replace(/\/$/, "")}/resources/${r.id}`,
]);
return out;
}
26 changes: 26 additions & 0 deletions nextjs-frontend/lib/csv.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/**
* CSV for spreadsheets (RFC 4180): a cell with a comma, quote or line break is quoted. A cell that starts the way a
* formula does (=, +, @, a tab or a carriage return) gets a leading apostrophe, so a title someone else wrote can't run
* as a formula when the file is opened.
*/
export function csvCell(v: string): string {
const safe = /^[=+@\t\r]/.test(v) ? `'${v}` : v;
return /[",\n\r]/.test(safe) ? `"${safe.replace(/"/g, '""')}"` : safe;
}

/** Rows of cells as one CSV file's text, ending with a line break. */
export function toCsvText(rows: string[][]): string {
return rows.map((r) => r.map(csvCell).join(",")).join("\n") + "\n";
}

/** Hand text to the browser as a file to save. */
export function saveText(text: string, filename: string, type = "text/csv;charset=utf-8"): void {
const href = URL.createObjectURL(new Blob([text], { type }));
const a = document.createElement("a");
a.href = href;
a.download = filename;
document.body.appendChild(a);
a.click();
a.remove();
setTimeout(() => URL.revokeObjectURL(href), 10_000);
}
Loading