Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,11 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
file or a page is as it comes in (a podcast, an interview, a scan), in Import, by API or to the assistant, and its
content type's pipeline runs from the start. A file, page or video the namespace already has links to the one there
instead of being imported again. Media browsers can't play, such as an AVI or HEVC from a phone, can get a copy they
can (off by default).
can (off by default). A bulk import (a folder scan, an archive's files, a link list, a source's files) is one run:
the toast after it, and the Import screen's list of recent runs, undo it whole.
([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika),
[import webhooks](docs/api.md#import-webhooks), [playable copies](docs/video.md#playable-copies);
[import webhooks](docs/api.md#import-webhooks), [playable copies](docs/video.md#playable-copies),
[import runs](docs/api.md#import-runs);
[checks](docs/requirements.md#bring-in-everything))
- **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a
part you pick into Markdown, lets you look it over, and sends it to a namespace. It signs in to Lens with nothing to
Expand Down
8 changes: 8 additions & 0 deletions changelog.d/import-runs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- **A bulk import is one run you can undo whole.** A folder scan, a zip or tar archive's files, a list of links or a
bookmarks file, and chosen files of a source each make one import run, and every asset it makes carries the run's
id (`import_run`). `GET /api/v1/import/runs` lists recent runs with how many assets each made and how many are left;
`POST /api/v1/import/runs/{id}/undo` deletes them all through the normal delete, needing owner rights on each
(nothing goes when any isn't yours to delete), audited per asset and for the run. The toast after a link or source
import has Undo, the Import screen lists recent runs with Undo on each, and the assistant has `import_runs` and
`undo_import` (always after your yes). Assets imported before keep no run, and nothing changes for them.
([Import runs](docs/api.md#import-runs))
34 changes: 32 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1046,6 +1046,9 @@ POST /api/v1/import
POST /api/v1/import/preview
POST /api/v1/import/web
POST /api/v1/import/web/links
GET /api/v1/import/runs
GET /api/v1/import/runs/{id}
POST /api/v1/import/runs/{id}/undo
```

`POST /import` queues the namespace's pipeline after the import, or the saved pipeline named by `pipeline` (any of
Expand All @@ -1068,7 +1071,33 @@ pipeline, or `pipeline`; transcripts are imported. A PDF, Word, text or Markdown
`documents_as` is `transcript` (its text only), or the server can't make a PDF of it. Each path gets a result: `queued` (with `recording` and `job`), `already` (the namespace has it from this
source, or, for a folder on this machine, has the same file from anywhere: another folder, an upload or a scan, by
what's in it, so it isn't read twice; with that `recording`), `skipped` (a folder, or not audio, video, a document, an image or a transcript) or `error`. Choosing a file
whose asset was deleted brings it back. Audited as `import.source`.
whose asset was deleted brings it back. The files queued are one [import run](#import-runs) (`import_run`). Audited
as `import.source`.

### Import runs

A bulk import is one import run, so a mistaken one is undone whole rather than asset by asset: a folder scan that
finds new files (`scan`), a zip or tar archive whose files become resources of their own (`archive`: the archive and
those resources), a list of links or a bookmarks file (`links`, `POST /import/web/links`) and chosen files of a source
(`source`, `POST /import/source`). Every asset a run made carries its id as `import_run` (`GET /recordings/{id}`), and
the import's answer names it (`import_run`). Any other upload, a pasted transcript or one web page makes no run,
and assets imported before runs existed have none.

`GET /api/v1/import/runs?limit=` lists recent runs, newest first: the ones you started and those in namespaces where
you may delete assets (everyone's, for admins), each with `kind`, `namespace`, `source` (what it read: the folders,
the archive, the links' file, the source and how many files), `created_by`, `created_at`, `assets` (how many it made),
`left` (how many are still here) and `status` (`imported`, `partly undone` or `undone`). `GET /import/runs/{id}` reads
one (404 when it isn't one you'd see listed).

`POST /api/v1/import/runs/{id}/undo` deletes every asset the run made that is still here, each exactly as
`DELETE /recordings/{id}` deletes one (the media files stay where they are, and scans and watched folders don't import
them again). It needs the rights a delete needs on each one (the remove permission in its namespace, owners, or admin
of its collection): when any is one you may not delete, it answers 403 and nothing is deleted. An asset a job is
working on is left (`busy`) and its job asked to stop; undoing the run again takes the rest. It answers `{run,
deleted, busy}`. Lens keeps no recently deleted list of assets, so an undo can't itself be undone: importing the files
again brings them back. Audited as `recording.delete` for each asset (with the run) and `import.undo` for the run. The
Import screen lists recent runs with Undo on each, and the toast after a bulk import has Undo for the whole run; the
assistant has `import_runs` and `undo_import`, which waits for your approval.

### Web pages

Expand Down Expand Up @@ -1098,7 +1127,8 @@ name until it's captured, and tagged with the bookmark folders it was in (the in
own: Bookmarks bar, Other bookmarks), unless `folders_as_tags` is false. At most 500 links at once (400 beyond); links
are taken in order, without repeats. It answers with each link's `status`: `queued` (with its `recording` and `job`),
`already` (a page captured from that address is in the namespace: its `recording`) or `skipped` (`detail` says why,
as the checks above would). Audited as `import.web` once, with how many were queued.
as the checks above would). The pages queued are one [import run](#import-runs) (`import_run`). Audited as
`import.web` once, with how many were queued.

### Videos and podcasts

Expand Down
3 changes: 3 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,9 @@ When the configured model supports function calling, chat becomes an agent.
bookmark export) and `import_urls` captures each page as a document, as `POST /api/v1/import/web/links` does. Like
`import_files`, it picks the namespace when you don't name one, takes a content type when you say what the pages
are, and waits for your approval.
- **Undo an import:** `import_runs` lists your recent bulk imports (a folder scan, an archive's files, a list of links,
a source's files) and `undo_import` deletes everything one of them made, as the Import screen's Undo does
([Import runs](api.md#import-runs)). It needs owner rights on each asset and always waits for your approval.
- **Keep typing:** what you send while an answer is being written waits its turn and goes next.
- **Your browser** (Chrome and Edge): asked from the Lens extension's side panel, the assistant can open, read, click
and type in that browser and download files into the conversation, on sites you allow, asking before anything that
Expand Down
3 changes: 2 additions & 1 deletion docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@ as missing on older rows: a chat message's record (`meta`,
[What each answer records](assistant.md#what-each-answer-records)) is one, and messages from before it simply have
none. [Deep research](assistant.md#deep-research) is another: its tables (`research`, and `research_run` for the claim
of the process running one) are new, a message's `kind` is missing on older ones, and notes accept `chat` links
without rewriting any. Anything else, such as rewriting stored values, moving data between
without rewriting any. [Import runs](api.md#import-runs) are a third: the `import_run` table and the index on a
recording's `import_run` are new, and assets imported before have no run. Anything else, such as rewriting stored values, moving data between
tables, or changing or removing an index that already exists, is a named step in `app/domain/migrations.py`:

* Every process that opens the database (the API, workers, `lens watch`, `lens` commands) runs the steps the database
Expand Down
6 changes: 4 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ How assets move through Lens: where they come from, what each step does, and how

- `scan` finds audio under each namespace's paths and fingerprints it: moved files keep their history, duplicates are skipped,
and so are files whose asset someone deleted or moved to another namespace (at the same path, or a copy of the same
file).
file). What one namespace's scan brings in is one [import run](api.md#import-runs), undone whole.
- `clean_audio` (only in a pipeline that has it) cleans up the sound before it's transcribed: noise, volume, rumble,
echo, and silence left out of what's transcribed ([Cleaning up audio](#cleaning-up-audio)).
- `transcribe` uses SenseVoice, faster-whisper or mlx-whisper. When the configured engine isn't installed on a worker (or
Expand Down Expand Up @@ -158,7 +158,9 @@ it, and its files are kept as its files, as an email's attachments are: those Le
their own that say which archive they came from, a chat export in it a chat and JSON in it records (so a WhatsApp
export with media is the chat and its photos). Only plain files are taken, an encrypted entry or a link is left
out, an archive inside one is kept but not opened, and reading stops at 100 files or 512 MB unpacked, so a zip bomb
can't fill the disk. What was left out, and why, is on its page. Archives get the Archive content type.
can't fill the disk. What was left out, and why, is on its page. Archives get the Archive content type. The archive
and the resources made of its files are one [import run](api.md#import-runs), so Undo on the Import screen takes
them all away again.

**Records.** JSON and JSON Lines that aren't a transcript (their rows have fields a transcript's lines don't, like an
order's `total`) are read as records, one line each: a JSON Lines file a record per line, a JSON file its list (or the
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ request, to another check that proves the same thing; loosening a requirement or
| A file, page or video the namespace already has links to the one there instead of being imported again: a copy in a watched folder on this machine, a page or video address imported twice | [test_uploads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_uploads.py) `test_a_copy_in_a_folder_on_this_machine_links_to_the_one_there`; [test_video_urls.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video_urls.py) `test_an_address_imported_again_links_to_what_the_namespace_has` |
| A video's own captions, written by people, become its transcript instead of transcribing it, off by default | [test_video_urls.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video_urls.py) `test_a_videos_own_captions_become_its_transcript` |
| Videos and podcasts imported by their address, off by default | [test_video_urls.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video_urls.py) `test_off_by_default`, `test_a_video_by_its_address_is_downloaded_then_transcribed`; [import-web.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/import-web.test.tsx) `imports a video or podcast by its address, when asked to` |
| A bulk import (a folder scan, an archive's files, a link list, a source's files) is one run, listed and undone whole by its owners, by API, the Import screen or the assistant | [test_import_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_runs.py) `test_a_link_import_is_one_run_undone_whole`, `test_an_uploaded_archive_and_its_files_are_one_run`, `test_a_folder_scan_is_one_run`, `test_chosen_files_of_a_source_are_one_run`, `test_the_assistant_lists_and_undoes_an_import_with_approval`; [import-runs.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/import-runs.test.tsx) |

## Clip from the browser

Expand Down
93 changes: 83 additions & 10 deletions fastapi_backend/app/api/v1/routes/imports.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""Importing transcripts: an uploaded file (base64 in JSON) or pasted text, and a preview that saves nothing; and
chosen files of a storage source."""
chosen files of a storage source. A bulk import is one import run, listed and undone whole (import_runs.py)."""

from __future__ import annotations

Expand All @@ -11,16 +11,32 @@
from contextlib import contextmanager
from typing import Any

from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, BackgroundTasks, HTTPException, Query, Request

from app.api.deps import Acl, AdminWriter, Cfg, Db, Writer, domain_errors
from app.domain import auth, content_types, convert, ingest, jobs, links, mediaurl, pipelines, sources, store, webcapture
from app.api.deps import Access, Acl, AdminWriter, Cfg, CurrentUser, Db, Principal, Writer, domain_errors
from app.domain import (
auth,
content_types,
convert,
import_runs,
ingest,
jobs,
links,
mediaurl,
pipelines,
render,
sources,
store,
webcapture,
)
from app.domain.store import DB
from app.schemas.imports import (
ImportPreview,
ImportPreviewRequest,
ImportRequest,
ImportResult,
ImportRun,
ImportRunUndone,
LinkImportResult,
LinksImport,
LinksImportRequest,
Expand Down Expand Up @@ -245,11 +261,13 @@ def import_web_links(body: LinksImportRequest, acl: Acl, user: Writer, db: Db, c
if len(found) > links.MAX:
raise HTTPException(400, f"import {links.MAX} links at most at once; this has {len(found)}")
sid = sid if sid is not None else store.ns_id(db, ns)
queued = links.queue(db, cfg, sid, found, user.email, body.collection, body.pipeline, body.folders_as_tags, body.content_type)
out = [LinkImportResult(**r) for r in queued]
said = body.filename or ("pasted links" if (body.text or "").strip() else f"{len(found)} link(s)")
made = links.queue(db, cfg, sid, found, user.email, body.collection, body.pipeline, body.folders_as_tags, body.content_type, said)
out = [LinkImportResult(**r) for r in made]
queued = sum(r.status == "queued" for r in out)
auth.audit(db, user.as_audit(), "import.web", f"space:{sid}", {"links": queued, "namespace": ns})
return LinksImport(results=out)
run = next((r.import_run for r in out if r.import_run), None)
auth.audit(db, user.as_audit(), "import.web", f"space:{sid}", store.clean({"links": queued, "namespace": ns, "import_run": run}))
return LinksImport(results=out, import_run=run)


@router.post("/preview")
Expand Down Expand Up @@ -304,8 +322,63 @@ def import_from_source(body: SourceImportRequest, request: Request, user: AdminW
with domain_errors():
results = sources.import_files(db, cfg, body.source, body.paths, sid, user.email, body.pipeline, body.collection, body.documents_as)
queued = [r for r in results if r["status"] == "queued"]
run = queued[0].get("import_run") if queued else None
if queued:
detail = {"namespace": ns, "files": len(queued), "recordings": [r["recording"] for r in queued]}
detail = {"namespace": ns, "files": len(queued), "recordings": [r["recording"] for r in queued], "import_run": run}
auth.audit(db, user.as_audit(), "import.source", f"source:{body.source}", detail)
request.app.state.graph_cache.clear()
return SourceImport(results=[SourceImportResult(**r) for r in results])
return SourceImport(results=[SourceImportResult(**r) for r in results], import_run=run)


# ---------- import runs: one bulk import, listed and undone whole ----------
def _import_run(db: DB, acl: Access, user: Principal, run_id: int) -> dict[str, Any]:
"""A run this person may see: theirs, one in a namespace where they may delete assets, or (admins) any; else 404."""
try:
run = import_runs.get(db, run_id)
except KeyError:
raise HTTPException(404, "not found") from None
if not (user.admin or run.get("created_by") == user.email or acl.can(run.get("space"), "remove")):
raise HTTPException(404, "not found")
return run


@router.get("/runs")
def list_import_runs(acl: Acl, user: CurrentUser, db: Db, limit: int = Query(50, ge=1, le=200)) -> list[ImportRun]:
"""Recent bulk imports, newest first, each with how many assets it made and how many are still here: the ones you
started and those in namespaces where you may delete assets (everyone's, for admins). A folder scan, an archive's
files, a list of links or bookmarks, and chosen files of a source are each one run."""
mine = [s for s in acl.roles if acl.can(s, "remove")]
return [ImportRun.model_validate(r) for r in import_runs.listing(db, None if user.admin else user.email, mine, limit)]


@router.get("/runs/{run_id}")
def get_import_run(run_id: int, acl: Acl, user: CurrentUser, db: Db) -> ImportRun:
return ImportRun.model_validate(_import_run(db, acl, user, run_id))


@router.post("/runs/{run_id}/undo")
def undo_import_run(run_id: int, acl: Acl, user: Writer, db: Db, cfg: Cfg, request: Request, tasks: BackgroundTasks) -> ImportRunUndone:
"""Undo a bulk import: every asset it made that is still here is deleted, as `DELETE /recordings/{id}` deletes
one (the media files stay where they are, and scans don't import them again). It needs the rights a delete needs on
each of them (owners); when any is one you may not delete, nothing is deleted (403). An asset a job is working on
is left (`busy`) and its job asked to stop: undo the run again for the rest. Lens keeps no recently deleted list of
assets, so this can't itself be undone; importing the files again brings them back. Audited as `recording.delete`
for each asset and `import.undo` for the run."""
run = _import_run(db, acl, user, run_id)

def can(space: int, collection: int | None) -> bool:
try:
acl.need_in(space, collection, "remove")
except HTTPException:
return False
return True

refused = import_runs.refused(db, run_id, can)
if refused:
raise HTTPException(403, f"you may not delete {len(refused)} of its {run['left']} assets; undoing it needs an owner")
with domain_errors():
out = import_runs.undo(db, cfg, run_id, user.as_audit())
request.app.state.graph_cache.clear()
for ns in out["namespaces"]:
tasks.add_task(render.refresh_overview, db, cfg, ns)
return ImportRunUndone(run=ImportRun.model_validate(import_runs.get(db, run_id)), deleted=out["deleted"], busy=out["busy"])
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/v1/routes/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ def queue_step(step: str, acl: Acl, user: Writer, db: Db, cfg: Cfg) -> StepQueue

def scan() -> None:
try:
ingest.scan(db, cfg, log=log.info)
ingest.scan(db, cfg, log=log.info, by=user.email)
jobs.enqueue_pending(db, by=user.email)
except Exception: # noqa: BLE001 - background thread; report and carry on
log.exception("scan failed")
Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -2685,6 +2685,7 @@ def _carry_out(db, cfg, a, user, editable, decision, base, admin, readable, via=
"create_namespace",
"import_files",
"import_urls",
"undo_import",
"invite_person",
"change_person",
"subscribe_feed",
Expand Down
Loading
Loading