Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Backups hold Lens's files too.** Each backup now also keeps the files under `data_dir` (uploads, objects, media,
frames, renditions, web pages, reports, exports, podcasts, certificates), and restoring one puts them back. Files
that haven't changed since the previous backup are hard links, so a backup takes only the room of what changed; the
first one takes as much room as the files. Downloads and copies on a storage connection carry them as well. Older
backups show *database only* ([Backups](docs/database.md#backups)).
- **Analytics.** A new Analytics page (admins) shows active people per day, week and month, stickiness, churn rate,
retention by sign-up week and feature use, worked out on the server from the activity ledger and sign-ins. Each day
Lens also keeps a few counted events (install, daily active, feature used, upgrade, inactive, return). They can
Expand Down
20 changes: 13 additions & 7 deletions docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,17 @@ first ([Updates and backups](updates.md)). Each is one entry in `<data_dir>/back
| `before-restore` | what the database held just before a restore replaced it | the newest three |

* With a server, `<kind>-<time>.surql.gz`: SurrealDB's export of the database, gzipped.
* Embedded, `<kind>-<time>.surrealkv`: a copy of the database folder, taken with it closed (downloaded as a
`.tar.gz`).

**Restoring** replaces the whole database with the backup's. What it held is backed up first (`before-restore`) and
* Embedded, `<kind>-<time>.surrealkv`: a copy of the database folder, taken with it closed.
* With either, the files Lens keeps under `data_dir` (uploads, `objects`, media, frames, renditions, web pages,
reports, exports, podcasts, certificates) in `<data_dir>/backups/.files/<name>/`. A file that hasn't changed since
the previous backup is a hard link to that backup's copy, so each backup takes only the room of what changed. Left
out: what Lens fetches or rebuilds by itself (`models`, `bin`, `python`, `cache`, the search index), passing files
(`tmp`, `locks`, `logs`, unfinished uploads), and `secret.key`. A backup is refused when the disk hasn't room for
the files that changed. A backup with files, or of an embedded database, downloads as one `.tar.gz` holding
`<name>` and `<name>.files/`. Backups from before files were kept show *database only* in Settings.

**Restoring** replaces the whole database, and the files under `data_dir`, with the backup's (a *database only*
backup leaves the files as they are). What was there is backed up first (`before-restore`) and
put back if the restore fails; a backup from an older Lens is brought up to date (schema and upgrade steps) afterwards.
People signed in since the backup was taken sign in again. Only an admin signed in with a passkey can download or
restore a backup from Settings (a download there is a link that works for two minutes); an admin API token can take,
Expand All @@ -163,8 +170,7 @@ has, so you can then restore or download it. If copying fails, the backup here i

By hand, with a server: `gunzip -k` the file, then `surreal import --conn http://host:8000 --user root --pass … --ns
archive --db main <file>.surql` into an empty database. Embedded: stop Lens and put the copy in place of
`<data_dir>/surrealdb`.
`<data_dir>/surrealdb`. Then copy the contents of `<name>.files/` into `data_dir`.

Backups hold the database only. Also back up `data_dir` (uploads, reports, frames, caches), and keep `secret.key`
(unless `ARCHIVE_SECRET_KEY` is set) apart from the backups: without it, stored credentials and server-held file keys
Keep `secret.key` (unless `ARCHIVE_SECRET_KEY` is set) apart from the backups: without it, stored credentials and server-held file keys
can't be decrypted, and with it a backup opens them ([encryption](encryption.md)).
5 changes: 3 additions & 2 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@ docker compose -f docker-compose.prod.yml logs backend | grep "setup code"
stored source credentials, LLM keys and everything [encrypted at rest](encryption.md) unreadable), `AUTH_SECRET`.
* **SurrealDB storage engine**: `surrealkv` (as in the compose file), RocksDB or TiKV. Not `memory`; see
[Database](database.md).
* **Backups.** `docker compose -f docker-compose.prod.yml exec backend lens backup` writes one of the database into
`data_dir/backups` ([Database](database.md#backups)); back up the `archive-data` volume as well.
* **Backups.** `docker compose -f docker-compose.prod.yml exec backend lens backup` writes one of the database and
Lens's files into `data_dir/backups` ([Database](database.md#backups)); copy them off the machine with a storage
connection (Settings › Backups) or by backing up the `archive-data` volume.
* **Mail** for sign-in links and access requests: **Settings → Email** in the app (or `MAIL_*` in `.env`).
* **Workers.** Scale with `docker compose up -d --scale worker=3`. The images carry no speech-to-text engine: a worker
fetches the configured one (SenseVoice by default) the first time a recording needs it ([Components](components.md)),
Expand Down
32 changes: 19 additions & 13 deletions fastapi_backend/app/api/v1/routes/backups.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
docs/database.md#backups; admins only). Listing, taking and deleting backups are in routes/updates.py.

Restoring (which replaces the database) and download links (a backup holds everything in it) need an admin signed in
with a passkey. A link works for LINK_SECONDS, so the browser saves the file itself however big it is; an embedded
database's folder comes as a .tar.gz.
with a passkey. A link works for LINK_SECONDS, so the browser saves the file itself however big it is; a backup with
files, or of an embedded database, comes as a .tar.gz.
"""

from __future__ import annotations
Expand Down Expand Up @@ -124,8 +124,8 @@ def backup_link(name: str, user: AdminReader, db: Db) -> BackupLink:
return BackupLink(url=sign_path(f"/api/v1/admin/backups/{name}/file", ttl=LINK_SECONDS), expires_in=LINK_SECONDS)


def _tar(folder) -> Iterator[bytes]:
"""A folder as a .tar.gz, written as it is read."""
def _tar(parts) -> Iterator[bytes]:
"""Files and folders, as (path, name), as a .tar.gz written as it is read."""

class Sink:
def __init__(self):
Expand All @@ -141,26 +141,32 @@ def take(self):

sink = Sink()
with tarfile.open(fileobj=sink, mode="w|gz") as tar: # type: ignore[call-overload]
for p in sorted(folder.rglob("*")):
tar.add(p, arcname=f"{folder.name}/{p.relative_to(folder)}", recursive=False)
if chunk := sink.take():
yield chunk
for top, arcname in parts:
for p in [top, *sorted(top.rglob("*"))] if top.is_dir() else [top]:
tar.add(p, arcname=f"{arcname}/{p.relative_to(top)}" if p != top else arcname, recursive=False)
if chunk := sink.take():
yield chunk
yield sink.take()


@router.get(
"/{name}/file",
response_class=StreamingResponse,
responses={200: {"content": {"application/gzip": {}}, "description": "the backup (an embedded one as a .tar.gz)"}},
responses={
200: {
"content": {"application/gzip": {}},
"description": "the backup: an export with no files as it is, anything else as a .tar.gz",
}
},
)
def backup_file(name: str, db: Db, exp: str | None = None, sig: str | None = None):
"""The backup itself, through a link from POST /admin/backups/{name}/link."""
if not verify_path(f"/api/v1/admin/backups/{name}/file", exp, sig):
raise HTTPException(403, "this download link has expired; download the backup again from Settings")
with domain_errors():
p = backups.path(db, name)
if p.is_dir():
parts = backups.parts(db, name)
if len(parts) > 1 or parts[0][0].is_dir():
return StreamingResponse(
_tar(p), media_type="application/gzip", headers={"Content-Disposition": f'attachment; filename="{name}.tar.gz"'}
_tar(parts), media_type="application/gzip", headers={"Content-Disposition": f'attachment; filename="{name}.tar.gz"'}
)
return FileResponse(p, media_type="application/gzip", filename=name)
return FileResponse(parts[0][0], media_type="application/gzip", filename=name)
81 changes: 60 additions & 21 deletions fastapi_backend/app/domain/backups.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
put back if importing fails. A backup from an older Lens is brought up to date afterwards (its schema, then the named
steps), and every process reloads its settings.

Backups hold the database only: files Lens keeps under data_dir, and secret.key (keep that apart from the backups,
docs/encryption.md), are backed up with the data folder.
Each backup also keeps the files under data_dir (uploads, objects, media and the rest; datafiles.py), and a restore puts
them back too. secret.key isn't in them: keep it apart from the backups (docs/encryption.md).
"""

from __future__ import annotations
Expand All @@ -25,7 +25,7 @@
import threading
import time

from . import migrations, schedule, store
from . import datafiles, migrations, schedule, store
from .store import R

log = logging.getLogger("lens")
Expand Down Expand Up @@ -64,8 +64,9 @@ def listing(db):
if not m:
continue
at = dt.datetime.strptime(m["stamp"], "%Y%m%d-%H%M%S").replace(tzinfo=dt.timezone.utc)
files = datafiles.path(db.data_dir, p.name)
try:
size = _size(p)
size = _size(p) + (_size(files) if files else 0)
except OSError:
continue
out.append(
Expand All @@ -74,6 +75,7 @@ def listing(db):
"label": m["label"],
"kind": "folder" if m["kind"] == "surrealkv" else "export",
"bytes": size,
"files": files is not None,
"taken_at": at.isoformat(timespec="seconds"),
}
)
Expand Down Expand Up @@ -101,6 +103,7 @@ def take(db):
def delete(db, name):
p = path(db, name)
shutil.rmtree(p) if p.is_dir() else p.unlink()
datafiles.drop(db.data_dir, name)


def in_progress(db):
Expand All @@ -114,6 +117,7 @@ def prune(db, label, keep):
for old in [b for b in listing(db) if b["label"] == label][keep:]:
with contextlib.suppress(OSError, ValueError):
delete(db, old["name"])
datafiles.tidy(db.data_dir, [b["name"] for b in listing(db)] + in_progress(db))


def _take(db, label):
Expand Down Expand Up @@ -148,14 +152,28 @@ def restore(db, cfg, name):
safety = _take(db, "before-restore") if migrations._has_data(db) else None
if safety:
prune(db, "before-restore", RESTORE_KEEP)
if db.embedded:
_swap(db, src)
else:
_import(db, src, safety and path(db, safety["name"]))
live = _db_path(db)
files = datafiles.restore(db.data_dir, name, live) # first: a failure here leaves everything as it was
try:
if db.embedded:
_swap(db, src)
else:
_import(db, src, safety and path(db, safety["name"]))
except Exception:
if files and safety: # the database was put back, so its files go back too
try:
datafiles.restore(db.data_dir, safety["name"], live)
except Exception: # noqa: BLE001
log.exception("putting the files back after a failed restore failed too; restore %s", safety["name"])
raise
_after(db, cfg, seq_was)
return safety


def _db_path(db):
return db.url.partition("://")[2] if db.embedded else None


def _swap(db, src):
live = pathlib.Path(db.url.partition("://")[2])
with db.closed():
Expand Down Expand Up @@ -331,7 +349,7 @@ def run_due(db, cfg, log=None, now=None):

# ---------- copies on a storage connection ----------
# Every new backup (taken by hand or on the schedule) is also copied to backups.connection, under backups.folder, as
# <name>.lensbk: the export (or the embedded folder as a .tar.gz), encrypted with a key derived from secret.key. Not with
# <name>.lensbk: the export (or the embedded folder, and the files kept with it, as a .tar.gz), encrypted with a key derived from secret.key. Not with
# a namespace's key: those are kept in the database, and a backup must open when the database is gone. So keep
# secret.key (or ARCHIVE_SECRET_KEY) somewhere other than the connection, or the copies can't be read.
SEALED = ".lensbk"
Expand Down Expand Up @@ -399,15 +417,27 @@ def remote(cfg):
return {"store": "connection", "connection": int(b["connection"]), "folder": b.get("folder") or "lens-backups", "crypt": False}


def _package(src: pathlib.Path, out: pathlib.Path):
"""A backup as one file: the export as it is, an embedded folder as a .tar.gz."""
def parts(db, name):
"""What the backup `name` is made of, as (path, its name in a .tar.gz): the database, and the files kept with it
under <name>.files/."""
src = path(db, name)
out = [(src, src.name)]
if files := datafiles.path(db.data_dir, name):
out.append((files, f"{name}{datafiles.FOLDER}"))
return out


def _package(db, name, out: pathlib.Path):
"""A backup as one file: an export with no files as it is, anything else as a .tar.gz."""
import tarfile

if not src.is_dir():
shutil.copyfile(src, out)
got = parts(db, name)
if len(got) == 1 and not got[0][0].is_dir():
shutil.copyfile(got[0][0], out)
return
with tarfile.open(out, "w:gz") as tar:
tar.add(src, arcname=src.name)
for p, arcname in got:
tar.add(p, arcname=arcname)


def send(db, cfg, name):
Expand All @@ -417,11 +447,11 @@ def send(db, cfg, name):
where = remote(cfg)
if where is None:
raise BackupError("choose a storage connection for backups first")
src = path(db, name)
path(db, name)
tmp = folder(db) / f".send-{time.time_ns()}"
sealed = tmp.with_name(tmp.name + SEALED)
try:
_package(src, tmp)
_package(db, name, tmp)
seal(cfg, tmp, sealed)
blobs._send(db, cfg, where, name + SEALED, sealed)
finally:
Expand Down Expand Up @@ -529,12 +559,21 @@ def fetch(db, cfg, name):
raise ValueError("no such backup on the connection") from None
unseal(cfg, sealed, plain)
part = dst.with_name(dst.name + ".fetching")
if name.endswith(".surrealkv"):
if tarfile.is_tarfile(plain):
out = d / f".untar-{time.time_ns()}"
with tarfile.open(plain, "r:gz") as tar:
tar.extractall(out, filter="data")
(out / name).rename(part)
shutil.rmtree(out, ignore_errors=True)
try:
with tarfile.open(plain, "r:gz") as tar:
tar.extractall(out, filter="data")
if not (out / name).exists():
raise BackupError("this copy doesn't hold the backup it is named after")
files = out / f"{name}{datafiles.FOLDER}"
if files.is_dir():
datafiles.drop(db.data_dir, name)
datafiles.folder(db.data_dir).mkdir(parents=True, exist_ok=True)
files.rename(datafiles.folder(db.data_dir) / name)
(out / name).rename(part)
finally:
shutil.rmtree(out, ignore_errors=True)
else:
plain.rename(part)
part.rename(dst)
Expand Down
Loading
Loading