Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Build context of Dockerfile.cloudron (and the source Cloudron uploads on `cloudron install`)
.git
.github
**/node_modules
**/.next
**/.next-*
**/.venv
**/__pycache__
**/.pytest_cache
**/.ruff_cache
**/.mypy_cache
**/.env
**/.env.*
!**/.env.example
fastapi_backend/archive-data
fastapi_backend/archive.yaml
fastapi_backend/tests
fastapi_backend/coverage.xml
nextjs-frontend/__tests__
local-audio
local-shared-data
docs
site
72 changes: 72 additions & 0 deletions .github/workflows/cloudron.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
name: Cloudron image

# Builds the Cloudron package (Dockerfile.cloudron) when it changes. A published release pushes the image to GHCR,
# tagged with the release version and `latest`; the Release workflow starts it with `tag` (a release published with
# GITHUB_TOKEN fires no release event). Run by hand without `tag`, it pushes the manifest's version. The image is for
# `cloudron install --image` or `cloudron versions add` (cloudron/README.md).
on:
release:
types: [published]
pull_request:
paths: [Dockerfile.cloudron, CloudronManifest.json, cloudron/**, .dockerignore]
workflow_dispatch:
inputs:
tag:
description: "Release tag to build and push as that version and latest (vX.Y.Z); empty: the manifest's version"
type: string
default: ""
push:
description: "Push to ghcr.io/<owner>/lens-cloudron"
type: boolean
default: true

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.tag || github.ref }}

- name: Image name and tags (the release's version, else the manifest's)
id: meta
env:
RELEASE_TAG: ${{ github.event.release.tag_name || inputs.tag }}
PUSH: ${{ github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && (inputs.push || inputs.tag != '')) }}
run: |
image="ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/lens-cloudron"
version="${RELEASE_TAG#v}"
[ -n "$version" ] || version=$(jq -r .version CloudronManifest.json)
tags="$image:$version"
[ -z "$RELEASE_TAG" ] || tags="$tags,$image:latest"
echo "image=$image" >> "$GITHUB_OUTPUT"
echo "version=$version" >> "$GITHUB_OUTPUT"
echo "tags=$tags" >> "$GITHUB_OUTPUT"
echo "push=$PUSH" >> "$GITHUB_OUTPUT"

- uses: docker/setup-buildx-action@v3

- if: steps.meta.outputs.push == 'true'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- uses: docker/build-push-action@v6
with:
context: .
file: Dockerfile.cloudron
push: ${{ steps.meta.outputs.push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Summary
if: steps.meta.outputs.push == 'true'
run: |
echo "Pushed \`${{ steps.meta.outputs.image }}:${{ steps.meta.outputs.version }}\`." >> "$GITHUB_STEP_SUMMARY"
echo "Make the package public on GitHub so Cloudron can pull it." >> "$GITHUB_STEP_SUMMARY"
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,16 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
"table 'seq' does not exist" (SurrealDB 3). The counters table is defined with the rest of the schema now, so
nothing reads it before it exists.

- **Runs on Cloudron.** The repository is now a Cloudron package (`CloudronManifest.json`, `Dockerfile.cloudron`,
`cloudron/`): `cloudron install --location lens -f Dockerfile.cloudron` from a checkout installs Lens on your own
Cloudron, which builds the image itself (cloudron/README.md).
- One app runs SurrealDB, the API, a job worker and the web app; secrets are made on first start and kept in
`/app/data`, password-reset mail goes through Cloudron's mail server, and the first-admin setup code is in
the app's logs.
- The database is exported to `/app/data/backup/lens.surql` every six hours, so Cloudron's backups always hold
a consistent copy alongside the live files.
- It can also be published as a community app (`cloudron versions add`), with the image built and pushed to
GHCR by the new "Cloudron image" workflow on every published release.
- **Lens for QNAP NAS.** `packaging/qnap/build.sh` builds a self-contained QPKG (with
[QDK](https://github.com/qnap-dev/QDK)) for Intel/AMD or ARM models that installs from the App Center's Install
Manually: it carries the Lens images and SurrealDB, loads them into Container Station on its first start, and runs
Expand Down
30 changes: 30 additions & 0 deletions CloudronManifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"id": "net.adeelahmad.lens",
"title": "Lens",
"author": "Adeel Ahmad",
"tagline": "An archive for recorded speech and video",
"description": "file://cloudron/DESCRIPTION.md",
"changelog": "file://cloudron/CHANGELOG",
"postInstallMessage": "file://cloudron/POSTINSTALL.md",
"version": "0.3.0",
"upstreamVersion": "0.3.0",
"website": "https://github.com/adeelahmad/lense",
"packagerName": "Adeel Ahmad",
"packagerUrl": "https://github.com/adeelahmad",
"documentationUrl": "https://github.com/adeelahmad/lense/tree/main/docs",
"icon": "file://cloudron/logo.png",
"iconUrl": "https://raw.githubusercontent.com/adeelahmad/lense/main/cloudron/logo.png",
"mediaLinks": [
"https://raw.githubusercontent.com/adeelahmad/lense/main/cloudron/logo.png"
],
"tags": ["media", "productivity"],
"healthCheckPath": "/api/v1/auth/status",
"httpPort": 3000,
"addons": {
"localstorage": {},
"sendmail": {}
},
"memoryLimit": 3221225472,
"minBoxVersion": "9.1.0",
"manifestVersion": 2
}
65 changes: 65 additions & 0 deletions Dockerfile.cloudron
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Lens as one Cloudron app: SurrealDB, the API, a job worker and the Next.js web app in one container, run by
# supervisord. Only the web app (port 3000) is public; it proxies the API's paths (lib/api/backend-proxy.ts).
# Build and install: see cloudron/README.md.
# --build-arg EXTRAS="sensevoice voices" optional backend extras, as in fastapi_backend/Dockerfile
# --build-arg FULL=true LibreOffice and fonts, to read Office files, text, saved web pages and emails
# (no Chromium: Ubuntu 24.04's apt package is a snap stub, so no web capture)

FROM cloudron/base:5.1.0@sha256:1c0666c9abe9e2090d33686826d4e97769b799124573118d41e0d7485135748e AS frontend
ENV PATH="/usr/local/node-24.19.0/bin:$PATH" COREPACK_ENABLE_DOWNLOAD_PROMPT=0
WORKDIR /app
RUN corepack enable
COPY nextjs-frontend/package.json nextjs-frontend/pnpm-lock.yaml nextjs-frontend/pnpm-workspace.yaml ./
RUN pnpm install --frozen-lockfile
COPY nextjs-frontend/ ./
ENV NEXT_TELEMETRY_DISABLED=1
RUN AUTH_SECRET=build-only pnpm build && pnpm prune --prod && rm -rf .next/cache

FROM cloudron/base:5.1.0@sha256:1c0666c9abe9e2090d33686826d4e97769b799124573118d41e0d7485135748e

ARG EXTRAS=""
ARG FULL=false

# tesseract: text on screen; antiword + poppler: legacy .doc and PDF fallbacks (ffmpeg is in the base image)
RUN apt-get update \
&& apt-get install -y --no-install-recommends tesseract-ocr antiword poppler-utils \
&& if [ "$FULL" = "true" ]; then apt-get install -y --no-install-recommends \
libreoffice-writer-nogui libreoffice-calc-nogui libreoffice-impress-nogui \
fonts-dejavu-core fonts-liberation2 fonts-noto-core fonts-noto-cjk; fi \
&& case " $EXTRAS " in *" doctr "*) apt-get install -y --no-install-recommends libgl1 libglib2.0-0;; esac \
&& rm -rf /var/lib/apt/lists/*

COPY --from=surrealdb/surrealdb:v3.2.4 /surreal /usr/local/bin/surreal
COPY --from=rclone/rclone:1.75.1 /usr/local/bin/rclone /usr/local/bin/rclone
COPY --from=ghcr.io/astral-sh/uv:0.8 /uv /usr/local/bin/uv

ENV PATH="/app/code/venv/bin:/usr/local/node-24.19.0/bin:$PATH" \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
NEXT_TELEMETRY_DISABLED=1

# The backend, in a virtualenv built against the base image's Python 3.12
WORKDIR /app/code/backend
COPY fastapi_backend/pyproject.toml fastapi_backend/uv.lock fastapi_backend/README.md ./
RUN UV_PROJECT_ENVIRONMENT=/app/code/venv UV_PYTHON_DOWNLOADS=never UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy \
uv sync --frozen --no-dev --no-install-project --python /usr/bin/python3.12 \
$(for e in $EXTRAS; do printf -- '--extra %s ' "$e"; done)
COPY fastapi_backend/ ./
RUN UV_PROJECT_ENVIRONMENT=/app/code/venv UV_PYTHON_DOWNLOADS=never UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy \
uv sync --frozen --no-dev --python /usr/bin/python3.12 \
$(for e in $EXTRAS; do printf -- '--extra %s ' "$e"; done) \
&& rm -rf /root/.cache

# The web app, built in the stage above; Next writes its cache to /run (the code is read-only on Cloudron)
WORKDIR /app/code/frontend
COPY --from=frontend /app/package.json /app/next.config.mjs ./
COPY --from=frontend /app/node_modules ./node_modules
COPY --from=frontend /app/.next ./.next
COPY --from=frontend /app/public ./public
RUN ln -s /run/lens/next-cache .next/cache

COPY cloudron/start.sh cloudron/supervisord.conf cloudron/archive.yaml cloudron/export-db.sh /app/code/cloudron/
RUN chmod +x /app/code/cloudron/*.sh

WORKDIR /app/code
CMD [ "/app/code/cloudron/start.sh" ]
2 changes: 2 additions & 0 deletions cloudron/CHANGELOG
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[0.3.0]
* First Cloudron package: SurrealDB 3.2.4, the API, a job worker and the web app in one app
17 changes: 17 additions & 0 deletions cloudron/DESCRIPTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
Lens is an archive for recorded speech and video. It transcribes recordings, tells speakers apart, makes everything
searchable (full text and by meaning), builds a knowledge graph of the people, places and topics mentioned, and
publishes recordings as IIIF. Its assistant answers questions with citations to the exact moment in a recording.

### Features

* **Transcription and speakers:** transcripts with word timings, speaker diarisation and voice IDs.
* **Search:** full-text and semantic search across transcripts, documents and images.
* **Knowledge graph:** entities and their relations, per namespace or shared.
* **Assistant:** chat with your archive through any OpenAI-compatible model server (Ollama, llama.cpp, vLLM, LM Studio).
* **Publishing:** an embeddable transcript player, static reports, and IIIF Presentation 3 with Content Search.

### This package

SurrealDB, the API, a background worker and the web app run together in one app. The database and the archive's
files live in the app's data directory and are included in Cloudron backups, with a fresh database export every
six hours.
6 changes: 6 additions & 0 deletions cloudron/POSTINSTALL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Open $CLOUDRON-APP-ORIGIN, enter the **setup code** to create the first admin, then follow the setup steps
(namespace, model provider, storage). The code is printed in the app's logs (search for "setup code"), or read it
in the web terminal with `grep LENS_SETUP_CODE /app/data/secrets.env`.

Recordings to watch go in `/app/data/media` (File manager, or `cloudron push`). Processing settings are in
`/app/data/archive.yaml` and in the app under Settings.
82 changes: 82 additions & 0 deletions cloudron/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Lens on Cloudron

The repository root is a Cloudron package: `CloudronManifest.json`, `Dockerfile.cloudron` and this folder. One app
runs SurrealDB, the API, a job worker and the web app under supervisord; only the web app is public, and it serves
the API's paths itself. Needs Cloudron 9.1 or later.

| Where | What |
|---|---|
| `/app/data/secrets.env` | secrets made on first start (session keys, database password, setup code); keep it |
| `/app/data/archive.yaml` | processing configuration, written once from `cloudron/archive.yaml`, then yours to edit |
| `/app/data/surreal` | the SurrealDB database (SurrealKV) |
| `/app/data/backup/lens.surql` | a database export, refreshed every six hours, so backups always hold a consistent copy |
| `/app/data/archive` | `data_dir`: uploads, frames, reports, model caches |
| `/app/data/media` | a folder the app may watch: put recordings here (File manager or `cloudron push`) |

Mail for password resets goes through Cloudron's mail server (the sendmail addon).

## Install on your Cloudron

With the [Cloudron CLI](https://www.npmjs.com/package/cloudron) on your computer, from a checkout of this
repository:

```bash
npm install -g cloudron
cloudron login my.example.com # your Cloudron's dashboard address
cloudron install --location lens -f Dockerfile.cloudron
```

The CLI uploads the source and your Cloudron builds the image itself, so no registry is needed. Then open the app:
enter the first-admin **setup code** (in the app's logs, or `grep LENS_SETUP_CODE /app/data/secrets.env` in the
web terminal), then follow the setup steps for the namespace, model provider and storage.

Update to a newer checkout with `cloudron update --app lens -f Dockerfile.cloudron`; logs with
`cloudron logs --app lens -f`.

Build options go with `--build-arg`: `FULL=true` adds LibreOffice and fonts (Office files, text, saved web pages,
emails; capturing live web pages needs Chromium, which this package leaves out), `EXTRAS="sensevoice voices"` adds backend extras as in `fastapi_backend/Dockerfile`. Raise the memory limit
in the app's Resources page for large models.

## Build once, install from an image

To build on another machine instead, push the image to a registry your Cloudron can pull from (a public one, or one
added under Settings → Docker registries):

```bash
docker build -f Dockerfile.cloudron -t ghcr.io/you/lens-cloudron:0.3.0 .
docker push ghcr.io/you/lens-cloudron:0.3.0
cloudron install --location lens --image ghcr.io/you/lens-cloudron:0.3.0
```

## Publish as a community app

Cloudron 9.1 and later install community apps from a `CloudronVersions.json` file at a public URL, with updates
offered as new versions are added to it.

Publishing a GitHub release builds the image and pushes it to `ghcr.io/adeelahmad/lens-cloudron:<release version>`
(and `:latest`) with the "Cloudron image" workflow; make that package public once on GitHub. Then:

```bash
cloudron versions init # once: creates CloudronVersions.json
cloudron versions add --image ghcr.io/adeelahmad/lens-cloudron:0.3.0
git add CloudronVersions.json && git commit -m "Cloudron: publish 0.3.0" && git push
```

Then install it from the dashboard (App Store, community app, with the file's raw URL) or with
`cloudron install --location lens --versions-url https://raw.githubusercontent.com/adeelahmad/lense/main/CloudronVersions.json`.
For each release: add a `[x.y.z]` entry to `cloudron/CHANGELOG`, run the Release workflow (it sets the version in
`CloudronManifest.json` and starts the image build), then `cloudron versions add` again. The image must be public.

## Restore the database from its export

If the live database files in a backup don't open, load the export into a fresh database (web terminal):

```bash
supervisorctl -c /app/code/cloudron/supervisord.conf stop api worker surrealdb
mv /app/data/surreal /app/data/surreal.broken && mkdir /app/data/surreal && chown cloudron:cloudron /app/data/surreal
supervisorctl -c /app/code/cloudron/supervisord.conf start surrealdb
source /app/data/secrets.env
surreal import --endpoint http://127.0.0.1:8001 --user root --pass "$SURREAL_PASS" \
--namespace archive --database main /app/data/backup/lens.surql
supervisorctl -c /app/code/cloudron/supervisord.conf start api worker
```
9 changes: 9 additions & 0 deletions cloudron/archive.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Lens processing configuration on Cloudron. Written to /app/data/archive.yaml on first start and yours to edit
# afterwards (restart the app to apply). Every key is documented in fastapi_backend/archive.example.yaml; most
# settings can also be changed in the web app.
data_dir: /app/data/archive # reports, frames, uploads, model caches, locks
workers:
inline: 0 # the worker process does the work
sources:
# folders the web app may watch: put recordings in /app/data/media (Cloudron's file manager, or `cloudron push`)
local_roots: [/app/data/media]
21 changes: 21 additions & 0 deletions cloudron/export-db.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash
# Writes a consistent SurrealQL export of the database to /app/data/backup every few hours, so Cloudron's backups
# (which copy /app/data while the app runs) always hold a clean copy next to the live database files.
set -u
umask 077 # the export holds the whole archive

INTERVAL="${LENS_EXPORT_INTERVAL_SECONDS:-21600}"
OUT=/app/data/backup/lens.surql

sleep 600 # let the app settle after a start
while true; do
if surreal export --endpoint http://127.0.0.1:8001 --user root --pass "${SURREAL_PASS}" \
--namespace "${SURREAL_NS:-archive}" --database "${SURREAL_DB:-main}" "${OUT}.tmp"; then
mv "${OUT}.tmp" "${OUT}"
echo "export-db: wrote ${OUT}"
else
rm -f "${OUT}.tmp"
echo "export-db: export failed, keeping the previous one"
fi
sleep "${INTERVAL}"
done
Binary file added cloudron/logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading