Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/qnap.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: QNAP packages

# Builds the Lens QPKGs (packaging/qnap) for Intel/AMD and ARM NAS and attaches them to the release.
# Runs when a release is published (release.yml makes the draft; publishing it starts this), or by hand for a tag.
on:
release:
types: [published]
workflow_dispatch:
inputs:
tag:
description: "Release tag to build and attach the packages to"
required: true
type: string

permissions:
contents: write

concurrency:
group: qnap-${{ github.event.release.tag_name || inputs.tag }}
cancel-in-progress: false

jobs:
qpkg:
name: QPKG ${{ matrix.arch }}
strategy:
fail-fast: false
matrix:
include:
- arch: x86_64
runner: ubuntu-latest
# Built under QEMU unless the repository variable QNAP_ARM_RUNNER names an ARM runner (ubuntu-24.04-arm)
- arch: arm_64
runner: ${{ vars.QNAP_ARM_RUNNER || 'ubuntu-latest' }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 300
env:
TAG: ${{ github.event.release.tag_name || inputs.tag }}
GH_TOKEN: ${{ github.token }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ env.TAG }}

- name: Make room for the images
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
docker system prune -af >/dev/null
df -h /

- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
with:
driver: docker # images go straight into the local Docker, where build.sh saves them from

- name: Build the package
run: |
version="${TAG#v}"
packaging/qnap/build.sh --arch "${{ matrix.arch }}" --version "$version"

- name: Attach to the release
run: |
ls -lh packaging/qnap/build/
# GitHub release assets are limited to 2 GiB each
for f in packaging/qnap/build/*.qpkg; do
[ "$(stat -c %s "$f")" -lt 2147483648 ] || { echo "::error::$f is over 2 GiB, too big for a release asset"; exit 1; }
done
gh release upload "$TAG" packaging/qnap/build/*.qpkg packaging/qnap/build/*.qpkg.md5 --clobber
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,15 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
"table 'seq' does not exist" (SurrealDB 3). The counters table is defined with the rest of the schema now, so
nothing reads it before it exists.

- **Lens for QNAP NAS.** `packaging/qnap/build.sh` builds a self-contained QPKG (with
[QDK](https://github.com/qnap-dev/QDK)) for Intel/AMD or ARM models that installs from the App Center's Install
Manually: it carries the Lens images and SurrealDB, loads them into Container Station on its first start, and runs
the stack there, so there are no containers to set up by hand. The first start writes the settings with fresh
secrets to a data folder (`/share/Container/lens`) that outlasts removing the app, and puts the first-admin setup
code in the QTS system log; Lens sees only `Multimedia/Lens` on the NAS, a folder the setup steps can watch.
Upgrades install over the old version and remove its images. Publishing a GitHub release builds both packages and
attaches them to it (`.github/workflows/qnap.yml`). Steps in packaging/qnap/README.md.

- **Fix: Chat answers no longer break off with "The answer stopped before it finished".** With some model servers an
answer ended mid-stream with nothing saved, so the question sat unanswered in the conversation. Now each one ends
in an answer, or an error that's shown and saved with the conversation.
Expand Down
2 changes: 2 additions & 0 deletions packaging/qnap/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.work/
build/
121 changes: 121 additions & 0 deletions packaging/qnap/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Lens for QNAP NAS

A self-contained QPKG, built with [QDK](https://github.com/qnap-dev/QDK), that installs Lens from the App Center like
any other app. Lens runs in **Container Station**, but you don't set up any containers: the package carries the
Lens images (API and worker, web app) and SurrealDB, loads them on its first start, and runs the stack. Nothing is
built or downloaded on the NAS.

- **NAS:** QTS 5 or QuTS hero 5 with Container Station 3, on an Intel/AMD (`x86_64`) or ARM 64-bit (`arm_64`) model.
4 GB of RAM or more; transcription is heavy.
- **Space:** a few GB on the volume: the package's image files while they're loaded on the first start (they're
deleted afterwards), the images in Container Station, and whatever your archive holds.

## Build the package

On any machine with Docker and buildx (Linux, macOS, Windows with WSL), from the repository root:

```bash
packaging/qnap/build.sh # Intel/AMD NAS: packaging/qnap/build/Lens_0.3.0_x86_64.qpkg
packaging/qnap/build.sh --arch arm_64 # ARM NAS: packaging/qnap/build/Lens_0.3.0_arm_64.qpkg
```

- The version comes from `fastapi_backend/pyproject.toml` (`--version` overrides it, 10 characters at most).
- The backend image is the `lean` target by default: it reads audio, video, PDFs and images. `--target full` adds
LibreOffice and Chromium, to read Office files, text, web pages and emails, and object detection, for a much bigger
package. `--extras "msg"` adds Outlook `.msg` emails (see docs/deployment.md).
- Building for the other architecture than your machine's uses QEMU. Docker Desktop has it; on Linux, once:
`docker run --privileged --rm tonistiigi/binfmt --install arm64` (or `amd64`). It's slow; building on a machine of
the NAS's architecture is quicker.
- The first run builds a small `lens-qdk` image with QDK in it, so QDK needn't be installed.
- The package's images are tagged `lens-backend:qnap-<version>` and `lens-frontend:qnap-<version>`, so they never
clash with Lens images built some other way. `--prebuilt` packages images with those tags already in Docker
instead of building them (from CI, say).

Releases get the packages automatically: publishing a GitHub release runs `.github/workflows/qnap.yml`, which builds
both and attaches `Lens_<tag>_x86_64.qpkg` and `Lens_<tag>_arm_64.qpkg` to it (run it by hand from the Actions tab,
with a tag, to rebuild them). The ARM package is built under QEMU, which is slow; set the repository variable
`QNAP_ARM_RUNNER` to an ARM runner (`ubuntu-24.04-arm`) if the repository has one.

## Install on the NAS

1. Install **Container Station** from the App Center and open it once, so it finishes its setup.
2. App Center > the gear icon (top right) > **Install Manually** > choose the `.qpkg` for your NAS > **Install**.
QTS warns that the app isn't digitally signed; accept to continue. (Or, over SSH:
`sh Lens_0.3.0_x86_64.qpkg`.)
3. The first start loads the images into Container Station, which takes a few minutes. When Lens shows as running,
click **Open** in the App Center (or browse to `http://<NAS address>:3000`).
4. Create the first admin account with the **setup code**. It's in the QTS system log (Control Panel > System >
System Logs, an entry from Lens saying "create the first admin account with setup code …"), in `lens.env` in the
data folder (`LENS_SETUP_CODE`), and over SSH: `/share/CACHEDEV1_DATA/.qpkg/Lens/lens.sh setup-code` (the
path depends on the volume you installed to: `getcfg Lens Install_Path -f /etc/config/qpkg.conf`).
5. Follow the setup steps: the first namespace, the model provider, and storage, where you can watch `/audio` (the
NAS's `Multimedia/Lens` folder). Each can be skipped, or answered ahead in `lens.env` (see Settings).

The containers show up in Container Station as the application `lens` (surrealdb, backend, worker, frontend), where
you can see their logs and resource use.

## Where things are

The **data folder** is `/share/Container/lens` (the `Container` shared folder Container Station creates), or
`Lens-data` next to the app's folder if that share doesn't exist. It holds:

| Path | What |
|---|---|
| `lens.env` | Settings and secrets, written on the first start. Keep it with the rest: losing `ARCHIVE_SECRET_KEY` makes stored credentials and model API keys unreadable. |
| `surrealdb/` | The database. |
| `archive/` | Uploads, media and derived files. |

Back up the whole folder (Hybrid Backup Sync works, as it's a shared folder) with Lens stopped. Removing the app keeps
the data folder and drops the images; reinstalling picks up where it was. Delete the folder yourself to remove
everything.

Your **recordings and documents**: Lens sees one folder on the NAS, `Lens` in the `Multimedia` share
(`/share/Multimedia/Lens`, created on the first start), as `/audio`, read-only. The setup steps after the first admin
account (namespace, model provider, storage) can watch it, or a folder in it, so what you put there is picked up. Set
`AUDIO_DIR` in `lens.env` to use another folder. Settings changed in the web app are kept in the database.

## Settings

Edit `lens.env` in the data folder, then stop and start Lens in the App Center (or `lens.sh restart` over SSH).

- **Address and port.** `FRONTEND_URL` is the NAS's LAN address at install time, `http://<ip>:3000`. Change it, and
add the host name to `ARCHIVE_ALLOWED_HOSTS`, if you reach Lens another way: a fixed host name, a reverse proxy with
HTTPS (QTS's own reverse proxy in Control Panel > Network Access, or myQNAPcloud), or if the NAS's address changes.
For another port set `LENS_PORT` and the port in `FRONTEND_URL` alike. Sharing and IIIF need HTTPS.
- **Mail** for password resets: the `MAIL_*` settings.
- **Unattended setup.** The `LENS_ADMIN_*`, `LENS_NAMESPACE`, `LENS_LLM_*` and `LENS_SETUP_WIZARD` settings answer the
first-run setup instead of the web app (see docs/configuration.md); empty ones are left to the web app.
- **Models.** Lens talks to OpenAI-compatible servers; set them up in the web app. A model server on the NAS itself
(Ollama in Container Station, say) is reachable at the NAS's LAN address, not `localhost`.
- To move the data folder: stop Lens, move the folder, then
`setcfg Lens Data_Path /share/<folder> -f /etc/config/qpkg.conf` and start Lens.

## Over SSH

`lens.sh` in the app's folder (`getcfg Lens Install_Path -f /etc/config/qpkg.conf`):

```bash
lens.sh status # the containers
lens.sh logs [backend] # follow the logs (all, or one service)
lens.sh setup-code # the first-admin setup code
lens.sh restart
```

## Upgrading

Build the new version's package and install it over the old one, as in step 2. The new images are loaded on the
next start and the old version's are removed; the data folder carries over.

## How the package is made

`build.sh` builds the images for the NAS's platform, saves them into `<arch>/images/` next to the `qpkg/` sources
(with a `manifest` naming each one), and runs QDK's `qbuild` on the lot. In the package:

- `qpkg.cfg`: requires Container Station 3, opens the web app on port 3000, gives the first start 15 minutes.
- `lens.sh`: the service script. On start it waits for Container Station's Docker, writes `lens.env` with fresh
secrets on the first run, loads any image that isn't in Docker yet, and runs `docker compose up` on
`docker-compose.yml` as the project `lens`. On removal it stops the stack and removes its own images (tagged `qnap-<version>`).
- `docker-compose.yml`: the same stack as `docker-compose.prod.yml`, with the bundled images, the data folder as bind
mounts and the web app on the NAS's port 3000.
- `archive.yaml`: the processing configuration: `/audio` as the folder the web app may watch, and no namespaces, so
the first-run setup asks for one.
103 changes: 103 additions & 0 deletions packaging/qnap/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
#!/usr/bin/env bash
# Builds the Lens QPKG for QNAP NAS: one self-contained .qpkg per architecture, with the Lens images (API and worker,
# web app) and SurrealDB inside, so the NAS needs nothing but Container Station.
#
# packaging/qnap/build.sh # x86_64 (Intel and AMD NAS), the lean image
# packaging/qnap/build.sh --arch arm_64 # ARM NAS (on an x86 machine this needs QEMU, see README.md)
# packaging/qnap/build.sh --arch x86_64 --arch arm_64 --target full --extras "msg"
#
# Needs only Docker with buildx (Linux, macOS, or Windows with WSL); QDK runs in a container built the first time.
# Output: packaging/qnap/build/Lens_<version>_<arch>.qpkg
set -euo pipefail

HERE=$(cd "$(dirname "$0")" && pwd)
REPO=$(cd "$HERE/../.." && pwd)
QDK_REF=v2.5.3
WORK="$HERE/.work"
OUT="$HERE/build"
SURREAL_IMAGE=surrealdb/surrealdb:v3.2.4

ARCHES=()
TARGET=lean # full: adds LibreOffice and Chromium, to read Office files, text, web pages and emails
EXTRAS=""
PREBUILT="" # --prebuilt: package the lens-backend:qnap-<version> and lens-frontend:qnap-<version> images already in Docker
VERSION=$(sed -n 's/^version = "\(.*\)"/\1/p' "$REPO/fastapi_backend/pyproject.toml" | head -n 1)

while [ $# -gt 0 ]; do
case "$1" in
--arch) ARCHES+=("$2"); shift 2 ;;
--target) TARGET="$2"; shift 2 ;;
--extras) EXTRAS="$2"; shift 2 ;;
--version) VERSION="$2"; shift 2 ;;
--prebuilt) PREBUILT=yes; shift ;;
-h|--help) sed -n '2,10p' "$0"; exit 0 ;;
*) echo "unknown option: $1" >&2; exit 2 ;;
esac
done
[ ${#ARCHES[@]} -gt 0 ] || ARCHES=(x86_64)
[ ${#VERSION} -le 10 ] || { echo "QPKG versions are at most 10 characters: $VERSION" >&2; exit 2; }

platform_of() {
case "$1" in
x86_64) echo linux/amd64 ;;
arm_64) echo linux/arm64 ;;
*) echo "unsupported architecture: $1 (x86_64 or arm_64)" >&2; exit 2 ;;
esac
}
for a in "${ARCHES[@]}"; do platform_of "$a" >/dev/null; done

# --- QDK, in a small Linux image of its own (qbuild expects GNU tools under /bin) --------------------------------
docker image inspect "lens-qdk:${QDK_REF#v}" >/dev/null 2>&1 ||
docker build -q -t "lens-qdk:${QDK_REF#v}" --build-arg QDK_REF="$QDK_REF" - >/dev/null <<'DOCKERFILE'
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates git gcc libc6-dev rsync bsdextrautils \
&& rm -rf /var/lib/apt/lists/*
ARG QDK_REF
RUN git clone -q --depth 1 --branch "$QDK_REF" https://github.com/qnap-dev/QDK.git /tmp/QDK \
&& cp -R /tmp/QDK/shared /usr/share/QDK && mkdir -p /etc/config \
&& printf 'QDK_VERSION=%s\nQDK_PATH=/usr/share/QDK\n' "${QDK_REF#v}" > /etc/config/qdk.conf \
&& gcc -o /usr/local/bin/qpkg_encrypt /tmp/QDK/src/qpkg_encrypt.c && rm -rf /tmp/QDK
ENV PATH="/usr/share/QDK/bin:$PATH"
DOCKERFILE

qbuild() {
docker run --rm -u "$(id -u):$(id -g)" -v "$HERE:/qnap" -w /qnap/.work "lens-qdk:${QDK_REF#v}" qbuild "$@"
}

save_args=()
mkdir -p "$OUT"
for arch in "${ARCHES[@]}"; do
platform=$(platform_of "$arch")
echo "==> Lens $VERSION for $arch ($platform), backend target $TARGET"

rm -rf "$WORK"
mkdir -p "$WORK"
cp -R "$HERE/qpkg/." "$WORK/"
images="$WORK/$arch/images"
mkdir -p "$images"

if [ -z "$PREBUILT" ]; then
docker buildx build --platform "$platform" --target "$TARGET" --build-arg EXTRAS="$EXTRAS" \
--load -t "lens-backend:qnap-$VERSION" "$REPO/fastapi_backend"
docker buildx build --platform "$platform" -f "$REPO/nextjs-frontend/Dockerfile.prod" \
--load -t "lens-frontend:qnap-$VERSION" "$REPO/nextjs-frontend"
fi
[ "$(docker image inspect -f '{{.Os}}/{{.Architecture}}' "$SURREAL_IMAGE" 2>/dev/null)" = "$platform" ] ||
docker pull -q --platform "$platform" "$SURREAL_IMAGE"

# With Docker's containerd image store, save only the platform being packaged
docker save --help 2>/dev/null | grep -q -- '--platform' && save_args=(--platform "$platform")
: > "$images/manifest"
for ref in "lens-backend:qnap-$VERSION" "lens-frontend:qnap-$VERSION" "$SURREAL_IMAGE"; do
got=$(docker image inspect -f '{{.Os}}/{{.Architecture}}' "$ref")
[ "$got" = "$platform" ] || { echo "$ref is $got, expected $platform" >&2; exit 1; }
file=$(echo "$ref" | tr '/:' '__').tar # left uncompressed: qbuild compresses the whole package
echo " saving $ref"
docker save ${save_args[@]+"${save_args[@]}"} -o "$images/$file" "$ref"
echo "$ref $file" >> "$images/manifest"
done

qbuild --build-arch "$arch" --build-version "$VERSION" --build-dir /qnap/build
done
rm -rf "$WORK"
ls -lh "$OUT"/*.qpkg
Binary file added packaging/qnap/qpkg/icons/Lens.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/qnap/qpkg/icons/Lens_80.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/qnap/qpkg/icons/Lens_gray.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 9 additions & 0 deletions packaging/qnap/qpkg/package_routines
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
######################################################################
# Lens install and removal hooks, sourced by QDK's qinstall.sh (see QDK's package_routines template for what's
# available). The containers, the image loading and the data folder are lens.sh's job, on the first start; removal
# runs "lens.sh stop" and "lens.sh remove" itself, which keep the data folder.
######################################################################

pkg_post_install(){
$CMD_CHMOD 755 "$SYS_QPKG_DIR/lens.sh"
}
27 changes: 27 additions & 0 deletions packaging/qnap/qpkg/qpkg.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#
# Lens for QNAP (QTS 5 / QuTS hero), built with QDK: https://github.com/qnap-dev/QDK
# packaging/qnap/build.sh fills in QPKG_VER and the per-architecture image folders (x86_64/, arm_64/).
#
QPKG_NAME="Lens"
QPKG_DISPLAY_NAME="Lens"
QPKG_VER="0.3.0"
QPKG_AUTHOR="Adeel Ahmad"
QPKG_LICENSE="MIT"
QPKG_SUMMARY="Searchable archive of your recordings and documents: transcripts, entities, chat."
QPKG_RC_NUM="190"
QPKG_SERVICE_PROGRAM="lens.sh"

# The containers run in Container Station's Docker engine, which also stores their images.
QPKG_REQUIRE="container-station >= 3.0"
QPKG_REQUIRE_MSG="Lens runs in Container Station. Install Container Station 3 from the App Center, then install Lens again."
QTS_MINI_VERSION="5.0.0"

# The web app. lens.sh rewrites Web_Port from LENS_PORT in the data folder's lens.env.
QPKG_WEB_PORT="3000"
QPKG_WEBUI="/"
QPKG_DESKTOP_APP="0"

# Loading the bundled images on the first start takes a few minutes.
QPKG_TIMEOUT="900,180"

QPKG_VOLUME_SELECT="1"
12 changes: 12 additions & 0 deletions packaging/qnap/qpkg/shared/archive.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Processing configuration for Lens on a QNAP NAS (lens.sh mounts it read-only; settings changed in the app are kept in
# the database). The folder AUDIO_DIR in lens.env (/share/Multimedia/Lens by default) appears inside Lens as /audio.
# No namespaces here: the first-run setup in the web app asks for one (or LENS_NAMESPACE in lens.env names it), and
# can watch a folder under /audio into it.
data_dir: /data
server:
# "backend" is the Host the web app uses to reach the API inside the compose network
allowed_hosts: [127.0.0.1, localhost, backend]
workers:
inline: 0 # the worker service does the work
sources:
local_roots: [/audio] # folders the web app may watch
Loading
Loading