Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .github/workflows/install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ on:
- fastapi_backend/pyproject.toml
- fastapi_backend/uv.lock
- fastapi_backend/start.sh
- microsandbox/**
- nextjs-frontend/Dockerfile
- nextjs-frontend/package.json
- nextjs-frontend/pnpm-lock.yaml
Expand All @@ -21,6 +22,7 @@ on:
paths:
- install.sh
- docker-compose.prod.yml
- microsandbox/**
- .github/workflows/install.yml
schedule:
- cron: "17 4 * * 1"
Expand Down Expand Up @@ -83,3 +85,63 @@ jobs:
cd "$LENS_DIR"
docker compose -f docker-compose.prod.yml down -v || true
fi

microsandbox:
name: One-line install in a microsandbox
# GitHub's own Linux runners: they have KVM, which microsandbox needs
runs-on: ubuntu-latest
timeout-minutes: 60
env:
LENS_REPO: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
LENS_REF: ${{ github.event.pull_request.head.ref || github.ref_name }}
LENS_DIR: /tmp/lens-install
LENS_NO_OPEN: "1"
LENS_BACKEND: microsandbox
LENS_TARGET: lean
steps:
- uses: actions/checkout@v4

- name: KVM for the runner's user
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
ls -l /dev/kvm

- name: Install (as a user would, piped into sh)
run: |
sh < install.sh | tee install.log
grep -Eo 'http://[^ ]+/setup\?code=[A-Za-z0-9]{12}' install.log | head -1 > setup-url
test -s setup-url || { echo "install.sh didn't hand over a setup address"; exit 1; }

- name: The setup address answers, and Lens waits for its first admin
run: |
url="$(cat setup-url)"
base="http://127.0.0.1:${url##*:}"; base="${base%%/setup*}"
curl -fsS -o /dev/null "$url"
status="$(curl -fsS "$base/api/v1/auth/status")"
echo "$status"
echo "$status" | grep -q '"setup_required":true'
grep -q '^LENS_BACKEND=microsandbox$' "$LENS_DIR/.env"

- name: Running it again updates in place and keeps the secrets
run: |
before="$(grep -E '^(ACCESS_SECRET_KEY|ARCHIVE_SECRET_KEY|AUTH_SECRET|SURREAL_PASS|LENS_SETUP_CODE)=' "$LENS_DIR/.env" | sha256sum)"
LENS_BACKEND='' sh < install.sh
after="$(grep -E '^(ACCESS_SECRET_KEY|ARCHIVE_SECRET_KEY|AUTH_SECRET|SURREAL_PASS|LENS_SETUP_CODE)=' "$LENS_DIR/.env" | sha256sum)"
test "$before" = "$after" || { echo "the second run changed the secrets"; exit 1; }
curl -fsS -o /dev/null "$(cat setup-url)"

- name: What the sandbox said
if: failure()
run: |
export PATH="$HOME/.local/bin:$PATH"
msb doctor || true
msb ls || true
msb logs lens --tail 300 || true

- name: Stop
if: always()
run: |
export PATH="$HOME/.local/bin:$PATH"
msb rm -f lens || true
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -170,3 +170,5 @@ local-shared-data/*

# ci/local-ci.sh logs
/.local-ci/
.lens-build
.lens-sandbox
4 changes: 4 additions & 0 deletions docs/get-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ before `sh`: `LENS_DIR`, `LENS_PORT`, `LENS_PUBLIC` (`1`: from the network, `0`:
(`lean` for the smaller image) and `LENS_REF` (a branch or tag) change the defaults; the top of `install.sh` lists
them.

Where it can (Linux with KVM, or a Mac with Apple silicon), a first install asks whether to run Lens in Docker (the
default) or in a [microsandbox](microsandbox.md) microVM, which needs no Docker; `LENS_BACKEND=microsandbox` chooses
it without asking.

## With Docker and make

You need Docker with Compose, and `make`.
Expand Down
72 changes: 72 additions & 0 deletions docs/microsandbox.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# microsandbox

Lens can run in a [microsandbox](https://github.com/superradcompany/microsandbox) microVM instead of Docker: one small
virtual machine with its own kernel, and no Docker on the machine. Docker stays the default.

**Status: built, early.** microsandbox itself is beta software.

## Install

Run the one-line install. On a first install where microsandbox can run, it asks which to use:

```
How should Lens run?
1) Docker (default): a container for each part, the usual way
2) microsandbox: one lightweight virtual machine, no Docker needed
```

Or choose without the question: `curl -fsSL .../install.sh | LENS_BACKEND=microsandbox sh`. The choice is kept in
`.env` (`LENS_BACKEND`), so running the line again updates the same install. An install that already runs on Docker
stays on Docker, because its database and archive are in Docker volumes.

It needs hardware virtualization:

- **Linux** on x86-64 or 64-bit ARM (a Raspberry Pi 4 or 5 on a 64-bit OS) with `/dev/kvm`. The installer adds you to
the `kvm` group. `msb doctor` checks the machine.
- **A Mac** with Apple silicon.

## How it runs

microsandbox runs images but can't build them, and its sandboxes can't reach each other by name, so Lens runs as one
sandbox named `lens`, started from the stock `python:3.12-slim-bookworm` image. Inside it,
[`microsandbox/run.sh`](https://github.com/adeelahmad/lense/blob/main/microsandbox/run.sh) does what the Docker images
do:

1. Installs the system packages, the same ones as the backend image (`LENS_TARGET=lean` leaves out LibreOffice and
Chromium).
2. Fetches SurrealDB 3.2.4, rclone, Node 22 and uv.
3. Builds the backend and the web app from the code, read-only at `/lens`. It builds again only when the code
changes.
4. Runs SurrealDB, the API, a job worker and the web app, and restarts any of them that stops.

The database, the archive, the builds and the downloaded packages are in the named volume `lens-data`, which outlives
the sandbox. A sandbox that is started again boots without running Lens, so Lens always starts in a fresh one:
`~/lens/.lens-sandbox`, which install.sh writes, replaces the sandbox. The first start downloads and builds everything,
which takes several minutes (longer on a Raspberry Pi). Later starts install the packages from the volume and reuse the
build, so they take a minute or two.

The sandbox can reach the internet, your network (mail servers, model servers) and this machine. On this machine,
Ollama or LM Studio are at `host.docker.internal`, the same address as with Docker. Only the web app's port and the
sensor ports are published, on the same addresses Docker would use (`LENS_BIND`, `LENS_SENSOR_BIND`).

On Linux with systemd, the installer adds a `lens-microsandbox` service that runs `.lens-sandbox` when the machine
starts. On a Mac, it adds a LaunchAgent that runs it at sign-in.

## Day to day

```bash
msb logs -f lens # follow the logs
msb stop lens # stop it; ~/lens/.lens-sandbox starts it again
msb exec lens -- sh # a shell inside it
```

To update, run the install line again.

## Not yet

- **Matterbridge, Fedora and OpenSearch.** The opt-in Compose services don't run in the sandbox yet. Use Docker for
these.
- **Prebuilt images.** Each machine builds Lens itself, as `install.sh` does with Docker. Published images would make
the first start quicker.
- **Moving an install.** A Docker install doesn't move into a sandbox yet: its database and archive stay in Docker's
volumes.
19 changes: 12 additions & 7 deletions fastapi_backend/app/domain/syslog.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,18 @@ class TcpServer(socketserver.ThreadingTCPServer):

if ":" in host:
UdpServer.address_family = TcpServer.address_family = socket.AF_INET6
self.udp = UdpServer((host, port), Udp)
self.port = self.udp.server_address[1]
try:
self.tcp = TcpServer((host, self.port), Tcp) if tcp else None
except OSError:
self.udp.server_close()
raise
# Port 0 (any free one): UDP picks it, and TCP needs the same number free too; when something else holds it
# on TCP, try another rather than fail
for attempt in range(1 if port else 20):
self.udp = UdpServer((host, port), Udp)
self.port = self.udp.server_address[1]
try:
self.tcp = TcpServer((host, self.port), Tcp) if tcp else None
break
except OSError:
self.udp.server_close()
if port or attempt == 19:
raise
self.threads = []

def start(self):
Expand Down
20 changes: 20 additions & 0 deletions fastapi_backend/tests/api/test_sensors.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,26 @@ def test_syslog_tcp_framing():
assert list(syslog._frames(io.BufferedReader(io.BytesIO(lines)))) == [b"<30>a b: c\n", b"<30>d e: f\r\n"]


def test_syslog_any_free_port_skips_one_taken_on_tcp(monkeypatch):
"""Port 0: UDP picks a port that may be in use on TCP (other tests, other programs); another is tried."""
import socketserver

bind, tries = socketserver.ThreadingTCPServer.server_bind, []

def busy_once(self):
tries.append(self.server_address)
if len(tries) == 1:
raise OSError(98, "Address already in use")
bind(self)

monkeypatch.setattr(socketserver.ThreadingTCPServer, "server_bind", busy_once)
listener = syslog.Listener("127.0.0.1", 0, [], lambda *a: None).start()
try:
assert len(tries) == 2 and listener.tcp.server_address[1] == listener.port
finally:
listener.stop()


# ---------- the broker on its own ----------
def test_broker_is_an_ordinary_local_broker():
got = []
Expand Down
Loading
Loading