Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,11 @@ jobs:
ls -l /dev/kvm
if [ -r /dev/kvm ] && [ -w /dev/kvm ]; then echo "this runner's user can already use KVM"; exit 1; fi

- name: The sandbox's scripts, tested apart (the MCP runner's loop, and the agent's services action)
run: |
docker run --rm -v "$PWD:/lens:ro" python:3.12-slim-trixie sh /lens/microsandbox/test-mcp.sh
sh packaging/host/test-services.sh

- name: Install (as a user would, piped into sh)
run: |
sh < install.sh | tee install.log
Expand Down Expand Up @@ -438,6 +443,36 @@ jobs:
grep -q '^LENS_TARGET=lean$' "$LENS_DIR/.env" || { echo "the run without LENS_TARGET changed the edition"; exit 1; }
curl -fsS -o /dev/null "$(cat setup-url)"

- name: The MCP runner, switched on from the web app, runs inside the sandbox as its own user, and off again
run: |
export PATH="$HOME/.local/bin:$HOME/.microsandbox/bin:$PATH"
ctl="$LENS_DIR/.lens-control"
# what the web app's switch writes (services.py): the agent adds mcp to COMPOSE_PROFILES in .env
turn() {
printf 'id=mcp-%s\naction=services\nprofile=mcp\nturn=%s\n' "$1" "$1" >"$ctl/inbox/.upgrade.ci" && mv "$ctl/inbox/.upgrade.ci" "$ctl/inbox/upgrade"
for _ in $(seq 60); do grep -qx "id=mcp-$1" "$ctl/services-status" 2>/dev/null && grep -qE '^state=(done|failed)$' "$ctl/services-status" && break; sleep 2; done
cat "$ctl/services-status"
grep -qx 'state=done' "$ctl/services-status"
}
turn on
grep -q '^COMPOSE_PROFILES=mcp$' "$LENS_DIR/.env"
for _ in $(seq 30); do msb exec lens -- curl -fsS http://127.0.0.1:8090/health && break; sleep 2; done
msb exec lens -- curl -fsS http://127.0.0.1:8090/health | grep -q '"ok": true'
# as the mcp user, with the volume closed to it, and its token where Lens reads it
# shellcheck disable=SC2016 # expanded in the sandbox
msb exec lens -- sh -c 'for p in /proc/[0-9]*; do tr "\0" " " <"$p/cmdline" 2>/dev/null | grep -q lens-mcp-runner.py && stat -c %U "$p"; done' | grep -qx mcp
if msb exec lens -- setpriv --reuid mcp --regid mcp --clear-groups -- ls /data; then echo "the volume is open to the mcp user"; exit 1; fi
msb exec lens -- setpriv --reuid lens --regid lens --init-groups -- test -s /var/lib/lens-mcp-key/token
# and a server from npm runs in it, through Lens's own client
msb exec lens -- setpriv --reuid lens --regid lens --init-groups -- \
env LENS_MCP_RUNNER_URL=http://127.0.0.1:8090 LENS_MCP_TOKEN_FILE=/var/lib/lens-mcp-key/token \
PYTHONPATH=/data/.lens/backend /data/.lens/venv/bin/python -c 'from app.domain import mcp_client; t = mcp_client.Session({"command": "npx", "args": ["-y", "@modelcontextprotocol/server-memory@2026.8.31"]}).tools(); assert any(x["name"] == "create_entities" for x in t), t; print(len(t), "tools")'
turn off
grep -q '^COMPOSE_PROFILES=' "$LENS_DIR/.env" && { echo "the profile is still in .env"; exit 1; }
for _ in $(seq 30); do msb exec lens -- curl -fsS -o /dev/null http://127.0.0.1:8090/health || break; sleep 2; done
if msb exec lens -- curl -fsS -o /dev/null http://127.0.0.1:8090/health; then echo "the runner still answers"; exit 1; fi
curl -fsS -o /dev/null "$(cat setup-url)"

- name: The full edition installs Chromium the first time Lens needs it, and again at each start
run: |
export PATH="$HOME/.local/bin:$HOME/.microsandbox/bin:$PATH"
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are versioned together.

## Unreleased

### Features

- **microsandbox:** the MCP runner runs inside the sandbox, turned on and off from the web app like Docker's; it runs as a user of its own with the database and the archive closed to it

## 0.16.0 <small>October 18, 2026</small> {id="0.16.0"}

### Features
Expand Down
9 changes: 6 additions & 3 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,9 +188,12 @@ plugin's stdio servers come along when an admin installs it. They run in the MCP
`mcp` Compose profile), a container with none of Lens's data or keys, reached over HTTP with a token on a volume only
it and the backend share. Recorded in the threat model.

Refine later: turn the runner on from Settings through the machine agent, as monitoring does, instead of
`COMPOSE_PROFILES=mcp` in `.env`; the runner on installs that don't use Docker (microsandbox, Cloudron, NAS packages);
plugin servers that run a program from the plugin's own folder; an egress allow-list for the runner's network.
Refine later: ~~turn the runner on from Settings through the machine agent~~ (done, the services switch);
~~the runner in a microsandbox~~ (done 2026-10-18: a process as the `mcp` user inside the sandbox, `microsandbox/mcp.sh`);
the runner on Cloudron and NAS packages; in a microsandbox, keeping the servers' packages across an update (they're on
the sandbox's disk, fetched again after it's replaced) and limits on the runner's memory and processes (the container
has them; a cgroup for the process would); plugin servers that run a program from the plugin's own folder; an egress
allow-list for the runner's network.

## 2026-10-10 · Pasted transcripts fetch no speech model

Expand Down
10 changes: 9 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -932,7 +932,15 @@ as an unprivileged user with every capability dropped, a read-only image and mem
network only with Lens's backend (and the internet, which `npx` and `uvx` need to fetch the servers). Each server gets
an empty folder as its home and only the environment its extension gives it; Lens talks to the runner over HTTP with
a token the runner writes to a volume only the two share. A server starts on first use, stays up while it's used,
and stops after 15 idle minutes. Installs that don't run on Docker don't have the runner yet.
and stops after 15 idle minutes.

In a [microsandbox](microsandbox.md) install the same switch works, and the runner is a process inside the sandbox
rather than a container: it runs as a user of its own, `mcp`, with the volume that holds the database and the archive
closed to it, with Lens's Node and uv reached through read-only mounts, listening on the sandbox's localhost only
(`microsandbox/mcp.sh`). The agent adds `mcp` to `COMPOSE_PROFILES` in `.env` and nothing else; the sandbox reads
`.env` every few seconds and starts or stops the runner to match, so the servers' packages come and go with the
sandbox (they're fetched again after an update). Other installs that don't run on Docker (Cloudron, NAS packages)
don't have the runner yet.

### The browser

Expand Down
6 changes: 4 additions & 2 deletions docs/microsandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ do:
3. Builds the backend and the web app from the code, read-only at `/lens`. It builds again only when the code
changes. The web app is built on the sandbox's own disk, and only what it builds is kept in the volume.
4. Runs SurrealDB, the API, a job worker and the web app, and restarts any of them that stops.
5. Runs the [MCP runner](assistant.md#the-mcp-runner) while `COMPOSE_PROFILES` in `.env` names `mcp`, as the switch
in the web app sets it: a process as its own user, `mcp`, with the volume closed to it, on the sandbox's localhost.

The database, the archive, the builds and the downloaded packages are in the named volume `lens-data`, which outlives
the sandbox. A sandbox that is started again boots without running Lens, so Lens always starts in a fresh one:
Expand Down Expand Up @@ -124,8 +126,8 @@ again for a few seconds, then moves what is left to `/data/.lens/trash` and goes

## Not yet

- **Matterbridge, Fedora and OpenSearch.** The opt-in Compose services don't run in the sandbox yet. Use Docker for
these.
- **Matterbridge, Fedora, OpenSearch, the browser, the code sandbox's gateway and monitoring.** The opt-in Compose
services other than the MCP runner don't run in the sandbox yet. Use Docker for these.
- **Prebuilt images.** Each machine builds Lens itself, as `install.sh` does with Docker. Published images would make
the first start quicker.
- **Moving an install.** A Docker install doesn't move into a sandbox yet: its database and archive stay in Docker's
Expand Down
4 changes: 3 additions & 1 deletion docs/updates.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ profiles are fixed in the agent: `mcp` (the MCP runner), `browser` (the browser
[code sandbox](assistant.md#the-gateway)'s gateway, its certificates and the client's key), `monitoring`
([Charts and history](prometheus.md#prometheus-and-grafana-ready-made): Prometheus, node-exporter and Grafana) and
`mesh` ([Nodes and the mesh](mesh.md#on-this-machine): Nebula on the primary); it refuses any other. It reports in `services-status` and `services.log`. A request from an older Lens, with
`monitoring=on` or `off` and no profile, still works.
`monitoring=on` or `off` and no profile, still works. In a [microsandbox](microsandbox.md) install only `mcp` can be
turned: the agent changes `.env`, and the sandbox, which reads it every few seconds, starts or stops the runner inside
itself ([The MCP runner](assistant.md#the-mcp-runner)); the other profiles are refused there.

With no agent (Lens wasn't installed with install.sh, the machine has neither systemd nor launchd, or install.sh ran
with `LENS_UPGRADE_AGENT=0`), the API gives the one line to run on the machine instead (`manual`).
Expand Down
10 changes: 8 additions & 2 deletions fastapi_backend/app/domain/services.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,15 @@ def _put(db, profile, run):


def manual(profile):
"""The line to run on the machine when there is no agent."""
"""The line to run on the machine when there is no agent (in a microsandbox, where the runner is a process the
sandbox starts as .env says, the setting to write instead: microsandbox/mcp.sh)."""
where = os.environ.get("LENS_HOST_DIR") or ""
cd = f"cd {where} && " if where and updates.REF.match(where.lstrip("/~")) else ""
ok = bool(where and updates.REF.match(where.lstrip("/~")))
if os.environ.get("LENS_BACKEND") == "microsandbox":
if profile != "mcp":
return f"this install runs in a microsandbox, where the {PROFILES[profile][0].lower()} doesn't run yet"
return f"add COMPOSE_PROFILES=mcp to {where + '/' if ok else ''}.env (the sandbox picks it up by itself)"
cd = f"cd {where} && " if ok else ""
return f"{cd}docker compose -f docker-compose.prod.yml --profile {profile} up -d"


Expand Down
11 changes: 11 additions & 0 deletions fastapi_backend/tests/api/test_services.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,17 @@ def test_monitoring_through_the_same_switch(client, db, control):
assert client.get("/api/v1/admin/monitoring", headers=h).json()["on"] is True


def test_in_a_microsandbox_the_runner_is_a_setting_in_env(client, db, monkeypatch):
"""The sandbox starts the runner itself as COMPOSE_PROFILES says (microsandbox/mcp.sh): without an agent, the
line to run is that setting; the other profiles don't run there."""
monkeypatch.delenv("LENS_CONTROL_DIR", raising=False)
monkeypatch.setenv("LENS_BACKEND", "microsandbox")
monkeypatch.setenv("LENS_HOST_DIR", "/home/pi/lens")
h = admin(client, db)
assert of(client, h, "mcp")["manual"] == "add COMPOSE_PROFILES=mcp to /home/pi/lens/.env (the sandbox picks it up by itself)"
assert of(client, h, "browser")["manual"] == "this install runs in a microsandbox, where the browser doesn't run yet"


def test_no_agent_shows_the_line_to_run(client, db, monkeypatch):
monkeypatch.delenv("LENS_CONTROL_DIR", raising=False)
h = admin(client, db)
Expand Down
19 changes: 17 additions & 2 deletions mcp-runner/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import re
import secrets
import shutil
import signal
import subprocess
import sys
import threading
Expand All @@ -38,6 +39,9 @@
HOME = os.environ.get("LENS_MCP_HOME", "/data")
TOKEN_FILE = os.environ.get("LENS_MCP_TOKEN_FILE", "/key/token")
PORT = int(os.environ.get("LENS_MCP_PORT", "8090"))
BIND = os.environ.get(
"LENS_MCP_BIND", "0.0.0.0"
) # 127.0.0.1 where Lens is on the same machine (a microsandbox)
START_SECONDS = 180 # a first `npx -y` downloads the server
CALL_SECONDS = 120
IDLE = 900
Expand Down Expand Up @@ -358,7 +362,7 @@ def do_POST(self):

def serve(port=PORT, token=None):
"""The runner on `port` (0: any); returns the server (its .token, .server_address)."""
srv = ThreadingHTTPServer(("0.0.0.0", port), Handler)
srv = ThreadingHTTPServer((BIND, port), Handler)
srv.daemon_threads = True
srv.token = token or secrets.token_urlsafe(32)
return srv
Expand All @@ -372,7 +376,18 @@ def main():
f.write(srv.token)
os.replace(tmp, TOKEN_FILE)
threading.Thread(target=reap, daemon=True).start()
print(f"MCP runner on :{PORT}", flush=True)

def quit(
*_,
): # stopped (docker stop, or the sandbox's loop): the servers go with it
with LOCK:
for s in SERVERS.values():
with s.lock:
s.stop()
sys.exit(0)

signal.signal(signal.SIGTERM, quit)
print(f"MCP runner on {BIND}:{PORT}", flush=True)
srv.serve_forever()


Expand Down
74 changes: 74 additions & 0 deletions microsandbox/mcp.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# The MCP runner in a microsandbox (docs/assistant.md "The MCP runner", docs/microsandbox.md), sourced by run.sh,
# which defines log(), CODE and OPT. Root only. test-mcp.sh tests it.
#
# With Docker the runner is a container of its own (mcp-runner/, the `mcp` Compose profile). Here it's a process in
# the sandbox as a user of its own, mcp, with the volume closed to it (/data is lens's alone, mode 750), so the servers
# it runs (the npx and uvx packages an admin named) see none of Lens's data, keys or database. Lens's Node and uv are
# given to it through read-only bind mounts, so it needn't see the volume they're kept in. It listens on the sandbox's
# localhost only, and writes its token where Lens reads it.
#
# It runs while COMPOSE_PROFILES in .env names mcp, the same setting as with Docker, which the machine's upgrade agent
# turns from the web app (packaging/host/lens-upgrade.sh: in a microsandbox it changes .env and nothing else).
# mcp_loop reads .env every few seconds (the install folder is mounted read-only, so the change shows at once),
# starts the runner, starts it again if it stopped, and stops it when the profile goes.

MCP_HOME=/var/lib/lens-mcp # the servers' homes and caches: the sandbox's own disk, like the web app's build
MCP_KEY=/var/lib/lens-mcp-key # the token the runner writes and Lens reads
MCP_RUNNER=/var/lib/lens-mcp-runner.py
MCP_TOOLS=/opt/lens-mcp # Node and uv, mounted from the volume
MCP_PORT=8090

# mcp_wanted: whether .env's COMPOSE_PROFILES names mcp, read each time (the agent rewrites .env while Lens runs)
mcp_wanted() {
sed -n 's/^COMPOSE_PROFILES=//p' "$CODE/.env" 2>/dev/null | head -1 | tr ',' '\n' | grep -qx mcp
}

# mcp_prepare: the user, its folders and a copy of the runner (/lens may be mounted with the host's permissions)
mcp_prepare() {
id mcp >/dev/null 2>&1 || useradd --system --no-create-home --shell /usr/sbin/nologin mcp
mkdir -p "$MCP_HOME" "$MCP_KEY" "$MCP_TOOLS/node" "$MCP_TOOLS/bin"
chown mcp:mcp "$MCP_HOME" "$MCP_KEY"
chmod 700 "$MCP_HOME" && chmod 755 "$MCP_KEY"
cp "$CODE/mcp-runner/runner.py" "$MCP_RUNNER" && chmod 644 "$MCP_RUNNER"
}

# mcp_tools: Node and uv for the runner, as read-only bind mounts (a way to them without a way into the volume)
mcp_tools() {
for t in node bin; do
mountpoint -q "$MCP_TOOLS/$t" || mount --bind -o ro "$OPT/$t" "$MCP_TOOLS/$t" || return 1
done
}

# mcp_run: the runner, as the mcp user with only the environment named here (the test stands in for it)
mcp_run() {
exec setpriv --reuid mcp --regid mcp --clear-groups -- env -i \
PATH="$MCP_TOOLS/node/bin:$MCP_TOOLS/bin:/usr/local/bin:/usr/bin:/bin" HOME="$MCP_HOME" LANG=C.UTF-8 \
LENS_MCP_HOME="$MCP_HOME" LENS_MCP_TOKEN_FILE="$MCP_KEY/token" LENS_MCP_BIND=127.0.0.1 LENS_MCP_PORT="$MCP_PORT" \
python3 "$MCP_RUNNER"
}

# mcp_loop: the runner on while COMPOSE_PROFILES names mcp, off when it doesn't, until the sandbox stops
mcp_loop() {
child=""
trap 'if [ -n "$child" ]; then kill "$child" 2>/dev/null; wait "$child" 2>/dev/null; fi; exit 0' TERM
while :; do
if [ -n "$child" ] && ! kill -0 "$child" 2>/dev/null; then
wait "$child" 2>/dev/null || true
child=""
if mcp_wanted; then log "The MCP runner stopped; starting it again in 5 seconds"; fi
fi
if mcp_wanted; then
if [ -z "$child" ]; then
log "Starting the MCP runner (COMPOSE_PROFILES names mcp)..."
(cd "$MCP_HOME" && mcp_run) &
child=$!
fi
elif [ -n "$child" ]; then
log "Stopping the MCP runner (COMPOSE_PROFILES no longer names mcp)"
kill "$child" 2>/dev/null || true
wait "$child" 2>/dev/null || true
child=""
fi
sleep 5
done
}
23 changes: 20 additions & 3 deletions microsandbox/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,10 @@ if [ ! -x "$OPT/node/bin/node" ]; then
tar -xJf "/tmp/lens/$file" -C "$OPT/node" --strip-components 1
rm "/tmp/lens/$file"
fi
if [ ! -x "$OPT/bin/uv" ]; then
if [ ! -x "$OPT/bin/uv" ] || [ ! -x "$OPT/bin/uvx" ]; then # uvx: the MCP runner's, for servers from PyPI
log "Getting uv..."
python3 -m pip install -q --no-cache-dir --target /tmp/lens/uv "uv>=0.8,<0.9"
cp /tmp/lens/uv/bin/uv "$OPT/bin/uv" && rm -rf /tmp/lens/uv
cp /tmp/lens/uv/bin/uv /tmp/lens/uv/bin/uvx "$OPT/bin/" && rm -rf /tmp/lens/uv
fi

# --- Build: the backend and the web app, again only when the code changed (the edition doesn't change them) -------
Expand Down Expand Up @@ -153,8 +153,10 @@ if [ "$(cat "$OPT/build-id" 2>/dev/null)" != "$build_id" ]; then
chown -R lens:lens "$OPT/web"
echo "$build_id" >"$OPT/build-id"
fi
# the archive is the volume itself (data_dir: /data, as in the containers); the database is a folder in it
# the archive is the volume itself (data_dir: /data, as in the containers); the database is a folder in it, and the
# volume is lens's alone: the MCP runner's user (mcp.sh) has no way in
chown lens:lens "$DATA"
chmod 750 "$DATA" || true
chown -R lens:lens "$DATA/surreal"

# public resolvers stood in for the downloads: this machine's again if they answer now, so names on its network resolve
Expand All @@ -180,6 +182,8 @@ mkdir -p "$DATA/.cache" && chown lens:lens "$DATA/.cache"
[ -d /control ] && export LENS_CONTROL_DIR=/control LENS_HOST_DIR="${LENS_DIR:-}"
# the archive in the volume, as /data in the containers; the config file the backend image ships
export ARCHIVE_CONFIG="$OPT/backend/docker/archive.yaml"
# the MCP runner (mcp.sh), where Lens reaches it and reads its token while it runs
export LENS_MCP_RUNNER_URL=http://127.0.0.1:8090 LENS_MCP_TOKEN_FILE=/var/lib/lens-mcp-key/token

# model servers on the host machine (Ollama, LM Studio) under the name the Docker setup uses for them too
if ! grep -q host.docker.internal /etc/hosts; then
Expand Down Expand Up @@ -242,6 +246,19 @@ if [ "$TARGET" = full ]; then
export LENS_PROGRAMS="$PROGRAMS"
fi

# --- The MCP runner: on while COMPOSE_PROFILES names mcp, as the web app's switch sets it (docs/assistant.md) ----
# shellcheck source=SCRIPTDIR/mcp.sh
. "$CODE/microsandbox/mcp.sh"
if mcp_prepare && mcp_tools; then
(
set +e # a runner that stops is started again, and the loop goes on
mcp_loop
) &
pids="$pids $!"
else
log "The MCP runner can't run in this sandbox (its user or tools couldn't be set up); MCP servers that are programs won't"
fi

log "Starting the database..."
keep surrealdb "$DATA" surreal start --log info --bind 127.0.0.1:8001 --user "$SURREAL_USER" --pass "$SURREAL_PASS" \
"surrealkv:$DATA/surreal/lens.db"
Expand Down
Loading
Loading