Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,11 @@ jobs:
grep -Eo 'http://[^ ]+/setup\?code=[A-Za-z0-9]{12}' install.log | head -1 > setup-url
test -s setup-url || { echo "install.sh didn't hand over a setup address"; exit 1; }

- name: When this machine's DNS has no address for a name, the sandbox looks it up with public resolvers
run: |
export PATH="$HOME/.local/bin:$HOME/.microsandbox/bin:$PATH"
msb exec lens -- sh /lens/microsandbox/test-dns.sh

- name: The setup address answers, and Lens waits for its first admin
run: |
url="$(cat setup-url)"
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **A microsandbox install gets past name lookups that fail.** The first build in a microsandbox could stop at the
backend's packages with `dns error ... No address associated with hostname` when this machine's resolvers didn't
answer inside the sandbox. The sandbox now waits for the names it downloads from, looks them up with public
resolvers (1.1.1.1 and 9.9.9.9; `LENS_DNS_FALLBACK` changes or turns that off) when this machine's don't answer
there, and runs each download step up to three times ([microsandbox](docs/microsandbox.md#name-lookups)).

- **A Mac app.** Lens.app (`Lens-macOS-arm64.dmg`, Apple silicon) installs Lens in the web installer shown in its
window, with microsandbox inside it so no Docker is needed, then lives in the menu bar: open, start, stop and update
Lens, and show its logs. Lens itself opens in the browser, where passkeys work
Expand Down
1 change: 1 addition & 0 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Lens is one repository with several projects, and [Nx](https://nx.dev/) runs the
| `docs` | `docs/` (with `mkdocs.yml`, `overrides/`, `README.md`, `CHANGELOG.md`) | `build` (strict, as CI), `serve` |
| `clipper` | `extensions/browser/` | `install` (`npm ci` when the lockfile is newer than `node_modules`), `test`, `build` (every browser target, checked) |
| `desktop` | `desktop/` | `lint` (`sh -n` and shellcheck on `lens-app.sh`), `test` (`lens-app.sh` checks on Linux; the Mac build stays in the Desktop workflow) |
| `microsandbox` | `microsandbox/` | `lint` (`sh -n` and shellcheck on `run.sh` and `dns.sh`), `test` (`test-dns.sh`, the name-lookup fallback, in the base image with Docker; the install itself runs in the Install workflow) |

```bash
pnpm exec nx run backend:test # one target (make test-backend)
Expand Down
22 changes: 20 additions & 2 deletions docs/microsandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ Docker Hub, which need no sign-in; Docker Hub itself last). Inside it,
[`microsandbox/run.sh`](https://github.com/adeelahmad/lense/blob/main/microsandbox/run.sh) does what the Docker images
do:

1. Installs the system packages, the same ones as the backend image (`LENS_TARGET=lean` leaves out LibreOffice and
Chromium).
1. Waits until the names it downloads from resolve (see [Name lookups](#name-lookups)), then installs the system
packages, the same ones as the backend image (`LENS_TARGET=lean` leaves out LibreOffice and Chromium).
2. Fetches SurrealDB 3.2.4, rclone, Node 22 and uv.
3. Builds the backend and the web app from the code, read-only at `/lens`. It builds again only when the code
changes.
Expand All @@ -53,6 +53,24 @@ sensor ports are published, on the same addresses Docker would use (`LENS_BIND`,
On Linux with systemd, the installer adds a `lens-microsandbox` service that runs `.lens-sandbox` when the machine
starts. On a Mac, it adds a LaunchAgent that runs it at sign-in.

### Name lookups

microsandbox answers the sandbox's name lookups itself, from this machine's resolvers. When those don't answer inside
the sandbox (the first build once stopped with uv's `dns error ... No address associated with hostname`), the steps
that download would fail, so the sandbox:

- waits for `deb.debian.org`, `files.pythonhosted.org` and `registry.npmjs.org` to resolve before it downloads;
- after about 15 seconds without an answer, looks names up with public resolvers instead (Cloudflare's 1.1.1.1 and
Quad9's 9.9.9.9), and says so in its log;
- runs each download step (the system packages, the backend's and the web app's packages) up to three times, with
fewer downloads at once after a failure;
- when Lens starts, goes back to this machine's resolvers if they answer by then, so names on your own network
resolve. Otherwise it keeps the public ones until the sandbox is replaced, and logs that.

If nothing answers within a minute or two, it stops with the name that didn't resolve and why. To use other public
resolvers, or none, set `LENS_DNS_FALLBACK` on the install line (`... | LENS_DNS_FALLBACK=8.8.8.8,8.8.4.4 sh`, or
`LENS_DNS_FALLBACK= sh` for none); it is kept in `.env` for later runs.

## Day to day

```bash
Expand Down
5 changes: 5 additions & 0 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
# LENS_TARGET full (reads Office files, web pages and emails too) or lean (smaller) (default: full)
# LENS_BACKEND docker or microsandbox (one microVM, no Docker: docs/microsandbox.md). Asked on a first install
# when microsandbox can run here (Linux with KVM, or a Mac with Apple silicon); default: docker
# LENS_DNS_FALLBACK in a microsandbox: the public resolvers it looks names up with when this machine's don't
# answer there, comma-separated (default: 1.1.1.1,9.9.9.9; empty: none; docs/microsandbox.md).
# Kept for later runs
# GITHUB_TOKEN a GitHub token to use when downloading (not needed; avoids GitHub's rate limits)
# LENS_NO_OPEN 1: don't open a browser
# LENS_YES 1: install what Lens needs without asking (it only asks at a terminal)
Expand Down Expand Up @@ -361,6 +364,8 @@ fi
setenv LENS_DIR "$DIR"
# the SMB share's port on this machine (docs/smb.md), when the install line names one: kept for later runs
case "${LENS_SMB_PORT:-}" in '' | *[!0-9]*) ;; *) setenv LENS_SMB_PORT "$LENS_SMB_PORT" ;; esac
# the microsandbox's public resolvers (microsandbox/dns.sh), when the install line names them, empty included: kept
[ -z "${LENS_DNS_FALLBACK+set}" ] || setenv LENS_DNS_FALLBACK "$(printf '%s' "$LENS_DNS_FALLBACK" | tr -s ' ' ',')"
# from the web installer (docs/configuration.md): the first namespace, and settings Lens saves once at its first start
for k in LENS_NAMESPACE LENS_SETUP_SETTINGS; do
eval "v=\${$k:-}"
Expand Down
78 changes: 78 additions & 0 deletions microsandbox/dns.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Name lookups in the sandbox (docs/microsandbox.md), sourced by run.sh, which defines log(). Root only: it may
# rewrite /etc/resolv.conf (in place: in a container it is a mount). test-dns.sh tests it.
#
# The sandbox asks microsandbox, which answers from this machine's resolvers. A download stops at the first name that
# doesn't resolve (uv: "dns error ... No address associated with hostname"), so run.sh waits for the names it needs
# before downloading, and retries what downloads. When this machine's resolvers don't answer those names in the
# sandbox, it asks public ones instead while it installs and builds (LENS_DNS_FALLBACK in .env, comma-separated;
# default Cloudflare's and Quad9's; empty: never), and keeps them only if this machine's still don't answer when Lens
# starts.

NAMES="deb.debian.org files.pythonhosted.org registry.npmjs.org"
DNS_FALLBACK="$(printf '%s' "${LENS_DNS_FALLBACK-1.1.1.1,9.9.9.9}" | tr ',' ' ')"
# tries 3 seconds apart: the public resolvers from the 5th, this machine's again from the 10th
DNS_TRIES=15

# NAME...: whether they all resolve, the way apt, uv and pnpm look them up; if not, which didn't and why (stderr)
resolves() {
RES_OPTIONS="timeout:2 attempts:1" python3 -c '
import socket, sys
for name in sys.argv[1:]:
try:
socket.getaddrinfo(name, 443, type=socket.SOCK_STREAM)
except OSError as e:
sys.exit(f"{name}: {e.strerror or e}")' "$@"
}

# Wait until NAMES resolve: switch to the public resolvers if this machine's don't answer them, give up after a while
dns_ready() {
tries=0
# shellcheck disable=SC2086
until why="$(resolves $NAMES 2>&1)"; do
tries=$((tries + 1))
if [ "$tries" = 1 ]; then log "Waiting for name lookups ($why)..."; fi
if [ "$tries" = 5 ] && [ -n "$DNS_FALLBACK" ] && [ ! -f /etc/resolv.conf.sandbox ]; then
cp /etc/resolv.conf /etc/resolv.conf.sandbox
# shellcheck disable=SC2086
printf 'nameserver %s\n' $DNS_FALLBACK >/etc/resolv.conf
log "This machine's DNS doesn't answer in the sandbox ($why); looking names up with $DNS_FALLBACK instead"
fi
if [ "$tries" = 10 ] && [ -f /etc/resolv.conf.sandbox ]; then # the public ones don't answer either
cp /etc/resolv.conf.sandbox /etc/resolv.conf && rm /etc/resolv.conf.sandbox
fi
if [ "$tries" -ge "$DNS_TRIES" ]; then
log "Can't look up names in the sandbox ($why)." \
"Check this machine's internet connection, then run the install line again."
return 1
fi
sleep 3
done
}

# COMMAND...: run it up to three times; before each retry the names are checked again, and uv downloads fewer at once
retry() {
n=1
until "$@"; do
[ "$n" -lt 3 ] || return 1
log "That failed; trying again in $((n * 15)) seconds..."
sleep $((n * 15))
dns_ready || return 1
export UV_CONCURRENT_DOWNLOADS=4
n=$((n + 1))
done
}

# After the downloads: this machine's resolvers again if they answer now (names on its own network resolve only there)
dns_restore() {
[ -f /etc/resolv.conf.sandbox ] || return 0
cp /etc/resolv.conf /etc/resolv.conf.fallback
cp /etc/resolv.conf.sandbox /etc/resolv.conf
# shellcheck disable=SC2086
if why="$(resolves $NAMES 2>&1)"; then
rm -f /etc/resolv.conf.sandbox /etc/resolv.conf.fallback
log "This machine's DNS answers in the sandbox again; using it"
else
cp /etc/resolv.conf.fallback /etc/resolv.conf && rm /etc/resolv.conf.fallback
log "Looking names up with $DNS_FALLBACK while Lens runs ($why)"
fi
}
26 changes: 26 additions & 0 deletions microsandbox/project.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"$schema": "../node_modules/nx/schemas/project-schema.json",
"name": "microsandbox",
"projectType": "application",
"tags": ["lang:shell"],
"targets": {
"lint": {
"executor": "nx:run-commands",
"options": {
"cwd": "{projectRoot}",
"commands": [
"sh -n run.sh && sh -n dns.sh && sh -n test-dns.sh",
"uvx --from shellcheck-py shellcheck -s sh -x run.sh dns.sh test-dns.sh"
],
"parallel": false
}
},
"test": {
"executor": "nx:run-commands",
"options": {
"cwd": "{workspaceRoot}",
"command": "docker run --rm -v \"$PWD:/lens:ro\" mirror.gcr.io/library/python:3.12-slim-bookworm sh /lens/microsandbox/test-dns.sh"
}
}
}
}
20 changes: 14 additions & 6 deletions microsandbox/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ esac
mkdir -p "$OPT/bin" "$DATA/surreal" /tmp/lens
export PATH="$OPT/bin:$OPT/node/bin:$OPT/venv/bin:$PATH"

# Name lookups: wait for them before downloading, with public resolvers when this machine's don't answer here
# shellcheck source=SCRIPTDIR/dns.sh
. "$CODE/microsandbox/dns.sh"
dns_ready

# --- System packages: in the sandbox's own disk, installed on each start (install.sh replaces the sandbox), from
# the packages kept in the volume, so only the first start downloads them ------------------------------------------
packages="ffmpeg tesseract-ocr antiword poppler-utils ca-certificates curl xz-utils unzip"
Expand All @@ -48,9 +53,9 @@ if [ ! -f "$stamp" ]; then
export DEBIAN_FRONTEND=noninteractive
rm -f /etc/apt/apt.conf.d/docker-clean # the image's setting that deletes downloaded packages
mkdir -p "$OPT/apt/partial"
apt-get update -q
retry apt-get update -q --error-on=any
# shellcheck disable=SC2086
apt-get install -y -q --no-install-recommends -o Dir::Cache::archives="$OPT/apt" $packages
retry apt-get install -y -q --no-install-recommends -o Dir::Cache::archives="$OPT/apt" $packages
rm -rf /var/lib/apt/lists/*
id lens >/dev/null 2>&1 || useradd --create-home --uid 1000 lens
touch "$stamp"
Expand Down Expand Up @@ -92,14 +97,14 @@ if [ "$(cat "$OPT/build-id" 2>/dev/null)" != "$build_id" ]; then

# shellcheck disable=SC2046
(cd "$OPT/backend" &&
UV_PROJECT_ENVIRONMENT="$OPT/venv" UV_PYTHON_DOWNLOADS=never UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy \
UV_HTTP_TIMEOUT=300 UV_HTTP_RETRIES=5 UV_CACHE_DIR="$OPT/uv-cache" \
uv sync --frozen --no-dev --python "$(command -v python3)" \
export UV_PROJECT_ENVIRONMENT="$OPT/venv" UV_PYTHON_DOWNLOADS=never UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy \
UV_HTTP_TIMEOUT=300 UV_HTTP_RETRIES=5 UV_CACHE_DIR="$OPT/uv-cache" &&
retry uv sync --frozen --no-dev --python "$(command -v python3)" \
$(for e in ${EXTRAS:-}; do printf -- '--extra %s ' "$e"; done))

(cd "$OPT/web-src" &&
export COREPACK_ENABLE_DOWNLOAD_PROMPT=0 COREPACK_HOME="$OPT/corepack" &&
corepack pnpm install --frozen-lockfile &&
retry corepack pnpm install --frozen-lockfile &&
AUTH_SECRET=build-only NEXT_TELEMETRY_DISABLED=1 NEXT_OUTPUT=standalone corepack pnpm build)
rm -rf "$OPT/web" && mkdir -p "$OPT/web/.next"
cp -a "$OPT/web-src/.next/standalone/." "$OPT/web/"
Expand All @@ -113,6 +118,9 @@ fi
chown lens:lens "$DATA"
chown -R lens:lens "$DATA/surreal"

# public resolvers stood in for the downloads: this machine's again if they answer now, so names on its network resolve
dns_restore

# --- Run ---------------------------------------------------------------------------------------------------------
# The same settings docker-compose.prod.yml gives the containers, with every service on this sandbox's localhost
export SURREAL_URL=ws://127.0.0.1:8001 SURREAL_USER="${SURREAL_USER:-root}"
Expand Down
96 changes: 96 additions & 0 deletions microsandbox/test-dns.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
#!/bin/sh
# Tests dns.sh against a resolver that answers every lookup with no address, the way uv's lookups failed in a
# microsandbox ("No address associated with hostname"). Run it as root in the image run.sh runs in, with the internet:
#
# docker run --rm -v "$PWD:/lens:ro" python:3.12-slim-bookworm sh /lens/microsandbox/test-dns.sh
# msb exec lens -- sh /lens/microsandbox/test-dns.sh # in the sandbox itself, through microsandbox's DNS
#
# It puts /etc/resolv.conf back as it was when it ends.
set -eu

here="$(cd "$(dirname "$0")" && pwd)"
work="$(mktemp -d)"
cp /etc/resolv.conf "$work/resolv.conf"
server=""
finish() {
[ -z "$server" ] || kill "$server" 2>/dev/null || true
cp "$work/resolv.conf" /etc/resolv.conf
rm -f /etc/resolv.conf.sandbox /etc/resolv.conf.fallback
rm -rf "$work"
}
trap finish EXIT

log() { printf '[lens] %s\n' "$*" | tee -a "$work/log"; }
fail() {
printf 'FAIL: %s\n' "$*" >&2
exit 1
}
ok() { printf 'ok: %s\n' "$*"; }

# A resolver on this machine's loopback that knows every name and has no address for any of them
python3 -c '
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(("127.0.0.1", 53))
while True:
q, peer = s.recvfrom(4096)
flags = (q[2] & 0x79 | 0x80).to_bytes(1, "big") + b"\x80" # a response, recursion available, NOERROR
s.sendto(q[:2] + flags + q[4:6] + b"\x00\x00\x00\x00\x00\x00" + q[12:], peer) # the question, no answers
' &
server=$!
sleep 1
nodata="nameserver 127.0.0.1"

# shellcheck source=SCRIPTDIR/dns.sh
. "$here/dns.sh"

# 1. The failure, reproduced: the names don't resolve
echo "$nodata" >/etc/resolv.conf
if why="$(resolves files.pythonhosted.org 2>&1)"; then fail "the empty resolver gave an address"; fi
case "$why" in *"No address associated with hostname"*) ok "reproduced: $why" ;; *) fail "unexpected: $why" ;; esac

# 2. It waits, then looks names up with the public resolvers, and they answer
dns_ready || fail "dns_ready gave up with the public resolvers to fall back on"
grep -q '^nameserver 1.1.1.1$' /etc/resolv.conf || fail "resolv.conf doesn't name them: $(cat /etc/resolv.conf)"
[ "$(cat /etc/resolv.conf.sandbox)" = "$nodata" ] || fail "the sandbox's own resolv.conf wasn't kept"
# shellcheck disable=SC2086
resolves $NAMES || fail "the names don't resolve with the public resolvers"
grep -q "looking names up with 1.1.1.1 9.9.9.9 instead" "$work/log" || fail "it didn't say it switched"
ok "fell back to the public resolvers"

# 3. Lens starts: this machine's resolvers still have no address, so the public ones stay
dns_restore
grep -q '^nameserver 1.1.1.1$' /etc/resolv.conf || fail "the public resolvers didn't stay"
[ -f /etc/resolv.conf.sandbox ] || fail "the sandbox's own resolv.conf is gone"
ok "kept the public resolvers while this machine's don't answer"

# 4. Lens starts and this machine's resolvers answer again: they are used
cp "$work/resolv.conf" /etc/resolv.conf.sandbox
dns_restore
[ "$(cat /etc/resolv.conf)" = "$(cat "$work/resolv.conf")" ] || fail "this machine's resolv.conf wasn't put back"
[ ! -f /etc/resolv.conf.sandbox ] || fail "resolv.conf.sandbox is left behind"
ok "went back to this machine's resolvers when they answered"

# 5. With no public resolvers allowed (LENS_DNS_FALLBACK empty), it gives up and says why
echo "$nodata" >/etc/resolv.conf
# shellcheck disable=SC2034 # read by dns_ready
DNS_FALLBACK="" DNS_TRIES=2
if dns_ready; then fail "dns_ready succeeded with nothing that answers"; fi
grep -q "Can't look up names in the sandbox (deb.debian.org: No address associated with hostname)" "$work/log" ||
fail "it didn't say why it gave up: $(tail -1 "$work/log")"
[ "$(cat /etc/resolv.conf)" = "$nodata" ] || fail "resolv.conf changed with no public resolvers allowed"
ok "gave up with the reason when nothing answers"

# 6. A step that fails once runs again
cp "$work/resolv.conf" /etc/resolv.conf
# shellcheck disable=SC2034 # read by dns_ready
DNS_TRIES=15
once() {
[ -f "$work/ran" ] && return 0
touch "$work/ran" && return 1
}
retry once || fail "retry didn't run the step again"
[ "${UV_CONCURRENT_DOWNLOADS:-}" = 4 ] || fail "the retry didn't make uv download fewer at once"
ok "a failed step ran again"

echo "All DNS tests passed"
Loading