From 3cacb110d8aef635067ea98adf0fd79a6180d581 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 06:09:42 +0000 Subject: [PATCH 1/4] feat(cli): lens ext lists, pulls, checks and pushes extension manifests `lens ext pull` writes extensions' manifests into a folder, `push` saves the changed ones as new versions and the new ones as extensions, `check` checks them without saving, and `list` lists them (--json for agents). It works on this install's database, as the first admin or --as, or on any Lens server with --url and an API token. Pulled files carry `version: N`, so a push won't save over changes made in Lens since, and a pull won't overwrite a file's own changes, without --force. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01FbmJ86QaV2YwusjgNowpf8 --- CHANGELOG.md | 6 + DEVLOG.md | 2 +- README.md | 3 +- docs/assistant.md | 23 ++ docs/get-started.md | 1 + docs/requirements.md | 1 + fastapi_backend/app/cli.py | 5 + fastapi_backend/app/cli_ext.py | 420 ++++++++++++++++++++++ fastapi_backend/tests/api/test_cli_ext.py | 161 +++++++++ 9 files changed, 620 insertions(+), 2 deletions(-) create mode 100644 fastapi_backend/app/cli_ext.py create mode 100644 fastapi_backend/tests/api/test_cli_ext.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ff564df5e..a71afeefe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased +- **`lens ext`: extensions as files.** `lens ext pull` writes your extensions' manifests into a folder (skills and + prompt tools as Markdown, the rest as YAML), `lens ext push` saves the changed ones as new versions and the new ones + as extensions, `lens ext check` checks them without saving, and `lens ext list` lists them (`--json` for agents). It + works on this install's database, or on any Lens server with `--url` and an API token. A pulled file says its + version, so a push won't save over changes made in Lens since, and a pull won't overwrite changes made in the file, + without `--force` ([Manifests as files](docs/assistant.md#manifests-as-files-lens-ext)). - **The upgrade agent starts the code sandbox's gateway.** Settings → Code sandbox → Run the gateway on this machine now works on installs with an upgrade agent: the agent knows the `sandbox` profile and starts or stops its three containers, as it does the MCP runner, the browser and monitoring. An older agent still says to run the line by hand diff --git a/DEVLOG.md b/DEVLOG.md index 718447286..1f3f1e2f1 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -559,7 +559,7 @@ Todo: - [x] Canvas editor for tool graphs in the web app (Extensions → Draw a tool) - [x] Authoring from chat and voice: the assistant drafts and saves extensions behind an approval card - [x] Extensions page in the web app: list, manifest editor, switch on/off, share, try, versions -- [ ] `lens ext` CLI (push/pull manifests) +- [x] `lens ext` CLI: list, pull, push and check manifests, here or over the API (docs/assistant.md) Refine later: hook-called tools don't show as steps; web tools reach ports 80/443 only and have no secret references yet; extensions aren't offered over MCP yet; marketplace on top of plugins. diff --git a/README.md b/README.md index f2f6051ee..26e854386 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p skills, hooks and MCP servers, by code, on the canvas or just by asking. Remote MCP servers connect by address, and ones that run as a program (`npx`, `uvx`) run in a sandboxed container of their own, off until you turn it on. Add one with a form or by pasting another app's config, its tokens sealed and each of its tools switched on or off. - Install skills and plugins from marketplaces in the open plugin format, or from a repository or an archive. + Install skills and plugins from marketplaces in the open plugin format, or from a repository or an archive, and keep + your own as files in a folder with `lens ext` (pull, push, check). ([Extending the assistant](docs/assistant.md#extending-the-assistant-tools-skills-hooks-and-plugins), [adding an MCP server](docs/assistant.md#adding-an-mcp-server), [the MCP runner](docs/assistant.md#the-mcp-runner); [checks](docs/requirements.md#extend-the-assistant)) diff --git a/docs/assistant.md b/docs/assistant.md index f5cad029a..8d35af2df 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -762,6 +762,29 @@ when: someone asks for a recap of a meeting Find the meeting, read it, and answer with three bullets: decisions, owners, dates. ``` +### Manifests as files: `lens ext` + +`lens ext` keeps extensions as files in a folder, for a git repository or an agent that writes them: + +```bash +lens ext list # the extensions you can see; --json for scripts and agents +lens ext pull --dir extensions # yours (admins: all) as files; names or ids pick some, --all adds shared ones +lens ext check extensions # read and check them without saving; exits 1 when one doesn't check +lens ext push extensions # changed ones become new versions, new ones extensions; --notes says what changed +``` + +With no address it works on this install's database, as the first admin or the account `--as` names (run it with +`docker compose exec backend lens ext …`). With `--url` and `--token` (or `LENS_URL` and `LENS_TOKEN`) it works on +any Lens server through `/api/v1/extensions`, as the [API token](authentication.md#api-tokens)'s owner; pushing takes a +write token. + +A skill or a prompt tool is pulled as `.md` (Markdown with frontmatter), the rest as `.yaml`; a folder's +other `.md` files, a README, are left alone. A pulled file says the version it was pulled at (`version: 3`, which the +check ignores), so neither side's changes are lost: a push refuses when the extension changed in Lens since that +version, and a pull refuses to overwrite a file changed since it was pulled, each until it's given `--force`. Pushing +a file that says what Lens has, written differently or with the defaults left out, saves nothing. `visibility` and +`namespaces` in a file share the extension as they say when it's pushed; without them, who sees it stays as it is. + ### Skills and plugins from marketplaces Lens reads Anthropic's open plugin and skill formats as they are, so what's made for Claude installs here unchanged: diff --git a/docs/get-started.md b/docs/get-started.md index 546c45ad0..e319fbbc3 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -203,6 +203,7 @@ lens embed # index what isn't yet searchable by lens reindex # after changing search.engine or search.stemming lens migrations # database upgrades: run, pending or failed lens backup # back the database up into /backups +lens ext pull --dir extensions # extensions' manifests as files; lens ext push sends changes back ``` Docker on macOS can't use the Apple GPU. Keep the database and API in Docker and run a native worker against the same diff --git a/docs/requirements.md b/docs/requirements.md index a7f3a2b3c..730ffa3b9 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -101,6 +101,7 @@ request, to another check that proves the same thing; loosening a requirement or | Remote MCP servers connect by address | [test_mcp_servers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp_servers.py) `test_an_admin_adds_a_server_and_the_assistant_uses_it` | | MCP servers that run as a program (`npx`, `uvx`) run in a sandboxed container of their own | [test_mcp_servers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp_servers.py) `test_a_server_that_runs_as_a_program_runs_in_the_runner`; [mcp-runner.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/mcp-runner.yml) `Runs an npm server and a PyPI server` | | An MCP server added by form or a pasted config, its tokens sealed and each tool switched on or off | [test_mcp_servers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp_servers.py) `test_a_pasted_config_reads_as_servers_to_add`, `test_its_secrets_are_sealed_and_never_shown`, `test_each_tool_is_switched_and_says_whether_it_asks`; [mcp-server-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/mcp-server-ui.test.tsx) `adding an MCP server` | +| Extensions kept as files: `lens ext` lists, pulls, checks and pushes manifests, here or over the API, losing neither side's changes | [test_cli_ext.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_cli_ext.py) `test_pull_edit_and_push_on_this_install`, `test_over_the_api_with_a_token` | | Skills and plugins from marketplaces in the open plugin format, a repository or an archive | [test_marketplaces.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_marketplaces.py) `test_a_marketplace_an_admin_adds`, `test_imported_from_a_link_or_a_file` | ## Bring in everything diff --git a/fastapi_backend/app/cli.py b/fastapi_backend/app/cli.py index ef5ad9456..0ba034d58 100644 --- a/fastapi_backend/app/cli.py +++ b/fastapi_backend/app/cli.py @@ -117,6 +117,7 @@ def _main_base(argv=None): p = sub.add_parser("migrations", help="list the database upgrades: run, pending or failed (docs/database.md)") p.add_argument("--run", action="store_true", help="run the pending ones now (Lens also runs them when it starts)") sub.add_parser("backup", help="back the database up into /backups (docs/database.md, Backups)") + sub.add_parser("ext", help="extensions' manifests as files: list, pull, push, check (lens ext -h)", add_help=False) sub.add_parser("reindex", help="rebuild the search index (after changing search.engine or search.stemming)") p = sub.add_parser("encrypt", help="encrypt the files Lens keeps, and keep encrypting new ones (docs/encryption.md)") p.add_argument("--off", action="store_true", help="decrypt them again and stop encrypting new ones") @@ -461,4 +462,8 @@ def main(argv=None): break if i < len(argv) and argv[i] in PLATFORM_CMDS: return platform_main(argv[i:], config) + if i < len(argv) and argv[i] == "ext": # extensions' manifests as files (cli_ext.py) + from . import cli_ext + + return cli_ext.main(argv[i + 1 :], config) return _main_base(argv) diff --git a/fastapi_backend/app/cli_ext.py b/fastapi_backend/app/cli_ext.py new file mode 100644 index 000000000..95b9b6c4b --- /dev/null +++ b/fastapi_backend/app/cli_ext.py @@ -0,0 +1,420 @@ +"""`lens ext`: extensions' manifests as files (docs/assistant.md, Manifests as files). + +`list` the extensions, `pull` their manifests into a folder, `push` a folder's changes back as new versions (and the new +ones as extensions), and `check` manifests without saving them. It talks to a Lens server over its API with an API +token (`--url` or LENS_URL, `--token` or LENS_TOKEN), or, with no address, to this install's database as one of its +accounts (`--as`, by default the first admin), as the other `lens` commands do. + +A pulled file carries `version: N`, the version it was pulled at (the check ignores it). That is how a push knows the +extension changed in Lens since, and a pull knows the file changed here since: neither overwrites the other's changes +unless told to (`--force`). +""" + +from __future__ import annotations + +import argparse +import json +import os +import pathlib +import re +import sys +import urllib.error +import urllib.parse +import urllib.request + +from .domain import auth, extensions, store + +SUFFIXES = (".md", ".yaml", ".yml", ".json") +HEAD_KEYS = ("name", "kind", "title", "description") +SHARING = ("visibility", "namespaces", "version") # in a file, not in the manifest Lens gives back +FRONTMATTER = re.compile(r"^(---[ \t]*\n)(.*?\n)(---[ \t]*(?:\n.*)?)$", re.S) +VERSION_LINE = re.compile(r"^version:.*(?:\n|$)", re.M) + + +class Api: + """A Lens server, through /api/v1/extensions with an API token.""" + + def __init__(self, url, token): + self.base = url.rstrip("/").removesuffix("/api/v1") + "/api/v1/extensions" + self.token = token + + def _call(self, method, path="", body=None, **query): + url = self.base + path + ("?" + urllib.parse.urlencode(query) if query else "") + req = urllib.request.Request( + url, + method=method, + data=None if body is None else json.dumps(body).encode(), + headers={"Authorization": f"Bearer {self.token}", "Content-Type": "application/json", "Accept": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=120) as r: + return json.loads(r.read() or b"null") + except urllib.error.HTTPError as e: + try: + detail = json.loads(e.read() or b"{}").get("detail") + except ValueError: + detail = None + if isinstance(detail, list): # the request's shape (422) + detail = "; ".join(str(d.get("msg", d)) if isinstance(d, dict) else str(d) for d in detail) + if e.code == 401: + raise SystemExit(f"{self.base}: the token was refused (make one in Settings › API tokens)") from None + if e.code in (400, 403, 404, 422): + raise ValueError(detail or e.reason) from None + raise SystemExit(f"{url}: {e.code} {detail or e.reason}") from None + except urllib.error.URLError as e: + raise SystemExit(f"can't reach {self.base}: {e.reason}") from None + + def list(self, kind=None): + return self._call("GET", **({"kind": kind} if kind else {})) + + def get(self, eid, version=None): + return self._call("GET", f"/{eid}", **({"version": version} if version else {})) + + def check(self, text): + return self._call("POST", "/check", {"text": text})["manifest"] + + def create(self, text): + return self._call("POST", "", {"text": text})["id"] + + def save(self, eid, text, notes): + return self._call("POST", f"/{eid}/versions", {"text": text, "notes": notes})["version"] + + def share(self, eid, visibility, namespaces): + self._call("PATCH", f"/{eid}", {"visibility": visibility, "namespaces": namespaces}) + + +class Local: + """This install's database, as one of its accounts: the one given, or the first admin. Saving is audited as the + API audits it.""" + + def __init__(self, db, email=None, need_account=True): + self.db = db + if email: + acct = auth.find_account(db, email) + if not acct: + raise SystemExit(f"no account for {email}") + else: + rows = db.rows("SELECT record::id(id) AS id, email, admin, disabled, removed_at FROM account WHERE admin = true ORDER BY id") + acct = next((r for r in rows if not r.get("disabled") and not r.get("removed_at")), None) + if acct: + self.me = extensions.who(acct["id"], acct["email"], acct.get("admin"), auth.roles(db, acct)) + elif need_account: + raise SystemExit("no admin account yet: sign up in the web app first, or give --url and --token for a Lens server") + else: # checking saves nothing, so a new install checks as an admin would + self.me = extensions.who(None, "lens ext", True, {}) + + def _audit(self, action, eid, detail): + auth.audit(self.db, self.me, action, f"extension:{eid}", {**detail, "via": "cli"}) + + def list(self, kind=None): + return extensions.visible(self.db, self.me, kind) + + def get(self, eid, version=None): + try: + return extensions.visible_one(self.db, self.me, eid, version) + except KeyError: + raise ValueError(f"no extension {eid}" + (f" v{version}" if version else "")) from None + + def check(self, text): + return extensions.check_manifest(extensions.parse_manifest(text), self.me, self.db) + + def create(self, text): + eid = extensions.create(self.db, self.me, extensions.parse_manifest(text)) + self._audit("extension.create", eid, {"origin": "code"}) + return eid + + def save(self, eid, text, notes): + try: + n = extensions.save_version(self.db, self.me, eid, extensions.parse_manifest(text), notes) + except (PermissionError, KeyError) as e: + raise ValueError(str(e)) from None + self._audit("extension.save", eid, {"version": n, "origin": "code"}) + return n + + def share(self, eid, visibility, namespaces): + try: + extensions.update(self.db, self.me, eid, visibility=visibility, namespaces=namespaces) + except (PermissionError, KeyError) as e: + raise ValueError(str(e)) from None + self._audit("extension.update", eid, {"visibility": visibility, "namespaces": namespaces}) + + +# ---------- manifests as files ---------- +def _read(text): + """A manifest's settings, flat (the spec's keys next to name and kind), read without checking.""" + m = extensions.parse_manifest(text) + if isinstance(m.get("spec"), dict): + m = {**{k: m[k] for k in m if k != "spec"}, **m["spec"]} + return m + + +def _core(text): + m = _read(text) + for k in SHARING: + m.pop(k, None) + return m + + +def _version(m): + try: + return int(m["version"]) if m.get("version") is not None else None + except (TypeError, ValueError): + return None + + +def stamped(text, version, add=True): + """The manifest with `version: N` in its head (its frontmatter, or the whole YAML): replacing the line that's + there, or, with `add`, adding one. None when it has none and `add` is off, or is JSON.""" + fm = FRONTMATTER.match(text) + before, head, after = (fm.group(1), fm.group(2), fm.group(3)) if fm else ("", text, "") + line = f"version: {version}\n" + head, n = VERSION_LINE.subn(line, head, count=1) + if not n: + if not add or head.lstrip().startswith(("{", "[")): + return None + head += ("" if not head or head.endswith("\n") else "\n") + line + return before + head + after + + +def manifest_files(paths): + """The files named, and the manifests in the folders named (not their subfolders): .md files with frontmatter + (a README is left alone), .yaml, .yml and .json.""" + out = [] + for p in map(pathlib.Path, paths): + if p.is_dir(): + for f in sorted(p.iterdir()): + if not f.is_file() or f.name.startswith(".") or f.suffix not in SUFFIXES: + continue + if f.suffix == ".md" and not f.read_text(encoding="utf-8").lstrip("").startswith("---"): + continue + out.append(f) + elif p.is_file(): + out.append(p) + else: + raise SystemExit(f"no such file or folder: {p}") + return out + + +def _held(files): + """{name: file} for the manifests that read, and [(file, problem)] for the others (unreadable, or a name two files + have).""" + have: dict[str, pathlib.Path] = {} + bad: list[tuple[pathlib.Path, str]] = [] + for f in files: + try: + name = str(_read(f.read_text(encoding="utf-8")).get("name") or "") + except ValueError as e: + bad.append((f, str(e))) + continue + if not name: + bad.append((f, "a manifest starts with its name, kind and description")) + elif name in have: + bad.append((f, f"{name} is in {have[name].name} too")) + else: + have[name] = f + return have, bad + + +# ---------- the commands ---------- +def cmd_list(be, a): + items = be.list(a.kind) + if a.json: + keep = ("id", "name", "kind", "title", "description", "current", "enabled", "visibility", "namespaces", "owner_email", "editable") + print(json.dumps([{k: x.get(k) for k in keep} for x in items], ensure_ascii=False, indent=1)) + return 0 + for x in items: + shared = x.get("visibility") + (f" ({', '.join(x['namespaces'])})" if x.get("namespaces") else "") + state = "on" if x.get("enabled") else "off" + mine = "" if x.get("editable") else " read only" + print(f"{x['id']:>5} {x['name']:<28} {x['kind']:<7} v{x['current']:<4} {state:<4} {shared:<20} {x.get('owner_email') or ''}{mine}") + if not items: + print("no extensions yet: write a manifest and run lens ext push") + return 0 + + +def _pull_one(be, x, folder, have, force): + cur = be.get(x["id"]) + text = stamped(cur["manifest"], cur["current"]) + suffix = ".md" if FRONTMATTER.match(text) else ".yaml" + old = have.get(x["name"]) + if old is not None: + mine = old.read_text(encoding="utf-8") + if mine == text: + return f"unchanged {old.name} (v{cur['current']})" + if not force and _core(mine) != _core(cur["manifest"]): + base = _version(_read(mine)) + try: + clean = base is not None and _core(mine) == _core(be.get(x["id"], base)["manifest"]) + except ValueError: + clean = False + if not clean: + raise ValueError(f"{old.name} has changes that aren't in Lens: push them, or pull --force to take Lens's") + keep = old is not None and (old.suffix == suffix or (suffix == ".yaml" and old.suffix == ".yml")) + path = old if keep else folder / f"{x['name']}{suffix}" + if path != old and path.exists() and not force: + raise ValueError(f"{path.name} is there already and holds something else: move it, or pull --force") + path.write_text(text, encoding="utf-8") + if old is not None and path != old: + old.unlink() + return f"pulled {x['name']} into {path.name}, instead of {old.name} (v{cur['current']})" + return f"pulled {path.name} ({x['kind']}, v{cur['current']})" + + +def cmd_pull(be, a): + folder = pathlib.Path(a.dir) + folder.mkdir(parents=True, exist_ok=True) + items = be.list(a.kind) + if a.names: + by = {x["name"]: x for x in items} | {str(x["id"]): x for x in items} + missing = [n for n in a.names if n not in by] + if missing: + raise SystemExit(f"no extension called {', '.join(missing)} that you can see (lens ext list)") + items = [by[n] for n in dict.fromkeys(a.names)] + elif not a.all: + items = [x for x in items if x.get("editable")] # yours (admins: all); --all adds the ones shared with you + have, _ = _held(manifest_files([folder])) + failed = 0 + for x in items: + try: + print(" ", _pull_one(be, x, folder, have, a.force)) + except ValueError as e: + failed += 1 + print(f" {x['name']}: {e}", file=sys.stderr) + if not items: + print("nothing to pull" + ("" if a.all else ": none of the extensions are yours (--all pulls the ones shared with you)")) + return 1 if failed else 0 + + +def _same(be, mine, theirs): + """Whether a file says what Lens has: read as they are, or, when they're written differently, as the check + tidies them (defaults filled in), so a hand-written file doesn't make a new version on every push.""" + if _core(mine) == _core(theirs): + return True + try: + a, b = be.check(mine), be.check(theirs) + except ValueError: + return False + for m in (a, b): + for k in SHARING: + m.pop(k, None) + return a == b + + +def _push_one(be, path, x, notes, force): + text = path.read_text(encoding="utf-8") + m = _read(text) + said = [] + if x is None: + eid, n = be.create(text), 1 + said.append(f"created {m['name']} ({m.get('kind')}, id {eid}) from {path.name}") + cur = None + else: + if not x.get("editable"): + raise ValueError(f"{x['name']} is {x.get('owner_email') or 'someone else'}'s: only its owner or an admin changes it") + if m.get("kind") != x["kind"]: + raise ValueError(f"{x['name']} is a {x['kind']} in Lens; a manifest can't change its kind (give it another name)") + eid, cur = x["id"], be.get(x["id"]) + n = cur["current"] + if _same(be, text, cur["manifest"]): + said.append(f"unchanged {path.name} (v{n})") + else: + base = _version(m) + if base is not None and base != n and not force: + raise ValueError(f"{x['name']} changed in Lens since v{base} (now v{n}): pull it first, or push --force") + n = be.save(eid, text, notes or f"lens ext push {path.name}") + said.append(f"saved {x['name']} v{n} from {path.name}") + if cur is not None and ("visibility" in m or "namespaces" in m): + vis = m.get("visibility") or cur["visibility"] + ns = sorted(m["namespaces"] if m.get("namespaces") is not None else cur.get("namespaces") or []) + was = sorted(cur.get("namespaces") or []) if cur["visibility"] == "namespace" else [] + if (vis, ns if vis == "namespace" else []) != (cur["visibility"], was): + be.share(eid, vis, ns) + said.append(f"shared {x['name']}: {vis}" + (f" ({', '.join(ns)})" if vis == "namespace" else "")) + again = stamped(text, n, add=False) # a file that says which version it is says the new one + if again is not None and again != text: + path.write_text(again, encoding="utf-8") + return "; ".join(said) + + +def cmd_push(be, a): + files = manifest_files(a.paths or ["."]) + have, bad = _held(files) + by = {x["name"]: x for x in be.list()} + failed = len(bad) + for f, why in bad: + print(f" {f.name}: {why}", file=sys.stderr) + for name, f in have.items(): + try: + print(" ", _push_one(be, f, by.get(name), a.notes, a.force)) + except ValueError as e: + failed += 1 + print(f" {f.name}: {e}", file=sys.stderr) + if not files: + print("no manifests here: .md files with frontmatter, .yaml, .yml or .json") + return 1 if failed else 0 + + +def cmd_check(be, a): + files = manifest_files(a.paths or ["."]) + _, bad = _held(files) + dup = {f for f, _ in bad} + failed = 0 + for f in files: + try: + if f in dup: + raise ValueError(next(why for g, why in bad if g == f)) + m = be.check(f.read_text(encoding="utf-8")) + print(f" ok {f.name}: {m['name']} ({m['kind']})") + except ValueError as e: + failed += 1 + print(f" {f.name}: {e}", file=sys.stderr) + if not files: + print("no manifests here: .md files with frontmatter, .yaml, .yml or .json") + return 1 if failed else 0 + + +COMMANDS = {"list": cmd_list, "pull": cmd_pull, "push": cmd_push, "check": cmd_check} + + +def parser(): + ap = argparse.ArgumentParser( + prog="lens ext", + description="Extensions' manifests as files: list them, pull them into a folder, push changes back, check them.", + ) + ap.add_argument( + "--url", default=os.environ.get("LENS_URL"), help="a Lens server's address (default: $LENS_URL; none: this install's database)" + ) + ap.add_argument( + "--token", default=os.environ.get("LENS_TOKEN"), help="an API token for it, write scoped to push (default: $LENS_TOKEN)" + ) + ap.add_argument("--as", dest="email", help="without --url: the account to act as (default: the first admin)") + sub = ap.add_subparsers(dest="action", required=True) + p = sub.add_parser("list", help="the extensions you can see") + p.add_argument("--kind", choices=extensions.KINDS) + p.add_argument("--json", action="store_true", help="as JSON, for scripts and agents") + p = sub.add_parser("pull", help="write manifests into a folder, each at the version Lens has") + p.add_argument("names", nargs="*", help="extensions by name or id (default: yours; admins: all)") + p.add_argument("--dir", default=".", help="the folder (default: this one)") + p.add_argument("--kind", choices=extensions.KINDS) + p.add_argument("--all", action="store_true", help="the ones shared with you too") + p.add_argument("--force", action="store_true", help="overwrite files that have changes Lens doesn't") + p = sub.add_parser("push", help="save changed manifests as new versions, and new ones as extensions") + p.add_argument("paths", nargs="*", help="manifest files or folders (default: this folder)") + p.add_argument("--notes", help="what changed, kept with each new version") + p.add_argument("--force", action="store_true", help="save over changes made in Lens since the file was pulled") + p = sub.add_parser("check", help="read and check manifests without saving them") + p.add_argument("paths", nargs="*", help="manifest files or folders (default: this folder)") + return ap + + +def main(argv, config=None): + a = parser().parse_args(argv) + if a.url: + if not a.token: + raise SystemExit("give an API token with --token or LENS_TOKEN (make one in Settings › API tokens)") + return COMMANDS[a.action](Api(a.url, a.token), a) + db = store.connect(store.load_config(config)) + try: + return COMMANDS[a.action](Local(db, a.email, need_account=a.action != "check"), a) + finally: + db.close() diff --git a/fastapi_backend/tests/api/test_cli_ext.py b/fastapi_backend/tests/api/test_cli_ext.py new file mode 100644 index 000000000..c8860eec5 --- /dev/null +++ b/fastapi_backend/tests/api/test_cli_ext.py @@ -0,0 +1,161 @@ +"""`lens ext`: extensions' manifests as files, pulled from Lens and pushed back, against this install's database or a +Lens server over its API with a token; neither side's changes are overwritten without --force.""" + +from __future__ import annotations + +import socket +import threading +import time + +import pytest + +from app import cli, cli_ext +from app.domain import auth, extensions +from tests.helpers import make_user + +SKILL = """--- +name: meeting_recap +kind: skill +description: Recaps a meeting +when: someone asks for a recap of a meeting +--- + +Find the meeting, read it, and answer with three bullets: decisions, owners, dates. +""" + +# written by hand: no effect and no parameter kinds, which the check fills in +TOOL = """name: shout +kind: tool +description: Says it louder. +params: + - name: text +run: + type: prompt + prompt: "Say this in capitals: {{text}}" +""" + + +def versions(db, name): + return [x["current"] for x in extensions.visible(db, extensions.who(0, "x", True, {})) if x["name"] == name] + + +def test_pull_edit_and_push_on_this_install(tmp_path, monkeypatch, capsys): + import io + + monkeypatch.delenv("SURREAL_URL", raising=False) + (tmp_path / "archive.yaml").write_text("data_dir: ./data\nnamespaces:\n pods:\n paths: []\n") + + def run(*args): + return cli.main(["--config", str(tmp_path / "archive.yaml"), "ext", *args]) + + mine, theirs = tmp_path / "mine", tmp_path / "theirs" + mine.mkdir() + (mine / "meeting_recap.md").write_text(SKILL) + (mine / "shout.yaml").write_text(TOOL) + (mine / "README.md").write_text("# My extensions\n") # not a manifest: left alone + with pytest.raises(SystemExit, match="no admin account yet"): + run("push", str(mine)) + assert run("check", str(mine)) == 0 # checking needs no account + assert "ok meeting_recap.md: meeting_recap (skill)" in capsys.readouterr().out + + monkeypatch.setattr("sys.stdin", io.StringIO("cli password 123\n")) + cli.main(["--config", str(tmp_path / "archive.yaml"), "users", "add", "root@x.io", "--admin", "--password-stdin"]) + assert run("push", str(mine)) == 0 + out = capsys.readouterr().out + assert "created meeting_recap (skill, id" in out and "created shout (tool, id" in out + assert run("push", str(mine)) == 0 # nothing changed, though the tool file leaves out what the check fills in + assert capsys.readouterr().out.count("unchanged") == 2 + assert run("list") == 0 + assert "meeting_recap" in capsys.readouterr().out + assert run("list", "--json") == 0 + assert '"name": "shout"' in capsys.readouterr().out + + # pulled at a version, changed here, pushed: a new version, and the file says which + assert run("pull", "--dir", str(theirs)) == 0 + pulled = (theirs / "meeting_recap.md").read_text() + assert pulled.startswith("---\nname: meeting_recap\n") and "version: 1\n---\n" in pulled + assert "version: 1\n---\n\nSay this in capitals: {{text}}" in (theirs / "shout.md").read_text() # a prompt tool + assert run("pull", "--dir", str(theirs)) == 0 + assert "unchanged meeting_recap.md (v1)" in capsys.readouterr().out + (theirs / "meeting_recap.md").write_text(pulled.replace("three bullets", "four bullets")) + assert run("push", str(theirs), "--notes", "four now") == 0 + assert "saved meeting_recap v2 from meeting_recap.md" in capsys.readouterr().out + assert "version: 2\n" in (theirs / "meeting_recap.md").read_text() + + # the first folder was never pulled: its files say no version, so one that differs from Lens is kept unless --force + assert run("pull", "--dir", str(mine)) == 1 + assert "meeting_recap: meeting_recap.md has changes that aren't in Lens" in capsys.readouterr().err + assert run("pull", "--dir", str(mine), "--force") == 0 + assert "pulled shout into shout.md, instead of shout.yaml (v1)" in capsys.readouterr().out + assert "four bullets" in (mine / "meeting_recap.md").read_text() and not (mine / "shout.yaml").exists() + + # changed in Lens since this folder pulled it: push refuses, pull takes it; a stale edit stays + (theirs / "meeting_recap.md").write_text((theirs / "meeting_recap.md").read_text().replace("four", "five")) + assert run("push", str(theirs)) == 0 # v3 + capsys.readouterr() + (mine / "meeting_recap.md").write_text((mine / "meeting_recap.md").read_text().replace("four", "six")) + assert run("push", str(mine)) == 1 + assert "changed in Lens since v2 (now v3): pull it first, or push --force" in capsys.readouterr().err + assert run("pull", "--dir", str(mine)) == 1 # it has changes of its own too + (mine / "meeting_recap.md").write_text((mine / "meeting_recap.md").read_text().replace("six", "four")) + assert run("pull", "--dir", str(mine)) == 0 # back as it was pulled: safe to take Lens's + assert "five bullets" in (mine / "meeting_recap.md").read_text() + + # a manifest that doesn't check, and two files with one name, are named and change nothing + (mine / "broken.yaml").write_text(TOOL.replace("shout", "broken").replace("{{text}}", "{{words}}")) + (mine / "again.md").write_text(SKILL) + assert run("check", str(mine)) == 1 + err = capsys.readouterr().err + assert "broken.yaml: the tool uses {{words}}, which aren't its parameters" in err + assert "meeting_recap is in again.md too" in err or "meeting_recap is in meeting_recap.md too" in err + + +@pytest.fixture +def server(app): + """The app on a real port, as a Lens server the command line reaches over HTTP.""" + import uvicorn + + sock = socket.socket() + sock.bind(("127.0.0.1", 0)) + srv = uvicorn.Server(uvicorn.Config(app, log_level="warning")) + t = threading.Thread(target=srv.run, kwargs={"sockets": [sock]}, daemon=True) + t.start() + while not srv.started: + time.sleep(0.02) + yield f"http://127.0.0.1:{sock.getsockname()[1]}" + srv.should_exit = True + t.join(5) + + +def test_over_the_api_with_a_token(server, db, tmp_path, capsys): + ed = make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"}) + vi = make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + tok_ed, tok_vi = (auth.create_token(db, uid, "cli", "write")[1] for uid in (ed, vi)) + (tmp_path / "meeting_recap.md").write_text(SKILL.replace("when:", "visibility: namespace\nnamespaces: [pods]\nwhen:")) + + def run(token, *args): + return cli_ext.main(["--url", server + "/", "--token", token, *args]) + + assert run(tok_ed, "check", str(tmp_path)) == 0 + assert run(tok_ed, "push", str(tmp_path)) == 0 + assert "created meeting_recap (skill, id" in capsys.readouterr().out + assert versions(db, "meeting_recap") == [1] + + # the viewer sees it, shared with pods, but can't change it; pulling it takes --all or its name + folder = tmp_path / "vi" + assert run(tok_vi, "pull", "--dir", str(folder)) == 0 + assert "none of the extensions are yours" in capsys.readouterr().out + assert run(tok_vi, "pull", "meeting_recap", "--dir", str(folder)) == 0 + (folder / "meeting_recap.md").write_text((folder / "meeting_recap.md").read_text().replace("three", "two")) + assert run(tok_vi, "push", str(folder)) == 1 + assert "meeting_recap is ed@x.io's: only its owner or an admin changes it" in capsys.readouterr().err + + # sharing written in the file follows it + (tmp_path / "meeting_recap.md").write_text(SKILL.replace("when:", "visibility: private\nwhen:")) + assert run(tok_ed, "push", str(tmp_path)) == 0 + assert "unchanged meeting_recap.md (v1); shared meeting_recap: private" in capsys.readouterr().out + assert run(tok_vi, "list") == 0 + assert "meeting_recap" not in capsys.readouterr().out + + with pytest.raises(SystemExit, match="the token was refused"): + run("la_nonsense", "list") From 3816274fa9ab161249f97a6d1dca0c2ff74af9ec Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 06:10:30 +0000 Subject: [PATCH 2/4] fix(cli): lens ext pushes neither of two files with one name Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01FbmJ86QaV2YwusjgNowpf8 --- fastapi_backend/app/cli_ext.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/fastapi_backend/app/cli_ext.py b/fastapi_backend/app/cli_ext.py index 95b9b6c4b..f552b932e 100644 --- a/fastapi_backend/app/cli_ext.py +++ b/fastapi_backend/app/cli_ext.py @@ -200,6 +200,7 @@ def _held(files): have).""" have: dict[str, pathlib.Path] = {} bad: list[tuple[pathlib.Path, str]] = [] + twice: dict[str, pathlib.Path] = {} for f in files: try: name = str(_read(f.read_text(encoding="utf-8")).get("name") or "") @@ -208,8 +209,11 @@ def _held(files): continue if not name: bad.append((f, "a manifest starts with its name, kind and description")) - elif name in have: - bad.append((f, f"{name} is in {have[name].name} too")) + elif name in have or name in twice: + bad.append((f, f"{name} is in {(have.get(name) or twice[name]).name} too")) + if name in have: + twice[name] = have.pop(name) # neither is pushed: which one is meant is the person's call + bad.append((twice[name], f"{name} is in {f.name} too")) else: have[name] = f return have, bad From bfc7e49a8686ba79d5acd810e47c803bb95a1b20 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 07:30:26 +0000 Subject: [PATCH 3/4] docs(readme): list answering questions about Lens itself as built, now #400 is released Main's requirements check fails since v0.15.0 released #400 while the README still listed it In progress. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01FbmJ86QaV2YwusjgNowpf8 --- README.md | 5 ++--- docs/requirements.md | 1 + 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b7339841c..6df6ec7a7 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p cost), opened from its header. It reads answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine (opt-in), any message in one click. Long conversations are summarised before they fill the model's context, and `/clear` and `/compact` start afresh or summarise now. Conversations title themselves after their first answer, unless - you name them. Each namespace can have its own assistant with + you name them. It answers questions about Lens itself from its docs and this server's settings, for people and, over + MCP, for agents. Each namespace can have its own assistant with its own name, instructions and memory, answering in Telegram and Slack (built in, with voice notes and files), in Discord, Matrix, WhatsApp and other chat rooms through Matterbridge, and by email at an address of its own that you email or forward things to. ([Chat](docs/assistant.md), [web search](docs/assistant.md#web-search-and-reading-pages), @@ -235,8 +236,6 @@ still loading. ([Components](docs/components.md), These are open pull requests, not yet part of Lens: -- **The assistant** answering questions about Lens itself, from its docs and this server - ([#400](https://github.com/adeelahmad/lense/pull/400)). - **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)). - **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)). - **Web clipper:** signing in, or using your API token, not only an import webhook's diff --git a/docs/requirements.md b/docs/requirements.md index 88d46b784..502274ad8 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -74,6 +74,7 @@ request, to another check that proves the same thing; loosening a requirement or | Requirement | Checks | |---|---| | A chat assistant on every page | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_asking_from_a_page`; [page-chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/page-chat.test.tsx); [ask-place.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ask-place.test.tsx) | +| It answers questions about Lens itself from its docs and this server, for people and agents | [test_lens_help.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_lens_help.py) `test_the_assistant_answers_about_lens_itself`, `test_agents_ask_lens_over_mcp`, `test_admins_see_where_each_setting_is` | | By voice | [assistant-mode.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/assistant-mode.test.tsx) `starts a voice chat from the mic`; [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_what_is_said_comes_back_as_text` | | It searches by words, and by meaning once search by meaning is set up | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_semantic.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_semantic.py) `test_chat_retrieval_by_meaning`, `test_off_or_unconfigured` | | Answers cite their sources, and only what the person may read | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_chat_cites_only_what_you_can_read`; [chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat.test.tsx) `splits [n], [n, m] and [n][m]` | From 131c54b7657efb39ff9da742d2cea7126f855615 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 11:05:07 +0000 Subject: [PATCH 4/4] Merge main; move the changelog entry to changelog.d Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01FbmJ86QaV2YwusjgNowpf8 --- CHANGELOG.md | 7 ------- changelog.d/lens-ext.md | 6 ++++++ 2 files changed, 6 insertions(+), 7 deletions(-) create mode 100644 changelog.d/lens-ext.md diff --git a/CHANGELOG.md b/CHANGELOG.md index c2bb51364..6ff27e90d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,13 +4,6 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version ## Unreleased -- **`lens ext`: extensions as files.** `lens ext pull` writes your extensions' manifests into a folder (skills and - prompt tools as Markdown, the rest as YAML), `lens ext push` saves the changed ones as new versions and the new ones - as extensions, `lens ext check` checks them without saving, and `lens ext list` lists them (`--json` for agents). It - works on this install's database, or on any Lens server with `--url` and an API token. A pulled file says its - version, so a push won't save over changes made in Lens since, and a pull won't overwrite changes made in the file, - without `--force` ([Manifests as files](docs/assistant.md#manifests-as-files-lens-ext)). - ## 0.16.0 October 18, 2026 {id="0.16.0"} ### Features diff --git a/changelog.d/lens-ext.md b/changelog.d/lens-ext.md new file mode 100644 index 000000000..50105026e --- /dev/null +++ b/changelog.d/lens-ext.md @@ -0,0 +1,6 @@ +- **`lens ext`: extensions as files.** `lens ext pull` writes your extensions' manifests into a folder (skills and + prompt tools as Markdown, the rest as YAML), `lens ext push` saves the changed ones as new versions and the new ones + as extensions, `lens ext check` checks them without saving, and `lens ext list` lists them (`--json` for agents). It + works on this install's database, or on any Lens server with `--url` and an API token. A pulled file says its + version, so a push won't save over changes made in Lens since, and a pull won't overwrite changes made in the file, + without `--force` ([Manifests as files](docs/assistant.md#manifests-as-files-lens-ext)).