From 18579ffb2ea43b7a915c401355627b2f80567257 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Sun, 18 Oct 2026 01:07:43 +0000 Subject: [PATCH] feat(secrets): read secret settings from Vaultwarden by reference A secret field in Settings can hold a reference such as vaultwarden://OpenAI/password instead of the secret. Lens reads the value from the vault when it builds its settings, keeps it in memory for secrets.cache_minutes and reads it again in the background; only the reference is stored, and it's checked against the vault before it's saved. Settings, the API, MCP and the assistant show references and whether they could be read, never a value. The built-in sealed store stays the default. Vaultwarden is read with python-vaultwarden (Apache-2.0), fetched as a component once an address is set; Lens reads one folder of its own account. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_019okSDRfFHV74BTpMzinTFg --- CHANGELOG.md | 10 + DEVLOG.md | 16 + README.md | 6 +- docs/assistant.md | 5 +- docs/components.md | 1 + docs/configuration.md | 3 +- docs/mcp.md | 6 +- docs/requirements.md | 1 + docs/secrets.md | 117 +++++ fastapi_backend/app/api/mcp_tools.py | 80 +++ fastapi_backend/app/api/v1/router.py | 2 + fastapi_backend/app/api/v1/routes/secrets.py | 54 ++ fastapi_backend/app/domain/components.py | 21 + fastapi_backend/app/domain/ops_tools.py | 64 ++- .../app/domain/secret_providers.py | 318 +++++++++++ fastapi_backend/app/domain/settings.py | 114 +++- fastapi_backend/app/domain/store.py | 13 + .../app/domain/vaultwarden_secrets.py | 335 ++++++++++++ fastapi_backend/app/schemas/secrets.py | 61 +++ fastapi_backend/pyproject.toml | 4 + fastapi_backend/tests/api/test_mcp.py | 1 + fastapi_backend/tests/api/test_secrets.py | 266 ++++++++++ fastapi_backend/tests/fake_vaultwarden.py | 177 +++++++ fastapi_backend/uv.lock | 64 ++- mkdocs.yml | 1 + nextjs-frontend/app/openapi-client/index.ts | 20 + nextjs-frontend/app/openapi-client/sdk.gen.ts | 58 +++ .../app/openapi-client/types.gen.ts | 302 +++++++++++ nextjs-frontend/openapi.json | 492 ++++++++++++++++++ threat-model.json | 8 + threat-model.md | 6 +- threat-model.yaml | 8 + 32 files changed, 2612 insertions(+), 22 deletions(-) create mode 100644 docs/secrets.md create mode 100644 fastapi_backend/app/api/v1/routes/secrets.py create mode 100644 fastapi_backend/app/domain/secret_providers.py create mode 100644 fastapi_backend/app/domain/vaultwarden_secrets.py create mode 100644 fastapi_backend/app/schemas/secrets.py create mode 100644 fastapi_backend/tests/api/test_secrets.py create mode 100644 fastapi_backend/tests/fake_vaultwarden.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a22956cb..2c1e37349 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,16 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version a time (for the same sender, say), when one came a number of times, when one didn't come in time after another (no reply within two days), or once for a batch collected over a time. What it waits on outlasts a restart, and switching the automation off drops it ([Joins](docs/automations.md#joins)). +- **Secrets from Vaultwarden.** A secret field in Settings (a model's API key, a chat app's token, a mail password) + can refer to an item in a Vaultwarden or Bitwarden vault, like `vaultwarden://OpenAI/password`, instead of holding + the secret. Lens reads the value, keeps it in memory for `secrets.cache_minutes` (10 by default) and reads it again + in the background, so a key changed in the vault reaches Lens by itself. Only the reference is stored, and it's + checked against the vault before it's saved. Lens signs in as an account of its own and reads one folder of it + (`secrets.vaultwarden_folder`, "Lens"). Settings, the API (`GET /api/v1/secrets`, `GET /api/v1/secrets/items`, + `POST /api/v1/secrets/test`), the MCP tools `list_secrets`, `list_vault_items` and `set_secret_reference`, and the + assistant's `list_secrets` and `vault_items` show references and whether they could be read, never a value. The + client (python-vaultwarden, Apache-2.0) is fetched once a Vaultwarden address is set. Secrets saved in Lens stay + the default ([Secrets](docs/secrets.md)). - **Specs from transcripts.** Turn a voice session or a meeting into a technical spec, saved as a note linked to it: the model writes the spec, then lists every named decision, every "no" and "not that, this", every constraint and every gap the speaker left to decide later, each with its quote and a link to the moment it was said; a quote the diff --git a/DEVLOG.md b/DEVLOG.md index 718447286..2c79a92ed 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -2,6 +2,22 @@ Plans and progress for work in flight. Newest first. +## 2026-10-18 · Secrets from Vaultwarden + +Adeel: a secrets manager, Vaultwarden first, with others to come. A secret field in Settings can hold a reference +(`vaultwarden://OpenAI/password`) instead of the secret; `secret_providers.py` reads it when the settings are built, +keeps it in memory for `secrets.cache_minutes` and reads it again in the background, and anything that shows settings +shows the reference, never the value. The built-in sealed store stays the default. Vaultwarden is read with +python-vaultwarden (Apache-2.0), fetched as a component once an address is set: the Bitwarden CLI is GPL and needs +Node, and Bitwarden's SDK isn't permissively licensed and reads only its Secrets Manager product. Lens reads one +folder of its own vault account. Tested against a fake Vaultwarden that encrypts with the library's own crypto. + +Next: Settings › Secrets in the web app, with a picker for the vault's items beside every secret field. + +Refine later: references in sources' credentials, MCP servers' environments and sign-in providers; checking a value +read from the vault the way a typed one is checked; a custom CA for a self-hosted vault; an environment variable for +the master password; more providers (1Password, HashiCorp Vault, Infisical). + ## 2026-10-17 · Read any chat message aloud Adeel: play any message in the chat through the TTS model. A speaker button on every question and answer reads it by diff --git a/README.md b/README.md index 04f55891d..f3052f296 100644 --- a/README.md +++ b/README.md @@ -199,7 +199,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p up in Settings, with certificates from Let's Encrypt or ZeroSSL that renew on their own, answering only for its own domains and, if you choose, only to devices holding a client certificate; or reach it from anywhere through a Cloudflare tunnel with nothing opened on your router. Either way, the S3 and WebDAV shares and the MQTT hub answer at - that same address once you turn them on. ([Encryption](docs/encryption.md), [access](docs/access.md), + that same address once you turn them on. Keep API keys and tokens in a Vaultwarden or Bitwarden vault: a secret + field in Settings can refer to an item there, and Lens reads it without anyone, agents included, seeing it. + ([Encryption](docs/encryption.md), [secrets](docs/secrets.md), [access](docs/access.md), [prompt injection](docs/assistant.md#third-party-content-and-prompt-injection), [HTTPS](docs/https.md), [remote access](docs/remote-access.md); [checks](docs/requirements.md#stay-private-and-safe)) - **Use open standards.** RDF and Dublin Core for every record, OAuth for apps and OpenID Connect sign-in, MCP, MQTT, @@ -245,6 +247,8 @@ Next, and not in a pull request yet; each links to where it is tracked: activity ledger ([speech providers](docs/speech-providers.md)). - **Storage:** uploads and assets' media in the file store, and a storage key to download ([storage](docs/storage.md)). +- **Secrets:** Settings › Secrets in the web app, with a picker for the vault's items, and references in sources' + credentials and MCP servers ([secrets](docs/secrets.md)). - **Logging:** sending straight to Loki or another HTTP endpoint, and client certificates for TLS ([logging](docs/logging.md)). - **Proof:** each release tested by upgrading an install of the last one, holding data diff --git a/docs/assistant.md b/docs/assistant.md index 34b039f8c..89eedebed 100644 --- a/docs/assistant.md +++ b/docs/assistant.md @@ -822,7 +822,8 @@ switch shows the line to run on the machine instead. Agents use the same switche Admins also get the server tools: `server_status` (what's set up and what's missing), `find_model_servers`, `read_settings` and `change_settings` (the processing and AI sections, not the server's hosts, cookies or tokens), -`create_namespace`, the news feed tools: `news_feeds`, `subscribe_feed` (follow a feed into a namespace in one step, +`create_namespace`, `list_secrets` and `vault_items` (which secret fields refer to a [password manager](secrets.md), +and the items they can refer to, never a value), the news feed tools: `news_feeds`, `subscribe_feed` (follow a feed into a namespace in one step, [News feeds](processing.md#news-feeds-rss-and-atom)) and `unsubscribe_feed`, and the people tools: `list_people`, `invite_person` (an account, optionally with a role in one namespace; Lens emails the sign-in link when mail is set up) and `change_person` (a role in a namespace, or disabling and enabling an account). Removing an account for good stays in the web app and the API. In an ordinary conversation @@ -830,7 +831,7 @@ their changes are approval cards. In a **setup conversation** (`POST /api/v1/chats` with `"kind": "setup"`, or **Finish with the assistant** in the setup wizard once a model is connected) the admin has asked the assistant to set the server up, so it makes the changes itself and says what it changed. Every change is audited with `assistant` in its detail, and all of them stay changeable in Settings. -Turning telemetry on or off, inviting people and changing their access always ask first, and so does every change once the answer has read text from outside +Turning telemetry on or off, changing the Secrets section, inviting people and changing their access always ask first, and so does every change once the answer has read text from outside Lens ([third-party content](#third-party-content-and-prompt-injection)). ### Routine choices diff --git a/docs/components.md b/docs/components.md index 89aca762e..5ed83dafc 100644 --- a/docs/components.md +++ b/docs/components.md @@ -29,6 +29,7 @@ never fetch a speech or voice model. | yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address | | Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) | | Code sandbox client | the [code sandbox](assistant.md#code-sandbox) is on (`sandbox.enabled`) | OpenShell's Python SDK (Apache-2.0) and gRPC, about 10 MB, to reach its gateway | +| Vaultwarden client | a Vaultwarden address is set (`secrets.vaultwarden_url`) | python-vaultwarden (Apache-2.0, about 10 MB with its crypto libraries), to read [secrets](secrets.md) from a Vaultwarden or Bitwarden vault | | Outlook .msg emails | listed in `components.also` | extract-msg (GPL-3.0, so only when asked for) | - Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win. diff --git a/docs/configuration.md b/docs/configuration.md index d03efd9c7..5ec7fbe0f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -115,7 +115,8 @@ search and search by meaning), reports and graph, video, OCR, faces and objects, [NSFW classification](nsfw.md), workers, sign-in, components (and the edition, full or lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications, -automations, telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys, and IIIF and metadata. The API +automations, telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys, +[secrets](secrets.md) (a password manager that secret fields can refer to), and IIIF and metadata. The API refuses an allowed-host list that leaves out the address you are using. The sections are in groups: AI and search, Speech and voices, Media and documents, Chat and email, Security and diff --git a/docs/mcp.md b/docs/mcp.md index d52ef6998..446d7e797 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -131,6 +131,9 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ | `set_file_storage` | keep new files on this machine or a storage connection (folder, rclone crypt); checked before it's saved. Admins only, with a write-scope token | | `services` | Lens's optional containers ([Optional containers](assistant.md#optional-containers)): the MCP runner, the browser and monitoring, whether each is on and answering, and how the machine's last change went. Admins only | | `set_service` | start or stop one of them (`mcp`, `browser` or `monitoring`): the machine's upgrade agent does it. Admins only, with a write-scope token | +| `list_secrets` | every secret field in Settings: whether it's set, and for one that refers to a password manager ([Secrets](secrets.md)) its reference and whether it could be read; and the password managers there are. Never a value. Admins only | +| `list_vault_items` | the items of a password manager Lens may refer to (those in its folder): name, type, the fields that hold something and the reference to each; never their values. Admins only | +| `set_secret_reference` | point a secret field (`llm.api_key`, `slack.bot_token`, ...) at a password manager's item, like `vaultwarden://OpenAI/password`; checked against the vault before it's saved. Admins only, with a write-scope token | | `picture_descriptions` | what describes the pages and shots of resources ([Descriptions](configuration.md#descriptions)): the LLM provider's model that can see images, or Florence-2 or Moondream on this server, its model, whether it's ready and why not, and the engines there are. Admins only | | `set_picture_descriptions` | choose what describes pictures (`engine` `llm`, `florence-2` or `moondream`), the provider's `vision_model`, Florence-2's size (`florence_model`), which Moondream (`moondream_model`) and `describe_max`. Admins only, with a write-scope token | | `install_settings` | whether other devices on the network reach Lens and its edition (full or lean), whether the machine's upgrade agent can change them, and the install line to run when it can't. Admins only | @@ -158,7 +161,8 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [published feed](feeds.md), `write_spec`, which queues a spec, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and -editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), and `set_monitoring`, `set_service`, +editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item +(admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), and `set_monitoring`, `set_service`, `set_sandbox_gateway` (the [code sandbox](assistant.md#code-sandbox)'s gateway) and `set_install_settings`, which have the machine change how Lens runs (admins). Importing stays in the web app and the API. diff --git a/docs/requirements.md b/docs/requirements.md index 002776f99..cb53e1da1 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -233,6 +233,7 @@ request, to another check that proves the same thing; loosening a requirement or | Passkeys instead of passwords | [test_passkeys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_passkeys.py) `test_first_admin_with_a_passkey_and_no_password`, `test_sign_in_with_a_passkey` | | A passkey at another of Lens's addresses (remote access, HTTPS) added from a signed-in one: Open signed in, then one fingerprint | [test_passkeys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_passkeys.py) `test_taking_your_sign_in_to_another_address`, `test_the_addresses_lens_opens_at`; [passkeys-panel.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/passkeys-panel.test.tsx) `opens an address you have no passkey for signed in, to add one there`; [webauthn.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/webauthn.test.ts) `points a failed sign-in to the address the passkey was made at` | | Encryption at rest, with a key per namespace | [test_keyring.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_keyring.py) `test_each_namespace_gets_its_own_key_kept_only_wrapped`; [test_encryption.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_encryption.py) `test_a_new_archive_encrypts_and_still_plays_seeks_and_processes` | +| Secret fields that refer to a Vaultwarden or Bitwarden vault, read by Lens and never shown to anyone, agents included | [test_secrets.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_secrets.py) `test_a_secret_field_reads_its_value_from_the_vault`, `test_the_vault_is_read_again_and_what_vanishes_stops_being_used`, `test_the_vault_is_set_up_and_checked_in_the_app`, `test_agents_refer_to_secrets_and_never_read_them` | | Roles per namespace: viewer, editor, owner, or custom roles made of permissions | [test_roles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_roles.py) `test_builtin_roles_keep_their_meaning`, `test_custom_roles_allow_exactly_their_permissions`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `person-add` | | Email invitations, and removing someone for good | [test_users.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_users.py) `test_invites_are_emailed_when_mail_is_set_up`, `test_admins_remove_people_for_good` | | An audit log | [test_settings.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_settings.py) `test_settings_are_editable_and_secret`; [test_files.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_files.py) `test_change_a_file_and_the_audit_log` | diff --git a/docs/secrets.md b/docs/secrets.md new file mode 100644 index 000000000..bee66cc73 --- /dev/null +++ b/docs/secrets.md @@ -0,0 +1,117 @@ +# Secrets + +Lens keeps the secrets it signs in with, such as model API keys, chat-app tokens and mail passwords, in Settings, +encrypted with the server's key ([Encryption](encryption.md)). That is the default, and it needs nothing set up. + +A secret can be kept in a password manager instead. The field in Settings then holds a **reference** to the item, +such as `vaultwarden://OpenAI/password`, and Lens reads the value from the manager. A key changed in the manager +reaches Lens without anyone saving a setting. Lens stores only the reference. People and agents see the reference +and whether it could be read, but never the value. + +| | | +|---|---| +| Built | References in Settings' secret fields; Vaultwarden (and Bitwarden) as the first password manager; the API, the assistant's tools and MCP tools for them | +| Planned | Settings › Secrets in the web app, with a picker for the vault's items; references in other stores of secrets (sources, MCP servers, sign-in providers); more password managers | + +## References + +A reference names the manager, the item and the field to read: + +| Reference | Reads | +|---|---| +| `vaultwarden://OpenAI` | the item's default field: a login's password, a note's text, a card's number | +| `vaultwarden://OpenAI/password` | a login's `username`, `password`, `totp` (the key as it's kept), `uri` or `notes` | +| `vaultwarden://Visa/code` | a card's `number`, `code`, `cardholderName`, `brand`, `expMonth` or `expYear` | +| `vaultwarden://Slack bot/app token` | one of the item's own custom fields, by its name | + +Item names match without regard to case, and field names exactly. A `/` or `%` in a name is written `%2F` or `%25`. An identity has +no default field, so its references name one, like `vaultwarden://Me/email`. + +Any secret field in Settings (`llm.api_key`, `slack.bot_token`, `mail.password` and the rest) can hold a reference, +except the Secrets section's own sign-in. The environment still wins: a field set in `.env` ignores what Settings +holds. + +A reference is checked when it's saved. If the manager isn't set up, or it has no such item or field, the save fails +and says why, so nothing is saved that can't be read. + +## Vaultwarden + +Lens reads from a [Vaultwarden](https://github.com/dani-garcia/vaultwarden) server, or from Bitwarden, as one account +of the vault: + +1. Make an account for Lens on the vault. Its items are all that Lens can read, so keep your own account out of it. +2. Make a folder called `Lens` in it. Put the items Lens may read there, or share them into it. Lens reads that folder + only. +3. In that account, open **Settings › Security › Keys › View API key**. Copy the `client_id` and `client_secret`. +4. Give Lens the vault's address, the account's email, the API key and the account's master password with + `PUT /api/v1/settings/secrets`, or ask the admin assistant to (`change_settings`, which asks first). Then + `POST /api/v1/secrets/test` signs in and counts the items in the folder. (Settings › Secrets in the web app is + planned.) + +The vault encrypts its items end to end, so the server holds them only encrypted. The master password is what opens +them, so Lens needs it as well as the API key. Both are sealed with the server's key, like any other secret in +Settings. + +The client is [python-vaultwarden](https://github.com/numberly/python-vaultwarden) (Apache-2.0). Lens fetches it as a +component (about 10 MB) the first time a Vaultwarden address is set, so the default install carries nothing for it +([Components](components.md)). + +| Setting | Default | | +|---|---|---| +| `secrets.vaultwarden_url` | none | the vault's address, like `https://vault.example.com` | +| `secrets.vaultwarden_email` | none | the email of Lens's account on the vault | +| `secrets.vaultwarden_client_id` | none | that account's API key client id (`user.…`) | +| `secrets.vaultwarden_client_secret` | none | its client secret, kept encrypted | +| `secrets.vaultwarden_password` | none | its master password, kept encrypted | +| `secrets.vaultwarden_folder` | `Lens` | the one folder Lens reads; items elsewhere can't be referred to | +| `secrets.cache_minutes` | 10 | how long a value read is used before it's read again, 1–1440 minutes | + +## How values are read + +Lens reads a reference's value when it first builds its settings. It keeps the value in memory, never on disk, for +`secrets.cache_minutes`. After that, it reads the value again in the background. The old value keeps serving until the +new one arrives, and the settings are rebuilt when the value changed. + +* If the vault can't be reached, or it refuses Lens's sign-in, each value read before stays in use. A reference that + was never read stays empty and is tried again a minute later. Lens logs why. +* If an item is deleted, renamed, or moved out of the folder, Lens stops using its value at the next read. The field + then shows as not read, with the reason. +* Saving the Secrets section drops every value and sign-in the saving process holds, so it reads them again. The + other processes sign in again with a new API key or password the next time they read. + +## Agents and the API + +Agents can refer to secrets and check them, but they can't read them. These tools are for admins: + +| Where | What | +|---|---| +| `GET /api/v1/secrets` | the password managers and whether each is set up; every secret field, whether it's set, and for a reference whether it was read, when, or why not | +| `GET /api/v1/secrets/items?q=` | the vault's items in Lens's folder: name, type, the fields that hold something, and the reference to each; never their values | +| `POST /api/v1/secrets/test` | sign in with the saved settings and count the items Lens may read | +| `PUT /api/v1/settings/
` | set a secret field to a value or a reference | +| MCP `list_secrets`, `list_vault_items` | the same as the first two ([MCP](mcp.md#tools)) | +| MCP `set_secret_reference` | point a secret field at a reference; checked against the vault before it's saved. Needs a write-scope token | +| Assistant `list_secrets`, `vault_items` | the same, in chat ([Assistant](assistant.md#setting-up-and-running-the-server-by-chat)) | + +The assistant's `change_settings` can set the Secrets section too, and that change always waits for the admin's +approval. Its approval card shows a secret field's new reference, but never a typed secret. + +## Security + +* Lens's vault account, its API key and its master password open everything in the account. That is why Lens gets an + account of its own, which holds only the items Lens needs, and reads one folder of it. +* The vault's decrypted keys, and the values read, are held in the memory of each Lens process that uses them + (the API server and the workers). They are never written to disk or to the database. +* Settings, the API and the agents' tools show references, never values. A field filled from the vault is used where + its typed value would be: someone who can change where Lens sends that secret (an admin) can still send it + elsewhere, as they could with a typed one. +* The [threat model](https://github.com/adeelahmad/lense/blob/main/threat-model.md): P-SECRET-REFERENCES. + +## Refine later + +* References in the other stores of secrets: sources' credentials, MCP servers' environments, sign-in providers. +* A value read from the vault isn't checked the way a typed one is (the telemetry headers' format, for one). It's + checked where it's used. +* A custom CA for a self-hosted vault with a private certificate. +* An environment variable for the master password, to keep it out of the database. +* More password managers (1Password, HashiCorp Vault, Infisical), each as a provider in `secret_providers.py`. diff --git a/fastapi_backend/app/api/mcp_tools.py b/fastapi_backend/app/api/mcp_tools.py index 2ed601155..d8e6b6378 100644 --- a/fastapi_backend/app/api/mcp_tools.py +++ b/fastapi_backend/app/api/mcp_tools.py @@ -66,6 +66,7 @@ render, research, rss, + secret_providers, services, settings, sources, @@ -2551,6 +2552,85 @@ def set_file_storage( return {"saved": changes, "check": checked} +# ---------- secrets kept in a password manager (admins) ---------- +@tool( + "list_secrets", + "Secrets and where they come from", + "Every secret field in Settings (API keys, tokens, passwords; docs/secrets.md): whether it's set and, when it comes " + "from a password manager, the reference it holds (like vaultwarden://OpenAI/password) and whether this server could " + "read its value, when, or why not; and the password managers Lens can read from, whether each is set up. Never a " + "value: agents refer to secrets and never read them. Admins only.", + admin=True, +) +def list_secrets(ctx: Context) -> dict[str, Any]: + _admin(ctx) + return settings.secrets_view(ctx.db, ctx.request.app.state.archive.base) + + +@tool( + "list_vault_items", + "A password manager's items", + "The items of the password manager Lens may refer to (the one folder it reads): each one's name, type, the fields " + "that hold something and the reference to each, never their values. q narrows by name. Admins only.", + Arg("provider", "string", "the password manager", default="vaultwarden", enum=("vaultwarden",)), + Arg("q", "string", "part of an item's name", max_length=200), + Arg("limit", "integer", "at most this many", default=200, minimum=1, maximum=1000), + admin=True, +) +def list_vault_items(ctx: Context, provider: str = "vaultwarden", q: str | None = None, limit: int = 200) -> dict[str, Any]: + _admin(ctx) + p = secret_providers.PROVIDERS.get(provider) + if p is None or not p.scheme: + raise ToolError("provider is vaultwarden") + if not p.configured(ctx.cfg): + raise ToolError(f"{p.label} isn't set up: an admin sets it up in Settings → Secrets ({ctx.web}/settings/secrets)") + try: + items = p.items(ctx.cfg, q, limit) + except secret_providers.Unavailable as e: + raise ToolError(str(e)) from None + return {"provider": p.id, "items": items} + + +@tool( + "set_secret_reference", + "Point a secret at a password manager", + "Make a secret field in Settings (section and key, from list_secrets) read its value from a password manager: ref " + "is a reference like vaultwarden://OpenAI/password (an item of the folder Lens reads, and a field: a login's " + "password, username, totp, uri or notes, a card's number or code, or a custom field by name; the item alone means a " + "login's password or a note's text). It's checked against the manager first, so nothing is saved that can't be " + "read; the value itself never passes through here. Admins only, with a token that has the write scope.", + Arg("section", "string", "the settings section, like llm", required=True, max_length=40), + Arg("key", "string", "the secret field in it, like api_key", required=True, max_length=60), + Arg("ref", "string", "the reference, like vaultwarden://OpenAI/password", required=True, max_length=500), + writes=True, + admin=True, +) +def set_secret_reference(ctx: Context, section: str, key: str, ref: str) -> dict[str, Any]: + _admin(ctx) + if not ctx.user.can_write: + raise ToolError("this token is read-only: ask for a token with the write scope to point a secret at a password manager") + if key not in settings.SECRETS.get(section, ()): + raise ToolError(f"{section}.{key} isn't a secret field (list_secrets names them)") + if not secret_providers.is_reference(ref): + raise ToolError("ref is a reference like vaultwarden://Item name/field: a value never passes through here") + if key in settings.locked(section): + raise ToolError(f"{section}.{key} is set by the environment and can't be changed in the app") + try: + settings.save(ctx.db, ctx.request.app.state.archive.base, section, {key: ref}, ctx.user.email) + except ValueError as e: + raise ToolError(str(e)) from None + auth.audit(ctx.db, ctx.user.as_audit(), "settings.save", section, [key, "mcp"]) + shown = next( + ( + f + for f in settings.secrets_view(ctx.db, ctx.request.app.state.archive.base)["references"] + if f["section"] == section and f["key"] == key + ), + None, + ) + return {"saved": {"section": section, "key": key, "ref": ref}, "status": shown} + + # ---------- monitoring: Prometheus and Grafana next to Lens (admins) ---------- @tool( "monitoring", diff --git a/fastapi_backend/app/api/v1/router.py b/fastapi_backend/app/api/v1/router.py index 5be72c927..b11080370 100644 --- a/fastapi_backend/app/api/v1/router.py +++ b/fastapi_backend/app/api/v1/router.py @@ -60,6 +60,7 @@ routines, search, searches, + secrets, sensors, setup, shares, @@ -88,6 +89,7 @@ for module in ( users, admin, + secrets, backups, # before updates: its /admin/backups/remote would otherwise be read as a backup's name updates, activity, diff --git a/fastapi_backend/app/api/v1/routes/secrets.py b/fastapi_backend/app/api/v1/routes/secrets.py new file mode 100644 index 000000000..a3459ed05 --- /dev/null +++ b/fastapi_backend/app/api/v1/routes/secrets.py @@ -0,0 +1,54 @@ +"""Secrets kept in a password manager (docs/secrets.md): the managers Lens can read from, which secret fields refer +to one and whether their values could be read, and a manager's items to refer to. Never a secret's value: a secret +field is set with ``PUT /settings/{section}`` (a value, or a reference like ``vaultwarden://Item/password``), and from +then on nothing shows it. Admins only.""" + +from __future__ import annotations + +from fastapi import APIRouter, HTTPException, Query, Request + +from app.api.deps import AdminReader, AdminWriter, Cfg, Db +from app.domain import secret_providers, settings +from app.schemas.secrets import SecretsTestResult, SecretsTry, SecretsView, VaultItems + +router = APIRouter(prefix="/secrets", tags=["admin"]) + + +@router.get("") +def get_secrets(user: AdminReader, request: Request, db: Db) -> SecretsView: + """The password managers Lens can read secrets from and whether each is set up; every secret field in Settings, + whether it's set, and for one that refers to a password manager its reference and whether this server could read + its value (and when, or why not).""" + return SecretsView(**settings.secrets_view(db, request.app.state.archive.base)) + + +@router.get("/items") +def list_vault_items( + user: AdminReader, + cfg: Cfg, + provider: str = Query("vaultwarden", description="the password manager"), + q: str | None = Query(None, max_length=200, description="part of an item's name"), + limit: int = Query(200, ge=1, le=1000), +) -> VaultItems: + """The items of a password manager Lens may refer to (those in its folder): each one's name, type, the fields that + hold something and the reference to each. Never their values. 409 when the manager isn't set up or can't be + reached, with why.""" + p = secret_providers.PROVIDERS.get(provider) + if p is None or not p.scheme: + raise HTTPException(400, f"provider is one of: {', '.join(k for k, v in secret_providers.PROVIDERS.items() if v.scheme)}") + if not p.configured(cfg): + raise HTTPException(409, f"{p.label} isn't set up: Settings › Secrets") + try: + items = p.items(cfg, q, limit) + except secret_providers.Unavailable as e: + raise HTTPException(409, str(e)) from None + return VaultItems(provider=p.id, items=items) + + +@router.post("/test") +def test_secrets(user: AdminWriter, cfg: Cfg, body: SecretsTry | None = None) -> SecretsTestResult: + """Sign in to a password manager with the settings saved and count the items Lens may read, to check them.""" + p = secret_providers.PROVIDERS.get((body or SecretsTry()).provider) + if p is None: + raise HTTPException(400, f"provider is one of: {', '.join(secret_providers.PROVIDERS)}") + return SecretsTestResult(**p.test(cfg)) diff --git a/fastapi_backend/app/domain/components.py b/fastapi_backend/app/domain/components.py index 4e7239211..5841c206f 100644 --- a/fastapi_backend/app/domain/components.py +++ b/fastapi_backend/app/domain/components.py @@ -772,6 +772,20 @@ def fetch(self, cfg, m, say): pip_install(cfg, "sandbox", say) +class VaultwardenClient(Component): + """python-vaultwarden, for secrets read from a Vaultwarden or Bitwarden vault (vaultwarden_secrets.py): fetched + once an admin sets the vault's address.""" + + def needed(self, cfg, m): + return bool((cfg.get("secrets") or {}).get("vaultwarden_url")) + + def present(self, cfg): + return importable("vaultwarden.clients.bitwarden") + + def fetch(self, cfg, m, say): + pip_install(cfg, "vaultwarden", say) + + class WebSearch(Component): """ddgs, for the assistant's web search through DuckDuckGo (web.py): fetched when an admin picks DuckDuckGo.""" @@ -990,6 +1004,13 @@ def detail(self, cfg): WebSearch( "web-search", "DuckDuckGo search", "the assistant's web search through DuckDuckGo (web.search; ddgs)", size_mb=30, license="MIT" ), + VaultwardenClient( + "vaultwarden", + "Vaultwarden client", + "reads secrets from a Vaultwarden or Bitwarden vault (secrets.vaultwarden_url; python-vaultwarden)", + size_mb=10, + license="Apache-2.0", + ), Msg("msg", "Outlook .msg emails", "reads Outlook .msg files (extract-msg)", optional=True, license="GPL-3.0", size_mb=5), ] BY_ID = {c.id: c for c in COMPONENTS} diff --git a/fastapi_backend/app/domain/ops_tools.py b/fastapi_backend/app/domain/ops_tools.py index 9b8e2f500..4aeb7510b 100644 --- a/fastapi_backend/app/domain/ops_tools.py +++ b/fastapi_backend/app/domain/ops_tools.py @@ -11,7 +11,25 @@ from __future__ import annotations -from . import auth, blobs, convert, decide, feeds, jobs, links, llm, metadata, roles, rss, settings, setup, sources, store, uploads +from . import ( + auth, + blobs, + convert, + decide, + feeds, + jobs, + links, + llm, + metadata, + roles, + rss, + secret_providers, + settings, + setup, + sources, + store, + uploads, +) R = store.R _S = {"type": "string"} @@ -47,10 +65,11 @@ "sensors", "shares", "web", + "secrets", ) # telemetry, analytics and web search send data elsewhere; sensors and shares open ports on the network; files moves -# where new files go -ALWAYS_ASK = {"telemetry", "analytics", "sensors", "shares", "files", "web"} +# where new files go; secrets is where Lens signs in to a password manager +ALWAYS_ASK = {"telemetry", "analytics", "sensors", "shares", "files", "web", "secrets"} ASK_TOOLS = {"invite_person", "change_person"} # who gets in always waits for the admin's approval ADMIN_TOOLS = [ @@ -104,6 +123,16 @@ [], False, ), + ( + "list_secrets", + "Every secret field in Settings (API keys, tokens, passwords): whether it's set, and the reference it holds when it " + "comes from a password manager (like vaultwarden://OpenAI/password) with whether that could be read; and the password " + "managers Lens can read from. Never a value. A secret field takes a reference through change_settings, e.g. llm " + "{api_key: 'vaultwarden://OpenAI/password'}; the manager is set up with change_settings secrets.", + {}, + [], + False, + ), ( "subscribe_feed", "Follow a news feed (RSS or Atom) into a namespace: url is the feed's address, or a page of the site that names " @@ -121,6 +150,14 @@ ["feed"], True, ), + ( + "vault_items", + "The items of the password manager Lens may refer to (the folder it reads): each one's name, type, the fields that " + "hold something and the reference to each, never their values. q: part of a name.", + {"q": _S, "provider": {"type": "string", "enum": ["vaultwarden"]}}, + [], + False, + ), ( "create_namespace", "Create a namespace: a separate set of assets (recordings, documents and images) with its own people and access. " @@ -431,10 +468,29 @@ def t_change_settings(self, section, changes): raise ValueError(f"{section} isn't one of {', '.join(SECTIONS)}") if not isinstance(changes, dict) or not changes: raise ValueError("changes is an object of the settings to change") - shown = {k: ("(a secret)" if k in settings.SECRETS.get(section, ()) else v) for k, v in changes.items()} + shown = { + k: (v if secret_providers.is_reference(v) else "(a secret)") if k in settings.SECRETS.get(section, ()) else v + for k, v in changes.items() + } what = ", ".join(f"{k} = {v}" for k, v in shown.items()) return self._change("change_settings", {"section": section, "changes": changes}, f"Set {section}: {what}") + def t_list_secrets(self): + if not self.admin: + raise PermissionError("only admins see the secrets settings") + return settings.secrets_view(self.db, self.base), "Listed the secret settings and where each comes from" + + def t_vault_items(self, q=None, provider="vaultwarden"): + if not self.admin: + raise PermissionError("only admins see the password manager's items") + p = secret_providers.PROVIDERS.get(provider) + if p is None or not p.scheme: + raise ValueError("provider is vaultwarden") + if not p.configured(self.cfg): + raise ValueError(f"{p.label} isn't set up yet: Settings → Secrets, or change_settings secrets") + items = p.items(self.cfg, q) + return {"provider": p.id, "items": items}, f"Listed {len(items)} item(s) Lens may refer to in {p.label}" + def t_file_storage(self, check=False): f = self.cfg.get("files") or {} out = { diff --git a/fastapi_backend/app/domain/secret_providers.py b/fastapi_backend/app/domain/secret_providers.py new file mode 100644 index 000000000..0b099c25a --- /dev/null +++ b/fastapi_backend/app/domain/secret_providers.py @@ -0,0 +1,318 @@ +"""Secrets kept outside Lens: a secret field can hold a reference to an item in a password manager instead of the secret. + +A secret field in Settings (an API key, a token, a password: settings.SECRETS) holds the secret itself, sealed with the +server's key (the built-in provider, the default), or a reference to where it is kept, such as +``vaultwarden://OpenAI/password``: the provider, the item and the field to read. Lens reads the value when it builds its +settings (settings.effective) and keeps it in memory for secrets.cache_minutes; after that it reads it again in the +background, so a key changed in the vault reaches Lens without anyone saving a setting. Only the reference is stored. + +People and agents see references, never what they point at: Settings, the API (``GET /settings``, ``GET /secrets``) +and the assistant's and MCP's ``list_secrets`` show a field's reference and whether it could be read, and listing a +vault shows its items' names and field names, not their values. A provider reads only the items it was allowed: +Vaultwarden, those in one folder (secrets.vaultwarden_folder, "Lens" by default). + +A vault that can't be reached for a while keeps the values read before it; an item that is gone, or moved out of the +folder, stops being used at the next read. Providers: builtin (the sealed store, always there) and vaultwarden +(vaultwarden_secrets.py). A new one is a Provider in PROVIDERS, with its own scheme. +""" + +from __future__ import annotations + +import dataclasses +import datetime as dt +import logging +import re +import threading +import time +import urllib.parse + +log = logging.getLogger("lens") + +RETRY_SECONDS = 60 # a reference that couldn't be read is tried again after this +_now = time.time # the clock (tests move it on) +FIELD_RX = re.compile(r"^[^\x00-\x1f]{1,200}$") + + +class Missing(ValueError): + """The vault answered: there's no such item or field where Lens may read (the reference stops being used).""" + + +class Unavailable(RuntimeError): + """The vault couldn't be asked: not set up, not installed yet, unreachable, or it refused Lens's sign-in.""" + + +@dataclasses.dataclass(frozen=True) +class Ref: + provider: str + item: str + field: str | None + text: str + + +class Provider: + """Where secrets can be kept. `scheme` is how references to it start (None: it's not referred to).""" + + id = "" + label = "" + scheme: str | None = None + + def configured(self, cfg) -> bool: + return False + + def describe(self, cfg) -> dict: + """What Settings, the API and agents see about it: never a secret.""" + return {"id": self.id, "label": self.label, "scheme": self.scheme, "configured": self.configured(cfg)} + + def fingerprint(self, cfg) -> str: + """What values read from it are kept under: another server, account or scope reads them again.""" + return self.id + + def read(self, cfg, refs: list[Ref]) -> dict[str, str | Exception]: + """Each reference's value, or why it couldn't be read (Missing); raises Unavailable when nothing could.""" + raise Unavailable(f"{self.label} can't be referred to") + + def items(self, cfg, search=None, limit=200) -> list[dict]: + """The items a reference may point at: names, field names and the reference to each; never their values.""" + return [] + + def test(self, cfg) -> dict: + return {"ok": True} + + def reset(self): + """Drop any sign-in held: its settings changed.""" + + +class Builtin(Provider): + """Secrets typed into Settings, kept in the database sealed with the server's key (settings.seal).""" + + id = "builtin" + label = "Saved in Lens" + + def configured(self, cfg): + return True + + def describe(self, cfg): + return {**super().describe(cfg), "scope": "encrypted with this server's key (AES-256-GCM)"} + + +def by_scheme(scheme): + return next((p for p in PROVIDERS.values() if p.scheme and p.scheme == scheme), None) + + +# ---------- references ---------- +def _quote(part): + return part.replace("%", "%25").replace("/", "%2F") + + +def reference(provider, item, field=None): + """The reference to an item's field, e.g. vaultwarden://OpenAI/password (a / or % in a name is escaped).""" + p = PROVIDERS[provider] + return f"{p.scheme}://{_quote(item)}" + (f"/{_quote(field)}" if field else "") + + +def parse(value) -> Ref | None: + """The reference a secret field holds, or None when it holds a secret (or nothing).""" + if not isinstance(value, str) or "://" not in value: + return None + text = value.strip() + scheme, _, rest = text.partition("://") + p = by_scheme(scheme.lower()) + if p is None: + return None + item, _, field = rest.partition("/") + return Ref(p.id, urllib.parse.unquote(item), urllib.parse.unquote(field) or None, text) + + +def is_reference(value) -> bool: + return parse(value) is not None + + +def check(value) -> Ref: + """A reference as a secret field may hold it; ValueError says what's wrong.""" + ref = parse(value) + if ref is None: + raise ValueError("not a reference: write it like vaultwarden://Item name/field") + if not ref.item or not FIELD_RX.match(ref.item): + raise ValueError(f"{ref.text} doesn't name an item: write it like {PROVIDERS[ref.provider].scheme}://Item name/field") + if ref.field is not None and not FIELD_RX.match(ref.field): + raise ValueError(f"{ref.text} names no field it can read") + return ref + + +# ---------- what was read, kept in this process ---------- +@dataclasses.dataclass +class _Entry: + value: str | None + error: str | None + read_at: float | None # when the value was read + tried_at: float # when it was last asked for + missing: bool = False # the vault answered that it isn't there (not: the vault couldn't be asked) + + +_CACHE: dict[tuple[str, str], _Entry] = {} +_LOCK = threading.Lock() # guards _CACHE and the counters +_READING = threading.Lock() # one read of a vault at a time in this process +_STATE = {"generation": 0, "next": 0.0, "busy": False} + + +def generation(): + """Changes whenever a value read from a vault (or why it couldn't be) changes: settings are built again.""" + return _STATE["generation"] + + +def forget(): + """Drop everything read and every sign-in held: the secrets settings changed (or a test starts afresh).""" + with _LOCK: + _CACHE.clear() + _STATE.update(next=0.0, busy=False) + _STATE["generation"] += 1 + for p in PROVIDERS.values(): + p.reset() + + +def _why(e) -> str: + return str(e)[:300] or type(e).__name__ + + +def _read(cfg, provider, refs): + """Read some references from one provider now, and keep what was read.""" + fp = provider.fingerprint(cfg) + with _READING: + t0 = _now() + try: + if not provider.configured(cfg): + raise Unavailable(f"{provider.label} isn't set up: Settings › Secrets") + got = provider.read(cfg, refs) + except Exception as e: # noqa: BLE001 - the whole vault: every reference keeps what it had + log.warning("secrets: couldn't read %s: %s", provider.label, _why(e)) + got = {r.text: e for r in refs} + changed = False + with _LOCK: + for r in refs: + old = _CACHE.get((fp, r.text)) + v = got.get(r.text) + if isinstance(v, str): + new = _Entry(v, None, t0, t0) + elif isinstance(v, Missing): # gone, or out of the folder: Lens stops using it + new = _Entry(None, _why(v), None, t0, missing=True) + else: # couldn't ask: keep what was read before, if anything + err = _why(v) if v is not None else f"{provider.label} didn't return it" + new = _Entry(old.value if old else None, err, old.read_at if old else None, t0) + if old is None or (old.value, old.error) != (new.value, new.error): + changed = True + _CACHE[(fp, r.text)] = new + if changed: + _STATE["generation"] += 1 + _STATE["next"] = 0.0 + + +def resolve(cfg, found: dict) -> dict: + """The values of references, {where: reference} -> {where: value or None}; what hasn't been read yet is read now.""" + by_provider: dict[str, list[tuple[object, Ref]]] = {} + for where, text in found.items(): + ref = parse(text) + if ref is not None: + by_provider.setdefault(ref.provider, []).append((where, ref)) + out = {} + for pid, items in by_provider.items(): + p = PROVIDERS[pid] + fp = p.fingerprint(cfg) + with _LOCK: + missing = list({r.text: r for _, r in items if (fp, r.text) not in _CACHE}.values()) + if missing: + _read(cfg, p, missing) + with _LOCK: + for where, r in items: + e = _CACHE.get((fp, r.text)) + out[where] = e.value if e else None + return out + + +def _due(cfg, texts, now): + """The references to read again: kept longer than secrets.cache_minutes, or not read and last tried a while ago.""" + ttl = 60 * int(((cfg.get("secrets") or {}).get("cache_minutes")) or 10) + due, nxt = {}, now + ttl + for text in texts: + ref = parse(text) + if ref is None: + continue + p = PROVIDERS[ref.provider] + e = _CACHE.get((p.fingerprint(cfg), text)) + at = (e.read_at + ttl) if e and e.value is not None and e.read_at else ((e.tried_at if e else 0) + RETRY_SECONDS) + if at <= now: + due.setdefault(ref.provider, []).append(ref) + else: + nxt = min(nxt, at) + return due, nxt + + +def tick(cfg, texts): + """Called as settings are used: once a value is due, it's read again in the background (the old one serves until + then), and the settings are built again when something changed. Cheap when nothing is due.""" + now = _now() + with _LOCK: + if _STATE["busy"] or now < _STATE["next"]: + return + due, nxt = _due(cfg, texts, now) + if not due: + _STATE["next"] = nxt + return + _STATE["busy"] = True + + def run(): + try: + for pid, refs in due.items(): + _read(cfg, PROVIDERS[pid], refs) + finally: + with _LOCK: + _STATE["busy"] = False + + threading.Thread(target=run, daemon=True, name="secrets-refresh").start() + + +def verify(cfg, value): + """Check a reference before it's saved: its provider is set up, and the vault has the item and field; ValueError + says what's wrong, so nothing is saved that can't be read.""" + ref = check(value) + p = PROVIDERS[ref.provider] + if not p.configured(cfg): + raise ValueError(f"set up {p.label} in Settings › Secrets before referring to it") + _read(cfg, p, [ref]) + with _LOCK: + e = _CACHE.get((p.fingerprint(cfg), ref.text)) + if e is None or e.value is None: + raise ValueError((e.error if e else None) or f"{p.label} didn't return {ref.text}") + return ref + + +def status(cfg, value) -> dict: + """Whether a reference could be read in this process, and when; never the value.""" + ref = parse(value) + if ref is None: + return {} + p = PROVIDERS[ref.provider] + with _LOCK: + e = _CACHE.get((p.fingerprint(cfg), ref.text)) + if e is None: + return {"ref": ref.text, "provider": ref.provider, "read": None} + when = dt.datetime.fromtimestamp(e.read_at, dt.timezone.utc).isoformat(timespec="seconds") if e.read_at else None + return {"ref": ref.text, "provider": ref.provider, "read": e.value is not None, "error": e.error, "read_at": when, "missing": e.missing} + + +def mask(cfg, value) -> dict: + """How a set secret field is shown: that it's set, and its reference when it holds one.""" + out = {"secret": True, "set": True} + if parse(value) is not None: + out.update(status(cfg, value)) + return out + + +def providers(cfg) -> list[dict]: + return [p.describe(cfg) for p in PROVIDERS.values()] + + +PROVIDERS: dict[str, Provider] = {"builtin": Builtin()} + +# each provider adds itself to PROVIDERS; at the end, since it builds on the classes above (and either module may be +# imported first) +from . import vaultwarden_secrets # noqa: E402,F401 diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py index 732202223..d3ded97f1 100644 --- a/fastapi_backend/app/domain/settings.py +++ b/fastapi_backend/app/domain/settings.py @@ -19,7 +19,7 @@ import threading import urllib.parse -from . import convert, ipgroups, logs, mail_routes, objects, store, telemetry +from . import convert, ipgroups, logs, mail_routes, objects, secret_providers, store, telemetry R = store.R EDITABLE = { @@ -62,6 +62,7 @@ "automations": None, "backups": None, "encryption": None, + "secrets": None, "fedora": None, "tunnel": None, "https": None, @@ -153,6 +154,7 @@ "documents": ("anytopdf_token",), "components": ("hf_token",), "web": ("api_key",), + "secrets": ("vaultwarden_client_secret", "vaultwarden_password"), } ENUMS = { ("transcribe", "engine"): {"sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"}, @@ -327,8 +329,12 @@ def _data(row): return json.loads(d) if isinstance(d, str) else dict(d or {}) -def effective(db, base, reveal=True): +def effective(db, base, reveal=True, refs=None): + """The configuration as saved: archive.yaml, the app's settings over it, the environment over both. A secret field + that holds a reference to a password manager (secret_providers.py) carries the value read from it (None until it's + read), or with reveal off its reference and whether it could be read; `refs`, a list, collects the references.""" cfg = copy.deepcopy(base) + found = {} for row in db.rows("SELECT record::id(id) AS section, data, sealed FROM app_setting"): sec = row["section"] if sec not in EDITABLE or not isinstance(cfg.get(sec), dict): @@ -339,17 +345,31 @@ def effective(db, base, reveal=True): server = (base.get("documents") or {}).get("web_networks") or [] cfg[sec]["web_networks"] = list(dict.fromkeys([*server, *(cfg[sec].get("web_networks") or [])])) for k, sealed in (row.get("sealed") or {}).items(): - if not reveal: - cfg[sec][k] = {"secret": True, "set": True} - continue try: - cfg[sec][k] = unseal(base, sealed, f"setting:{sec}.{k}") + value = unseal(base, sealed, f"setting:{sec}.{k}") except Exception: # noqa: BLE001 - key changed since it was saved - cfg[sec][k] = None + value = None + if secret_providers.is_reference(value): + found[(sec, k)] = value + cfg[sec][k] = None # the value read from the password manager, below + elif not reveal: + cfg[sec][k] = {"secret": True, "set": True} + else: + cfg[sec][k] = value for sec, key in ENV_OVERRIDES: v = env_value(sec, key) if v is not None: + found.pop((sec, key), None) cfg[sec][key] = {"secret": True, "set": True} if not reveal and key in SECRETS.get(sec, ()) else v + if found: # once every section is in, the secrets section (the password manager's own sign-in) is complete + if refs is not None: + refs.extend(found.values()) + if reveal: + for (sec, k), v in secret_providers.resolve(cfg, found).items(): + cfg[sec][k] = v + else: + for (sec, k), text in found.items(): + cfg[sec][k] = secret_providers.mask(cfg, text) return cfg @@ -365,19 +385,26 @@ def keep_passwords(db, base): class Settings: - """The effective configuration, rebuilt only when someone saves a setting.""" + """The effective configuration, rebuilt only when someone saves a setting, or a secret read from a password + manager changes (secret_providers.generation).""" def __init__(self, db, base): self.db, self.base, self._cfg, self._ver, self._lock = db, base, None, None, threading.Lock() + self._gen, self._refs = None, [] def current(self): v = self.db.values("SELECT VALUE n FROM $r", r=R("seq", "settings")) + g = secret_providers.generation() with self._lock: - if self._cfg is None or v != self._ver: - self._cfg, self._ver = effective(self.db, self.base), v + if self._cfg is None or v != self._ver or g != self._gen: + refs = [] + self._cfg, self._ver, self._gen, self._refs = effective(self.db, self.base, refs=refs), v, g, refs telemetry.apply(self._cfg) # turned on, changed or off in the app: this process follows logs.apply(self._cfg) # and its logging (level, file, syslog) - return self._cfg + cfg, refs = self._cfg, self._refs + if refs: + secret_providers.tick(cfg, refs) # a value kept past secrets.cache_minutes is read again, in the background + return cfg def view(db, base): @@ -409,6 +436,22 @@ def view(db, base): return out +def secrets_view(db, base): + """Every secret field in Settings: whether it's set and, when it holds a reference to a password manager, the + reference and whether its value could be read; and the password managers there are. Never a value.""" + eff = effective(db, base, reveal=False) + fields, refs = [], [] + for sec, keys in SECRETS.items(): + for k in keys: + v = (eff.get(sec) or {}).get(k) + row = {"section": sec, "key": k, "set": bool(isinstance(v, dict) and v.get("set")), "locked": k in locked(sec)} + if isinstance(v, dict) and v.get("ref"): + row["ref"] = v["ref"] + refs.append({"section": sec, "key": k, **{x: v.get(x) for x in ("ref", "provider", "read", "error", "read_at", "missing")}}) + fields.append(row) + return {"providers": secret_providers.providers(eff), "references": refs, "fields": fields} + + def _updates_setting(key, value): if key == "check_hours": if not (isinstance(value, int) and not isinstance(value, bool) and 1 <= value <= 24 * 30): @@ -502,6 +545,8 @@ def _check(section, key, value, default): return _backup_setting(key, value) if section == "fedora": return _fedora_setting(key, value) + if section == "secrets": + return _secrets_setting(key, value) if section == "shares": return _share_setting(key, value) if section == "sensors": @@ -1626,6 +1671,13 @@ def save(db, base, section, changes, user=None): if k in SECRETS.get(section, ()): if v is None or v == "": sealed.pop(k, None) + elif isinstance(v, str) and secret_providers.is_reference(v): + # a reference to a password manager's item (secret_providers.py), kept in the secret's place: checked + # against the manager now, so nothing is saved that can't be read + if section == "secrets": + raise ValueError(f"secrets.{k} is the password manager's own sign-in: it's kept in Lens, not read from one") + ref = secret_providers.verify(effective(db, base), v) + sealed[k] = seal(base, ref.text, f"setting:{section}.{k}") elif isinstance(v, str): if (section, k) == ("telemetry", "headers"): telemetry.parse_headers(v) # ValueError when malformed @@ -1713,11 +1765,51 @@ def save(db, base, section, changes, user=None): d={"data": json.dumps(data), "sealed": sealed, "updated_at": store.now(), "updated_by": user}, ) db.next_id("settings") + if section == "secrets": + secret_providers.forget() # the password manager's sign-in or scope changed: what was read is read again FEDORA_RANGES = {"max_file_mb": (0, 1_000_000), "sync_seconds": (10, 86400), "full_hours": (1, 720)} +SECRETS_RANGES = {"cache_minutes": (1, 1440)} +VAULT_FOLDER_RX = re.compile(r"^[^\x00-\x1f/]{1,100}$") + + +def _secrets_setting(key, value): + if key in SECRETS_RANGES: + lo, hi = SECRETS_RANGES[key] + if not (isinstance(value, int) and not isinstance(value, bool) and lo <= value <= hi): + raise ValueError(f"secrets.{key} is a whole number of minutes from {lo} to {hi}") + return value + if key == "vaultwarden_url": + if value in (None, ""): + return None + u = urllib.parse.urlsplit(value.strip()) if isinstance(value, str) else None + if not (u and VIEWER_URL.match(value.strip()) and u.hostname and u.username is None and u.password is None and not u.query): + raise ValueError( + "secrets.vaultwarden_url is the server's http(s) address with no password in it, such as https://vault.example" + ) + return value.strip().rstrip("/") + if key == "vaultwarden_email": + if value in (None, ""): + return None + if not (isinstance(value, str) and EMAIL_RX.match(value.strip())): + raise ValueError("secrets.vaultwarden_email is the email address of the account Lens signs in as") + return value.strip() + if key == "vaultwarden_client_id": + if value in (None, ""): + return None + if not (isinstance(value, str) and re.fullmatch(r"[A-Za-z0-9._-]{1,200}", value.strip())): + raise ValueError("secrets.vaultwarden_client_id is the account's API key client id, like user.8c3d…") + return value.strip() + if key == "vaultwarden_folder": + if not (isinstance(value, str) and VAULT_FOLDER_RX.match(value.strip())): + raise ValueError("secrets.vaultwarden_folder is the name of the one folder Lens reads") + return value.strip() + raise ValueError(f"unknown setting secrets.{key}") + + def _fedora_setting(key, value): if key in ("enabled", "files"): if not isinstance(value, bool): diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 362812c12..dae1c8498 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -681,6 +681,19 @@ def labels(): # encryption at rest (docs/encryption.md): files Lens keeps under data_dir, encrypted with their namespace's key; # work_minutes: how long a plain working copy for ffmpeg and the other tools is kept after its last use "encryption": {"files": False, "work_minutes": 30, "vault_minutes": 60}, + # secrets kept in a password manager (docs/secrets.md; domain/secret_providers.py): a secret field in Settings can + # hold a reference like vaultwarden://Item/password instead of the secret, read when the settings are built and + # again after cache_minutes. Vaultwarden (or Bitwarden): the server's address, the account Lens signs in as (its + # email, API key and master password, the last two secrets) and the one folder of it Lens may read. + "secrets": { + "cache_minutes": 10, + "vaultwarden_url": None, + "vaultwarden_email": None, + "vaultwarden_client_id": None, + "vaultwarden_client_secret": None, + "vaultwarden_password": None, + "vaultwarden_folder": "Lens", + }, # the activity ledger (docs/activity.md): every change and call in or out, with its cost, per resource. On by # default; reads: log API reads (GET) too; keep_days: how long rows are kept "activity": {"enabled": True, "reads": False, "keep_days": 365}, diff --git a/fastapi_backend/app/domain/vaultwarden_secrets.py b/fastapi_backend/app/domain/vaultwarden_secrets.py new file mode 100644 index 000000000..dc93557e7 --- /dev/null +++ b/fastapi_backend/app/domain/vaultwarden_secrets.py @@ -0,0 +1,335 @@ +"""Secrets read from a Vaultwarden or Bitwarden vault (secret_providers.py; docs/secrets.md). + +Lens signs in as one account of the vault with that account's API key (client id and secret, from the vault's +Settings › Security › Keys) and its master password, which the vault's end-to-end encryption needs: the server only +ever holds the items encrypted, so without the password there is nothing to read. Make an account for Lens, put the +items it may read (or share them) into one folder of it (secrets.vaultwarden_folder, "Lens" by default), and keep +everything else out: Lens reads only that folder. Each read syncs the vault, decrypts the folder's item names to find +the item, and the one field referred to; the sign-in is held in memory, never written down, and the values read are +kept in memory for secrets.cache_minutes. + +A reference is ``vaultwarden://`` (the item's password, or a note's text) or ``vaultwarden:///``: a login's ``username``, ``password``, ``totp`` (the key as kept), ``uri`` or ``notes``, a card's +``number`` or ``code``, or one of the item's own custom fields by its name. + +The client is python-vaultwarden (Apache-2.0, its crypto from bitwardentools), fetched into the data folder as a +component the first time a Vaultwarden address is set (components.py), so the default install carries nothing for it. +""" + +from __future__ import annotations + +import hashlib +import logging +import threading +import uuid + +from . import components +from .secret_providers import PROVIDERS, Missing, Provider, Unavailable, reference + +log = logging.getLogger("lens") + +TYPES = {1: "login", 2: "note", 3: "card", 4: "identity"} +# a type's own fields, in the part of the item they're kept in; "notes" is on every item +OWN = { + 1: ("login", ("username", "password", "totp", "uri")), + 2: ("secureNote", ()), + 3: ("card", ("cardholderName", "number", "brand", "expMonth", "expYear", "code")), + 4: ( + "identity", + ( + "title", "firstName", "middleName", "lastName", "address1", "address2", "address3", "city", "state", + "postalCode", "country", "company", "email", "phone", "ssn", "username", "passportNumber", "licenseNumber", + ), + ), +} # fmt: skip +DEFAULT_FIELD = {1: "password", 2: "notes", 3: "number", 4: None} +TIMEOUT = 20 + + +def settings(cfg): + s = (cfg or {}).get("secrets") or {} + return { + "url": (s.get("vaultwarden_url") or "").strip().rstrip("/"), + "email": (s.get("vaultwarden_email") or "").strip(), + "client_id": (s.get("vaultwarden_client_id") or "").strip(), + "client_secret": s.get("vaultwarden_client_secret") or "", + "password": s.get("vaultwarden_password") or "", + "folder": (s.get("vaultwarden_folder") or "Lens").strip(), + } + + +def _get(d, *names): + """A field of what the server sent, whichever case it uses (Vaultwarden answers camelCase, Bitwarden did PascalCase).""" + for n in names: + for k in (n, n[:1].upper() + n[1:]): + if isinstance(d, dict) and d.get(k) is not None: + return d[k] + return None + + +def _hash(*parts): + """A key in memory for settings, some of them secret: never shown or kept.""" + return hashlib.sha256("\n".join(parts).encode()).hexdigest()[:16] + + +def device_id(cfg): + """A device identifier of this server's own, the same every time, derived from its secret key (nothing kept).""" + from . import settings as app_settings + + h = hashlib.sha256(b"lens vaultwarden device " + app_settings.secret_key(cfg)).digest() + return str(uuid.UUID(bytes=h[:16], version=4)) + + +class _Item: + """One item of the folder, with its fields decrypted only when asked for.""" + + def __init__(self, raw, key, decrypt): + self.raw, self.key, self._decrypt = raw, key, decrypt + self.type = int(_get(raw, "type") or 0) + self.name = self._text(_get(raw, "name")) or "" + + def _text(self, value): + if not value: + return None + return self._decrypt(value, self.key).decode("utf-8") + + def fields(self): + """The names of the fields that hold something: the type's own, notes, and the custom ones.""" + part, own = OWN.get(self.type, (None, ())) + sub = _get(self.raw, part) if part else None + names = [f for f in own if _get(sub, f) or (f == "uri" and _get(sub, "uris"))] + if _get(self.raw, "notes"): + names.append("notes") + for f in _get(self.raw, "fields") or []: + n = self._text(_get(f, "name")) + if n and _get(f, "value"): + names.append(n) + return names + + def value(self, field): + """The field's text; Missing when the item has no such field (or it's empty).""" + part, own = OWN.get(self.type, (None, ())) + sub = _get(self.raw, part) if part else None + if field in own: + if field == "uri": + uris = _get(sub, "uris") or [] + raw = _get(uris[0], "uri") if uris else _get(sub, "uri") + else: + raw = _get(sub, field) + text = self._text(raw) + if text: + return text + elif field == "notes": + text = self._text(_get(self.raw, "notes")) + if text: + return text + else: + for f in _get(self.raw, "fields") or []: + if self._text(_get(f, "name")) == field: + text = self._text(_get(f, "value")) + if text: + return text + break + raise Missing(f"the item {self.name} has no {field} (it has: {', '.join(self.fields()) or 'nothing'})") + + +class _Vault: + """What one sync found in the folder: items by name.""" + + def __init__(self, folder, items): + self.folder, self.items = folder, items + self.by_name = {} + for it in items: + self.by_name.setdefault(it.name.casefold(), []).append(it) + + def item(self, name): + found = self.by_name.get(name.strip().casefold()) or [] + if not found: + raise Missing(f"there's no item called {name} in the folder {self.folder}") + if len(found) > 1: + raise Missing(f"there are {len(found)} items called {name} in the folder {self.folder}: rename one") + return found[0] + + def value(self, name, field): + it = self.item(name) + f = field or DEFAULT_FIELD.get(it.type) + if not f: + raise Missing(f"say which field of {name} to read, like {reference('vaultwarden', name, 'email')}") + return it.value(f) + + +class Vaultwarden(Provider): + id = "vaultwarden" + label = "Vaultwarden" + scheme = "vaultwarden" + + def __init__(self): + self._lock = threading.Lock() + self._client = None # (sign-in settings' hash, client): one signed-in client, kept while they stay + self._vault = None # (fingerprint, _Vault): the last sync, for listing + + def configured(self, cfg): + s = settings(cfg) + return all(s[k] for k in ("url", "email", "client_id", "client_secret", "password")) + + def fingerprint(self, cfg): + s = settings(cfg) + return "vaultwarden:" + _hash(s["url"], s["email"], s["client_id"], s["folder"]) + + def describe(self, cfg): + s = settings(cfg) + return { + **super().describe(cfg), + "url": s["url"] or None, + "email": s["email"] or None, + "folder": s["folder"], + "installed": self.installed(cfg), + "scope": f"the items in the folder {s['folder']} of the account {s['email']}" + if s["email"] + else "the items in one folder of one account", + } + + def installed(self, cfg): + components.activate(cfg) + return components.importable("vaultwarden.clients.bitwarden") + + def reset(self): + with self._lock: + self._client = self._vault = None + + # ---------- talking to the vault ---------- + def _sdk(self, cfg): + components.activate(cfg) # the package, fetched into the data folder once an address was set + try: + from vaultwarden.clients.bitwarden import BitwardenAPIClient + from vaultwarden.utils import crypto + except ImportError: + raise Unavailable( + "the Vaultwarden client isn't installed yet: it's fetched once an address is set (Settings › Components)" + ) from None + return BitwardenAPIClient, crypto + + def _signed_in(self, cfg): + s = settings(cfg) + fp = _hash(*(s[k] for k in ("url", "email", "client_id", "client_secret", "password"))) # a new key or password signs in again + with self._lock: + if self._client and self._client[0] == fp: + return self._client[1] + BitwardenAPIClient, _ = self._sdk(cfg) + client = BitwardenAPIClient( + s["url"], s["email"], s["password"], s["client_id"], s["client_secret"], device_id(cfg), timeout=TIMEOUT + ) + with self._lock: + self._client = (fp, client) + return client + + def _sync(self, cfg): + """Sign in (or use the sign-in held) and sync; what went wrong, in words that name no secret.""" + import httpx + + s = settings(cfg) + client = self._signed_in(cfg) + _, crypto = self._sdk(cfg) + try: + return client.sync(force_refresh=True), client, crypto + except httpx.HTTPStatusError as e: + self.reset() + code = e.response.status_code + if code in (400, 401, 403): + raise Unavailable( + f"Vaultwarden at {s['url']} refused Lens's sign-in (HTTP {code}): check the API key and the account" + ) from None + raise Unavailable(f"Vaultwarden at {s['url']} answered HTTP {code}") from None + except httpx.HTTPError as e: + self.reset() + raise Unavailable(f"couldn't reach Vaultwarden at {s['url']}: {type(e).__name__}: {e}"[:300]) from None + + def _read_vault(self, cfg): + sync, client, crypto = self._sync(cfg) + s = settings(cfg) + decrypt = crypto.decrypt + try: + user_key = client.connect_token.user_key + folders = {str(_get(f, "id")): decrypt(_get(f, "name"), user_key).decode("utf-8") for f in sync.Folders if _get(f, "name")} + except Exception as e: # noqa: BLE001 - the HMAC check fails when the master password is wrong + self.reset() + raise Unavailable(f"Vaultwarden's items don't open: check the master password ({type(e).__name__})") from None + wanted = {fid for fid, name in folders.items() if name.strip().casefold() == s["folder"].casefold()} + org_keys: dict[str, bytes | None] = {} + + def org_key(oid): + if oid not in org_keys: + try: + private = client.connect_token.orgs_key + org = next(o for o in sync.Profile.Organizations if str(o.Id) == oid) + org_keys[oid] = decrypt(org.Key, private) + except Exception as e: # noqa: BLE001 - an organization Lens's account can't open: its items are skipped + log.warning("secrets: vaultwarden organization %s can't be opened: %s", oid, type(e).__name__) + org_keys[oid] = None + return org_keys[oid] + + items = [] + for raw in sync.Ciphers: + if _get(raw, "deletedDate") or str(_get(raw, "folderId") or "") not in wanted: + continue + key = org_key(str(_get(raw, "organizationId"))) if _get(raw, "organizationId") else user_key + if key is None: + continue + try: + if _get(raw, "key"): # an item with a key of its own, kept wrapped by the account's or organization's + key = decrypt(_get(raw, "key"), key) + items.append(_Item(raw, key, decrypt)) + except Exception as e: # noqa: BLE001 - one item that doesn't open doesn't hide the others + log.warning("secrets: a vaultwarden item in %s can't be opened: %s", s["folder"], type(e).__name__) + vault = _Vault(s["folder"], items) + with self._lock: + self._vault = (self.fingerprint(cfg), vault) + return vault + + # ---------- the provider ---------- + def read(self, cfg, refs): + vault = self._read_vault(cfg) + out = {} + for r in refs: + try: + out[r.text] = vault.value(r.item, r.field) + except Missing as e: + out[r.text] = e + return out + + def items(self, cfg, search=None, limit=200): + vault = self._read_vault(cfg) + q = (search or "").strip().casefold() + out = [] + for it in sorted(vault.items, key=lambda i: i.name.casefold()): + if q and q not in it.name.casefold(): + continue + default = DEFAULT_FIELD.get(it.type) + out.append( + { + "name": it.name, + "type": TYPES.get(it.type, "other"), + "fields": it.fields(), + "ref": reference(self.id, it.name) if default else None, + "refs": {f: reference(self.id, it.name, f) for f in it.fields()}, + } + ) + if len(out) >= limit: + break + return out + + def test(self, cfg): + s = settings(cfg) + if not self.configured(cfg): + return { + "ok": False, + "error": "set the address, the account's email, its API key (client id and secret) and its master password", + } + try: + vault = self._read_vault(cfg) + except Unavailable as e: + return {"ok": False, "error": str(e)} + return {"ok": True, "items": len(vault.items), "folder": s["folder"], "account": s["email"]} + + +PROVIDERS["vaultwarden"] = Vaultwarden() diff --git a/fastapi_backend/app/schemas/secrets.py b/fastapi_backend/app/schemas/secrets.py new file mode 100644 index 000000000..383f7d43e --- /dev/null +++ b/fastapi_backend/app/schemas/secrets.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +from typing import Any + +from pydantic import Field + +from app.schemas.common import RequestModel, ResponseModel + + +class SecretProvider(ResponseModel): + id: str + label: str + scheme: str | None = Field(None, description="how references to it start (vaultwarden://…); none for the built-in store") + configured: bool = Field(description="set up, so a reference to it can be read") + scope: str | None = Field(None, description="what it reads, in words: never a secret") + url: str | None = None + email: str | None = None + folder: str | None = None + installed: bool | None = Field(None, description="its client is fetched (a component), when it needs one") + + +class SecretReference(ResponseModel): + section: str + key: str + ref: str = Field(description="the reference the field holds, like vaultwarden://OpenAI/password") + provider: str + read: bool | None = Field(None, description="whether its value could be read by this server process; null before the first read") + error: str | None = Field(None, description="why it couldn't be read, last time it was tried") + read_at: str | None = Field(None, description="when its value was last read") + missing: bool = Field(False, description="the password manager answered that the item or field isn't there") + + +class SecretsView(ResponseModel): + providers: list[SecretProvider] + references: list[SecretReference] = Field(description="every secret field that refers to a password manager") + fields: list[dict[str, Any]] = Field(description="every secret field in Settings: section, key, set, and its ref when it holds one") + + +class VaultItem(ResponseModel): + name: str + type: str = Field(description="login, note, card, identity or other") + fields: list[str] = Field(description="the names of the fields that hold something") + ref: str | None = Field(None, description="the reference to the item's default field (a login's password, a note's text)") + refs: dict[str, str] = Field(default_factory=dict, description="the reference to each field") + + +class VaultItems(ResponseModel): + provider: str + items: list[VaultItem] + + +class SecretsTry(RequestModel): + provider: str = "vaultwarden" + + +class SecretsTestResult(ResponseModel): + ok: bool + error: str | None = None + items: int | None = Field(None, description="how many items the folder Lens reads holds") + folder: str | None = None + account: str | None = None diff --git a/fastapi_backend/pyproject.toml b/fastapi_backend/pyproject.toml index a7675df8f..0c17cb6d9 100644 --- a/fastapi_backend/pyproject.toml +++ b/fastapi_backend/pyproject.toml @@ -71,6 +71,9 @@ sandbox = ["openshell>=0.1.4,<0.2"] web = ["ddgs>=9.16,<10"] # Outlook .msg emails (extract-msg is GPL-3.0; .eml needs nothing) msg = ["extract-msg"] +# secrets read from a Vaultwarden or Bitwarden vault (vaultwarden_secrets.py): python-vaultwarden (Apache-2.0); fetched +# when Settings › Secrets has a Vaultwarden address +vaultwarden = ["python-vaultwarden>=1.1.2,<2"] [project.scripts] lens = "app.cli:main" @@ -100,6 +103,7 @@ dev = [ "impacket>=0.13", # the SMB share's server, and the SMB client its tests sign in with "openshell>=0.1.4,<0.2", # the code sandbox's client, tested against a fake OpenShell gateway "botocore>=1.43.113", + "python-vaultwarden>=1.1.2,<2", # the Vaultwarden client, tested against a fake Vaultwarden ] [tool.hatch.build.targets.wheel] diff --git a/fastapi_backend/tests/api/test_mcp.py b/fastapi_backend/tests/api/test_mcp.py index fd0027c52..2efb8dab6 100644 --- a/fastapi_backend/tests/api/test_mcp.py +++ b/fastapi_backend/tests/api/test_mcp.py @@ -92,6 +92,7 @@ def test_clients_are_sent_to_sign_in(client, app, env): "set_namespace_integrations", "set_picture_descriptions", "set_file_storage", + "set_secret_reference", "set_monitoring", "set_service", "set_sandbox_gateway", diff --git a/fastapi_backend/tests/api/test_secrets.py b/fastapi_backend/tests/api/test_secrets.py new file mode 100644 index 000000000..447eb167e --- /dev/null +++ b/fastapi_backend/tests/api/test_secrets.py @@ -0,0 +1,266 @@ +"""Secrets kept in a password manager (docs/secrets.md): a secret field in Settings holds a reference to a Vaultwarden +item instead of the secret, Lens reads the value from the vault and nobody, agents included, ever sees it.""" + +from __future__ import annotations + +import time + +import pytest + +from app.domain import secret_providers, settings +from tests import fake_vaultwarden as fv +from tests.api.test_mcp import tool, tool_error +from tests.helpers import login, make_user + +SETUP = { + "vaultwarden_email": fv.EMAIL, + "vaultwarden_client_id": fv.CLIENT_ID, + "vaultwarden_client_secret": fv.CLIENT_SECRET, + "vaultwarden_password": fv.PASSWORD, + "vaultwarden_folder": "Lens", +} + + +@pytest.fixture +def vault(): + secret_providers.forget() + v, url = fv.server() + v.add("OpenAI", username="billing@x.io", password="sk-from-the-vault", uri="https://api.openai.com", notes="the team key") + v.add("Brave", password="brave-key-1", fields={"Endpoint token": "ep-token-9"}) + v.add("Bank", folder="Personal", password="never-read") + v.add("Wiki", type=2, notes="a page of notes", own_key=True) + v.add("Card", type=3, number="4111 1111 1111 1111", code="123", holder="A. Person") + yield v, url + v.srv.shutdown() + secret_providers.forget() + + +def _admin(client, db): + make_user(db, "root@x.io", "root password 1", admin=True) + return login(client, "root@x.io", "root password 1") + + +def _set_up(client, h, url, **over): + r = client.put("/api/v1/settings/secrets", json={"vaultwarden_url": url, **SETUP, **over}, headers=h) + assert r.status_code == 200, r.text + + +def test_a_secret_field_reads_its_value_from_the_vault(app, client, db, vault): + v, url = vault + h = _admin(client, db) + # before a vault is set up, a reference can't be saved; the built-in store is the default and stays as it was + r = client.put("/api/v1/settings/llm", json={"api_key": "vaultwarden://OpenAI/password"}, headers=h) + assert r.status_code == 400 and "Settings › Secrets" in r.text + _set_up(client, h, url) + got = client.get("/api/v1/secrets", headers=h).json() + vw = next(p for p in got["providers"] if p["id"] == "vaultwarden") + assert (vw["configured"], vw["scheme"], vw["folder"], vw["email"]) == (True, "vaultwarden", "Lens", fv.EMAIL) + assert next(p for p in got["providers"] if p["id"] == "builtin")["configured"] is True + # the vault's own sign-in is shown as set, never its values, and never comes from a vault itself + view = client.get("/api/v1/settings", headers=h).json()["secrets"]["values"] + assert view["vaultwarden_password"] == {"secret": True, "set": True} and view["vaultwarden_client_id"] == fv.CLIENT_ID + assert fv.PASSWORD not in client.get("/api/v1/settings", headers=h).text + r = client.put("/api/v1/settings/secrets", json={"vaultwarden_password": "vaultwarden://Bank"}, headers=h) + assert r.status_code == 400 and "kept in Lens" in r.text + + # a reference in a secret field: checked against the vault when saved, then read when the settings are built + r = client.put( + "/api/v1/settings/llm", + json={"base_url": "http://127.0.0.1:1234/v1", "model": "m", "api_key": "vaultwarden://OpenAI/password"}, + headers=h, + ) + assert r.status_code == 200, r.text + assert app.state.settings.current()["llm"]["api_key"] == "sk-from-the-vault" + everything = client.get("/api/v1/settings", headers=h) + assert "sk-from-the-vault" not in everything.text + assert "sk-from-the-vault" not in str(db.rows("SELECT * FROM app_setting")) # only the reference is kept, sealed + shown = everything.json()["llm"]["values"]["api_key"] + assert (shown["secret"], shown["set"], shown["ref"], shown["read"], shown["provider"]) == ( + True, + True, + "vaultwarden://OpenAI/password", + True, + "vaultwarden", + ) + assert shown["read_at"] and shown["error"] is None + refs = client.get("/api/v1/secrets", headers=h).json()["references"] + assert [(r["section"], r["key"], r["ref"], r["read"]) for r in refs] == [("llm", "api_key", "vaultwarden://OpenAI/password", True)] + fields = client.get("/api/v1/secrets", headers=h).json()["fields"] + assert {"section": "llm", "key": "api_key", "set": True, "locked": False, "ref": "vaultwarden://OpenAI/password"} in fields + assert {"section": "web", "key": "api_key", "set": False, "locked": False} in fields + + # every field a reference can name: a login's fields, a custom field, a note's text, a card's number; the item + # alone is a login's password, a note's text or a card's number + for ref, value in ( + ("vaultwarden://OpenAI", "sk-from-the-vault"), + ("vaultwarden://openai/username", "billing@x.io"), + ("vaultwarden://OpenAI/uri", "https://api.openai.com"), + ("vaultwarden://OpenAI/notes", "the team key"), + ("vaultwarden://Brave/Endpoint token", "ep-token-9"), + ("vaultwarden://Wiki", "a page of notes"), + ("vaultwarden://Card", "4111 1111 1111 1111"), + ("vaultwarden://Card/code", "123"), + ): + assert client.put("/api/v1/settings/web", json={"api_key": ref}, headers=h).status_code == 200, ref + assert app.state.settings.current()["web"]["api_key"] == value, ref + # what can't be read is refused when it's saved, with why + for ref, why in ( + ("vaultwarden://Bank", "no item called Bank in the folder Lens"), # another folder: out of Lens's reach + ("vaultwarden://OpenAI/totp", "has no totp"), + ("vaultwarden://Nothing/password", "no item called Nothing"), + ("vaultwarden://", "doesn't name an item"), + ): + r = client.put("/api/v1/settings/web", json={"api_key": ref}, headers=h) + assert r.status_code == 400 and why in r.text, (ref, r.text) + assert app.state.settings.current()["web"]["api_key"] == "123" # the last one saved stands + # a plain value still works as before, sealed in Lens + assert client.put("/api/v1/settings/web", json={"api_key": "typed-in"}, headers=h).status_code == 200 + assert app.state.settings.current()["web"]["api_key"] == "typed-in" + assert "ref" not in client.get("/api/v1/settings", headers=h).json()["web"]["values"]["api_key"] + # the vault saw a sign-in with the API key, and syncs; the master password never went over the wire + assert [s["form"]["grant_type"] for s in v.seen if s["method"] == "POST"][0] == "client_credentials" + assert fv.PASSWORD not in str(v.seen) + # the audit log has the change, not the reference's value + audit = client.get("/api/v1/audit", headers=h).json() + assert ("settings.save", "secrets") in {(a["action"], a["target"]) for a in audit} + + +def test_the_vault_is_read_again_and_what_vanishes_stops_being_used(app, client, db, vault, monkeypatch): + v, url = vault + h = _admin(client, db) + _set_up(client, h, url, cache_minutes=1) + assert client.put("/api/v1/settings/llm", json={"api_key": "vaultwarden://Brave"}, headers=h).status_code == 200 + assert app.state.settings.current()["llm"]["api_key"] == "brave-key-1" + syncs = lambda: sum(s["path"].startswith("/api/sync") for s in v.seen) # noqa: E731 + before = syncs() + clock = time.time() + + def later(seconds): + """Move the clock on, let the settings notice, and wait for the read that starts in the background.""" + nonlocal clock + clock += seconds + monkeypatch.setattr(secret_providers, "_now", lambda: clock) + gen, n = secret_providers.generation(), syncs() + app.state.settings.current() + for _ in range(200): + if secret_providers.generation() != gen or (syncs() > n and not secret_providers._STATE["busy"]): + break + time.sleep(0.02) + return app.state.settings.current()["llm"]["api_key"] + + # the key is rotated in the vault: the old one serves until the cache runs out, then the new one is read, in the + # background, without anyone saving a setting + v.items[1]["login"]["password"] = v.enc("brave-key-2") + assert later(30) == "brave-key-1" + assert syncs() == before # not asked again before it's due + assert later(60) == "brave-key-2" + # the item is moved out of the folder: at the next read Lens stops using it and says so + v.items[1]["folderId"] = v.folders["Personal"] + assert later(90) is None + shown = client.get("/api/v1/settings", headers=h).json()["llm"]["values"]["api_key"] + assert shown["read"] is False and "no item called Brave" in shown["error"] and shown["missing"] is True + # back in the folder: read again once the retry is due + v.items[1]["folderId"] = v.folders["Lens"] + assert later(secret_providers.RETRY_SECONDS + 1) == "brave-key-2" + # the vault goes away: what was read before keeps serving, and the status says why it couldn't be read again + v.refuse = 500 + secret_providers.PROVIDERS["vaultwarden"].reset() # as if the process had to sign in again + assert later(90) == "brave-key-2" + shown = client.get("/api/v1/settings", headers=h).json()["llm"]["values"]["api_key"] + assert shown["read"] is True and "HTTP 500" in shown["error"] and shown["missing"] is False + + +def test_the_vault_is_set_up_and_checked_in_the_app(app, client, db, vault): + v, url = vault + h = _admin(client, db) + # not set up yet: the check says what's missing, and the items can't be listed + assert client.post("/api/v1/secrets/test", json={"provider": "vaultwarden"}, headers=h).json()["ok"] is False + assert client.get("/api/v1/secrets/items", headers=h).status_code == 409 + for bad in ( + {"vaultwarden_url": "ftp://vault"}, + {"vaultwarden_url": "https://user:pw@vault.example"}, + {"vaultwarden_email": "not an address"}, + {"vaultwarden_folder": ""}, + {"cache_minutes": 0}, + ): + assert client.put("/api/v1/settings/secrets", json=bad, headers=h).status_code == 400, bad + _set_up(client, h, url, vaultwarden_password="wrong") + r = client.post("/api/v1/secrets/test", headers=h).json() + assert r["ok"] is False and "master password" in r["error"] + _set_up(client, h, url, vaultwarden_client_secret="wrong") + r = client.post("/api/v1/secrets/test", headers=h).json() + assert r["ok"] is False and "refused Lens's sign-in" in r["error"] + _set_up(client, h, "http://127.0.0.1:9") + r = client.post("/api/v1/secrets/test", headers=h).json() + assert r["ok"] is False and "couldn't reach" in r["error"] + _set_up(client, h, url) + assert client.post("/api/v1/secrets/test", headers=h).json() == { + "ok": True, + "items": 4, + "folder": "Lens", + "account": fv.EMAIL, + "error": None, + } + # the items Lens may refer to: names, fields and references, never a value + r = client.get("/api/v1/secrets/items", headers=h) + assert r.status_code == 200, r.text + items = {i["name"]: i for i in r.json()["items"]} + assert list(items) == ["Brave", "Card", "OpenAI", "Wiki"] # Bank is in another folder + assert items["OpenAI"]["type"] == "login" and items["OpenAI"]["fields"] == ["username", "password", "uri", "notes"] + assert items["OpenAI"]["ref"] == "vaultwarden://OpenAI" and items["OpenAI"]["refs"]["username"] == "vaultwarden://OpenAI/username" + assert items["Brave"]["refs"]["Endpoint token"] == "vaultwarden://Brave/Endpoint token" + assert items["Wiki"]["type"] == "note" and items["Wiki"]["ref"] == "vaultwarden://Wiki" + assert items["Card"]["type"] == "card" and items["Card"]["fields"] == ["cardholderName", "number", "code"] + for secret in ("sk-from-the-vault", "brave-key-1", "ep-token-9", "4111", "a page of notes", "billing@x.io"): + assert secret not in r.text + assert [i["name"] for i in client.get("/api/v1/secrets/items", params={"q": "ope"}, headers=h).json()["items"]] == ["OpenAI"] + # the folder is the scope: another one, and Bank is what Lens sees + assert client.put("/api/v1/settings/secrets", json={"vaultwarden_folder": "Personal"}, headers=h).status_code == 200 + assert [i["name"] for i in client.get("/api/v1/secrets/items", headers=h).json()["items"]] == ["Bank"] + # admins only + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + hv = login(client, "vi@x.io", "viewer password 1") + assert client.get("/api/v1/secrets", headers=hv).status_code == 403 + assert client.get("/api/v1/secrets/items", headers=hv).status_code == 403 + assert client.post("/api/v1/secrets/test", headers=hv).status_code == 403 + + +def test_agents_refer_to_secrets_and_never_read_them(app, client, db, vault): + v, url = vault + h = _admin(client, db) + _set_up(client, h, url) + make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"}) + hv = login(client, "vi@x.io", "viewer password 1") + assert "only admins" in tool_error(client, hv, "list_secrets") + assert "only admins" in tool_error(client, hv, "list_vault_items") + got = tool(client, h, "list_vault_items", q="brave") + assert [i["name"] for i in got["items"]] == ["Brave"] and "brave-key-1" not in str(got) + # pointing a field at the vault: checked first, and the reference is what's saved + assert "isn't a secret field" in tool_error(client, h, "set_secret_reference", section="llm", key="model", ref="vaultwarden://OpenAI") + assert "a value never passes" in tool_error(client, h, "set_secret_reference", section="llm", key="api_key", ref="sk-typed") + assert "no item called Nope" in tool_error(client, h, "set_secret_reference", section="llm", key="api_key", ref="vaultwarden://Nope") + out = tool(client, h, "set_secret_reference", section="llm", key="api_key", ref="vaultwarden://OpenAI") + assert out["saved"] == {"section": "llm", "key": "api_key", "ref": "vaultwarden://OpenAI"} and out["status"]["read"] is True + assert app.state.settings.current()["llm"]["api_key"] == "sk-from-the-vault" + listed = tool(client, h, "list_secrets") + assert [(r["section"], r["key"], r["ref"]) for r in listed["references"]] == [("llm", "api_key", "vaultwarden://OpenAI")] + assert "sk-from-the-vault" not in str(listed) and fv.PASSWORD not in str(listed) + # a read-only token can list, not point + tok = client.post("/api/v1/tokens", json={"name": "ro", "scope": "read"}, headers=h).json()["token"] + ro = {"Authorization": f"Bearer {tok}"} + assert tool(client, ro, "list_secrets")["references"] + assert "read-only" in tool_error(client, ro, "set_secret_reference", section="llm", key="api_key", ref="vaultwarden://Brave") + # the environment wins over a reference, as over any saved value + assert ("llm", "api_key") in settings.ENV_OVERRIDES + + +def test_references_are_parsed_and_written_the_same_way(): + r = secret_providers.parse("vaultwarden://My item/Endpoint token") + assert (r.provider, r.item, r.field) == ("vaultwarden", "My item", "Endpoint token") + assert secret_providers.parse("VAULTWARDEN://x").provider == "vaultwarden" + assert secret_providers.reference("vaultwarden", "a/b", "c%d") == "vaultwarden://a%2Fb/c%25d" + assert secret_providers.parse(secret_providers.reference("vaultwarden", "a/b", "c%d")).item == "a/b" + assert secret_providers.parse(secret_providers.reference("vaultwarden", "a/b", "c%d")).field == "c%d" + for plain in ("sk-abc", "http://not-a-provider/x", "nothing://x", None, 3, ""): + assert secret_providers.parse(plain) is None + assert secret_providers.parse("vaultwarden://OpenAI").field is None diff --git a/fastapi_backend/tests/fake_vaultwarden.py b/fastapi_backend/tests/fake_vaultwarden.py new file mode 100644 index 000000000..83e886eaa --- /dev/null +++ b/fastapi_backend/tests/fake_vaultwarden.py @@ -0,0 +1,177 @@ +"""A Vaultwarden server for tests (vaultwarden_secrets.py), on loopback, encrypting its items the way Bitwarden does: +``POST /identity/connect/token`` signs an API key in (client_credentials) and hands over the account's user key +wrapped by the master key, and ``GET /api/sync`` answers the folders and items, each field encrypted with the user key +(or an item's own key, for one that has one). The library's own crypto (python-vaultwarden) encrypts here, so the test +proves Lens decrypts what a real vault sends. Every request is kept in `seen`.""" + +from __future__ import annotations + +import http.server +import json +import secrets +import threading +import urllib.parse +import uuid + +from vaultwarden.models.bitwarden import Kdf, KdfType +from vaultwarden.utils.crypto import encrypt_sym, make_master_key + +EMAIL = "lens@vault.example" +PASSWORD = "correct horse battery staple" +CLIENT_ID = "user.3f4c8f3e-1111-4f5e-9d2a-0a1b2c3d4e5f" +CLIENT_SECRET = "ApiKeySecretForTests0123456789" +ITERATIONS = 5000 # Bitwarden uses 600000; the test only needs the same number on both sides + + +class Vault: + """The vault's contents: folders by name, and items (type 1 login, 2 note, 3 card) in folders.""" + + def __init__(self): + self.user_key = secrets.token_bytes(64) + self.master_key = make_master_key(PASSWORD, EMAIL, Kdf(KdfType.Pbkdf2, ITERATIONS)) + self.folders = {"Lens": str(uuid.uuid4()), "Personal": str(uuid.uuid4())} + self.items = [] + self.seen = [] + self.refuse = None # an HTTP status the token endpoint answers with instead + + def enc(self, text, key=None): + return encrypt_sym(text, key or self.user_key) if text is not None else None + + def add(self, name, folder="Lens", type=1, own_key=False, fields=None, notes=None, **login): + """A login (username, password, totp, uri), note or card in a folder; own_key wraps it with a key of its own.""" + key = secrets.token_bytes(64) if own_key else None + + def e(t): + return self.enc(t, key) + + item = { + "object": "cipherDetails", + "id": str(uuid.uuid4()), + "organizationId": None, + "folderId": self.folders[folder] if folder else None, + "type": type, + "name": e(name), + "notes": e(notes), + "favorite": False, + "reprompt": 0, + "revisionDate": "2026-10-17T09:00:00Z", + "deletedDate": None, + "key": self.enc(key) if key else None, + "fields": [{"name": e(n), "value": e(v), "type": 1, "linkedId": None} for n, v in (fields or {}).items()], + "collectionIds": [], + "passwordHistory": None, + "attachments": None, + } + if type == 1: + item["login"] = { + "username": e(login.get("username")), + "password": e(login.get("password")), + "totp": e(login.get("totp")), + "uris": [{"uri": e(login["uri"]), "match": None}] if login.get("uri") else None, + "passwordRevisionDate": None, + } + elif type == 2: + item["secureNote"] = {"type": 0} + elif type == 3: + item["card"] = {"cardholderName": e(login.get("holder")), "number": e(login.get("number")), "code": e(login.get("code"))} + self.items.append(item) + return item + + def token(self): + return { + "access_token": "token-" + secrets.token_hex(8), + "expires_in": 3600, + "token_type": "Bearer", + "refresh_token": "refresh-" + secrets.token_hex(8), + "scope": "api", + "Kdf": 0, + "KdfIterations": ITERATIONS, + "KdfMemory": None, + "KdfParallelism": None, + "Key": self.enc(self.user_key, self.master_key), + "PrivateKey": self.enc(b"not a real key: no organizations here"), + "ForcePasswordReset": False, + "ResetMasterPassword": False, + "unofficialServer": True, + } + + def sync(self): + return { + "object": "sync", + "profile": { + "object": "profile", + "id": str(uuid.uuid4()), + "name": "Lens", + "email": EMAIL, + "emailVerified": True, + "premium": False, + "masterPasswordHint": None, + "culture": "en-US", + "twoFactorEnabled": False, + "key": self.enc(self.user_key, self.master_key), + "privateKey": self.enc(b"not a real key"), + "securityStamp": "stamp", + "organizations": [], + "providers": [], + "providerOrganizations": [], + "forcePasswordReset": False, + "avatarColor": None, + "_status": 0, + }, + "folders": [ + {"object": "folder", "id": fid, "name": self.enc(name), "revisionDate": "2026-10-17T09:00:00Z"} + for name, fid in self.folders.items() + ], + "collections": [], + "ciphers": self.items, + "domains": None, + "policies": [], + "sends": [], + } + + +class Handler(http.server.BaseHTTPRequestHandler): + vault: Vault + + def log_message(self, *args): + pass + + def _send(self, code, body): + raw = json.dumps(body).encode() + self.send_response(code) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + def do_POST(self): # noqa: N802 + raw = self.rfile.read(int(self.headers.get("Content-Length") or 0)).decode() + form = dict(urllib.parse.parse_qsl(raw)) + self.vault.seen.append({"method": "POST", "path": self.path, "form": {k: v for k, v in form.items() if k != "client_secret"}}) + if self.path != "/identity/connect/token": + return self._send(404, {"error": "not found"}) + if self.vault.refuse: + return self._send(self.vault.refuse, {"error": "refused"}) + if form.get("grant_type") == "refresh_token": + return self._send(200, self.vault.token()) + if form.get("client_id") != CLIENT_ID or form.get("client_secret") != CLIENT_SECRET: + return self._send(400, {"error": "invalid_client", "ErrorModel": {"Message": "client_id or client_secret is incorrect"}}) + return self._send(200, self.vault.token()) + + def do_GET(self): # noqa: N802 + self.vault.seen.append({"method": "GET", "path": self.path, "auth": bool(self.headers.get("Authorization"))}) + if not (self.headers.get("Authorization") or "").startswith("Bearer token-"): + return self._send(401, {"error": "no"}) + if self.path.startswith("/api/sync"): + return self._send(200, self.vault.sync()) + return self._send(404, {"error": "not found"}) + + +def server(): + """A Vaultwarden server on loopback: (vault, its address). Stop it with vault.srv.shutdown().""" + vault = Vault() + handler = type("VaultHandler", (Handler,), {"vault": vault}) + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler) + threading.Thread(target=srv.serve_forever, daemon=True).start() + vault.srv = srv + return vault, f"http://127.0.0.1:{srv.server_address[1]}" diff --git a/fastapi_backend/uv.lock b/fastapi_backend/uv.lock index 9842acede..729189c88 100644 --- a/fastapi_backend/uv.lock +++ b/fastapi_backend/uv.lock @@ -193,6 +193,40 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079, upload-time = "2026-09-05T10:42:37.923Z" }, ] +[[package]] +name = "argon2-cffi" +version = "25.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "argon2-cffi-bindings" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/89/ce5af8a7d472a67cc819d5d998aa8c82c5d860608c4db9f46f1162d7dab9/argon2_cffi-25.1.0.tar.gz", hash = "sha256:694ae5cc8a42f4c4e2bf2ca0e64e51e23a040c6a517a85074683d3959e1346c1", size = 45706, upload-time = "2025-06-03T06:55:32.073Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4f/d3/a8b22fa575b297cd6e3e3b0155c7e25db170edf1c74783d6a31a2490b8d9/argon2_cffi-25.1.0-py3-none-any.whl", hash = "sha256:fdc8b074db390fccb6eb4a3604ae7231f219aa669a2652e0f20e16ba513d5741", size = 14657, upload-time = "2025-06-03T06:55:30.804Z" }, +] + +[[package]] +name = "argon2-cffi-bindings" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0b/43/bb8b6e8708d49a5ab36781333af092d9f483b198a2710d01281204640055/argon2_cffi_bindings-26.1.0.tar.gz", hash = "sha256:63505c71542a44b68b1e38060450fb006404170da375feb31af153e7f9c6205d", size = 1790807, upload-time = "2026-08-20T07:44:22.492Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/d2/0ae991f1b2181e5be49007c574710a800ad36c2978683addb3e67c474e55/argon2_cffi_bindings-26.1.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:21ca0396fe5ec995dd54431c32698189666f9224810acfa752e50d2bd94d9df2", size = 25521, upload-time = "2026-08-20T07:32:43.019Z" }, + { url = "https://files.pythonhosted.org/packages/7e/e4/ad91d8297638aa2258aad4501c306aca99480dfe76ccd638173fa3702db9/argon2_cffi_bindings-26.1.0-cp310-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78de2d65e0b9ea7ce9d1b1c3e87297b2d7305a02c266ee2a2d6910daddd7ee69", size = 27177, upload-time = "2026-08-20T07:32:44.158Z" }, + { url = "https://files.pythonhosted.org/packages/6f/86/5363df11b86d02cf3662208e7406496327649cc90eb365bf6f4e8a54a41f/argon2_cffi_bindings-26.1.0-cp310-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:27f1821903e2ceadcb88ec2b45ef190897b7682449c772f4d9b53e42c520cf29", size = 26597, upload-time = "2026-08-20T07:32:45.172Z" }, + { url = "https://files.pythonhosted.org/packages/f4/b5/a14dcc592652347dad23ee93b278a4da5d2a25c9ed3ebd10d68eea823a4f/argon2_cffi_bindings-26.1.0-cp310-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d88e5f7e60f28ae0b0cc6b2f16c43e87cd642a196a86f85e0d8bb6fe016fc16d", size = 27403, upload-time = "2026-08-20T07:32:46.13Z" }, + { url = "https://files.pythonhosted.org/packages/b3/81/b4a20d4902af7f796390bf9245ff83c5217dfa7367efa1d14986956c482b/argon2_cffi_bindings-26.1.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:34b7d9c24a4165a2c61cc8ae11d44d48c9ce2830fb536cb7914e11fdd9962728", size = 27132, upload-time = "2026-08-20T07:32:47.13Z" }, + { url = "https://files.pythonhosted.org/packages/7e/1b/c8de358af07b1c490e0fcb863ef98e46ddb486e45567aca5a60bd68d9daa/argon2_cffi_bindings-26.1.0-cp310-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:224865cbbcb7a2bd1356741dff12b0134df726b6d44bb7b500df8e303cbd9e81", size = 27588, upload-time = "2026-08-20T07:32:48.087Z" }, + { url = "https://files.pythonhosted.org/packages/48/2f/7ee62a6e79f9309f9d9982d301b22a00010adb580c05c8109b94d7b33de0/argon2_cffi_bindings-26.1.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:ffff613aaa9ce6236766e2fc6dc560bb5abde7a2e2416e3db1f9ae395a2b4dd4", size = 26785, upload-time = "2026-08-20T07:32:48.977Z" }, + { url = "https://files.pythonhosted.org/packages/e9/10/960d0ee93d4897741bcaf4799c697dae2d81499f66fd1ed042a7dd54c1f4/argon2_cffi_bindings-26.1.0-cp310-abi3-win32.whl", hash = "sha256:a86c069c91a747a2c4e5c51473590aeb48172fff9b2130d23729a42d98665ecb", size = 23898, upload-time = "2026-08-20T07:32:50.114Z" }, + { url = "https://files.pythonhosted.org/packages/6d/3a/0cc14a05810e6add9bce5e87693334baa2222de5f647fa31781885b6573f/argon2_cffi_bindings-26.1.0-cp310-abi3-win_amd64.whl", hash = "sha256:2c36ff87b5dfaa477d0bd51e9d7f6abdae7c8955d2983c97419085d842154b3e", size = 25730, upload-time = "2026-08-20T07:32:51.091Z" }, + { url = "https://files.pythonhosted.org/packages/4e/db/d83cf2af140547f0b9cdaece05b2dc2dcbf991be4667331d073eff771435/argon2_cffi_bindings-26.1.0-cp310-abi3-win_arm64.whl", hash = "sha256:f9c4420a7a864fe1b86ce35befc95b8e39fb852493b81cf798671ddc265de638", size = 24478, upload-time = "2026-08-20T07:32:52.111Z" }, + { url = "https://files.pythonhosted.org/packages/bb/5f/f652055e18d2627e2eed94c7f31a792127cfe38df786635395d742321674/argon2_cffi_bindings-26.1.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:af11ac37a7c53dc16cb7950a6190851b0870fe218b6c60c0bb7ac355234e3083", size = 15434, upload-time = "2026-08-20T07:32:53.143Z" }, +] + [[package]] name = "ast-serialize" version = "0.12.1" @@ -1473,6 +1507,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/73/63/ca511b6f802f28cf3489b280fe77475bcca8de85e81a6299d7916b5b5555/hf_xet-1.6.0-cp38-abi3-win_arm64.whl", hash = "sha256:3dc3e35441ba395006af5aaacc40ef2e603c51ef46c3530b9156185f00935ea3", size = 3859359, upload-time = "2026-08-03T22:33:11.725Z" }, ] +[[package]] +name = "hkdf" +version = "0.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/be/327e072850db181ce56afd51e26ec7aa5659b18466c709fa5ea2548c935f/hkdf-0.0.3.tar.gz", hash = "sha256:622a31c634bc185581530a4b44ffb731ed208acf4614f9c795bdd70e77991dca", size = 3959, upload-time = "2015-06-16T03:55:40.044Z" } + [[package]] name = "httpcore" version = "1.0.9" @@ -2014,6 +2054,9 @@ spacy = [ validate = [ { name = "jsonschema" }, ] +vaultwarden = [ + { name = "python-vaultwarden" }, +] voices = [ { name = "speechbrain" }, { name = "torch" }, @@ -2046,6 +2089,7 @@ dev = [ { name = "pytest-mock" }, { name = "pytest-xdist" }, { name = "python-dotenv" }, + { name = "python-vaultwarden" }, { name = "ruff" }, { name = "tokenizers" }, { name = "types-pyyaml" }, @@ -2087,6 +2131,7 @@ requires-dist = [ { name = "pyobjc-framework-vision", marker = "sys_platform == 'darwin' and extra == 'mac-ocr'" }, { name = "pypdf", specifier = ">=6.20.1" }, { name = "python-doctr", marker = "extra == 'doctr'", specifier = ">=1.0" }, + { name = "python-vaultwarden", marker = "extra == 'vaultwarden'", specifier = ">=1.1.2,<2" }, { name = "pyyaml", specifier = ">=6" }, { name = "rapidocr-onnxruntime", marker = "extra == 'rapidocr'" }, { name = "rdflib", specifier = ">=7.1,<8" }, @@ -2108,7 +2153,7 @@ requires-dist = [ { name = "transformers", marker = "extra == 'punctuation'" }, { name = "webauthn", specifier = ">=3.0.1,<4" }, ] -provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "spacy", "injection", "florence", "moondream", "validate", "smb", "sandbox", "web", "msg"] +provides-extras = ["sensevoice", "whisper", "mlx", "laya", "voices", "pyannote", "faces", "mac-ocr", "rapidocr", "doctr", "insightface", "objects", "punctuation", "spacy", "injection", "florence", "moondream", "validate", "smb", "sandbox", "web", "msg", "vaultwarden"] [package.metadata.requires-dev] dev = [ @@ -2131,6 +2176,7 @@ dev = [ { name = "pytest-mock", specifier = ">=3.14.0,<4" }, { name = "pytest-xdist", specifier = ">=3.6,<4" }, { name = "python-dotenv", specifier = ">=1.2.4,<2" }, + { name = "python-vaultwarden", specifier = ">=1.1.2,<2" }, { name = "ruff", specifier = ">=0.17.0,<0.18" }, { name = "tokenizers", specifier = ">=0.23.3" }, { name = "types-pyyaml" }, @@ -4257,6 +4303,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" }, ] +[[package]] +name = "python-vaultwarden" +version = "1.1.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "argon2-cffi" }, + { name = "hkdf" }, + { name = "httpx" }, + { name = "pycryptodome" }, + { name = "pydantic" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/30/cb/7d9369ee50fbba4db7950306dce15e75ea96d4fe59b894efecebc6db619c/python_vaultwarden-1.1.2.tar.gz", hash = "sha256:6dd39d900f03b623454acc5fc9975c8749cd4e246c2ba4a604c5661b3ad6937e", size = 20783, upload-time = "2026-05-27T20:10:19.412Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f9/96/27cfee4f64c33cf9ae4dba4945b4bbb474818c8dcbba5d467938810d24e6/python_vaultwarden-1.1.2-py3-none-any.whl", hash = "sha256:5fc88fb9bd5e5fdea1426b60026ca3dc7c1540adc75c822ae3744a13926f38db", size = 23272, upload-time = "2026-05-27T20:10:18.38Z" }, +] + [[package]] name = "pytorch-lightning" version = "2.6.6" diff --git a/mkdocs.yml b/mkdocs.yml index 6fd7e1338..7b338ea58 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -108,6 +108,7 @@ nav: - Budgets: budgets.md - Operate: - Encryption: encryption.md + - Secrets: secrets.md - Storage: storage.md - Database: database.md - Telemetry: telemetry.md diff --git a/nextjs-frontend/app/openapi-client/index.ts b/nextjs-frontend/app/openapi-client/index.ts index 123eda165..6d9399b3c 100644 --- a/nextjs-frontend/app/openapi-client/index.ts +++ b/nextjs-frontend/app/openapi-client/index.ts @@ -1452,6 +1452,9 @@ export type { GetSandboxGatewayData, GetSandboxGatewayResponse, GetSandboxGatewayResponses, + GetSecretsData, + GetSecretsResponse, + GetSecretsResponses, GetSensorData, GetSensorError, GetSensorErrors, @@ -2134,6 +2137,11 @@ export type { ListUsersData, ListUsersResponse, ListUsersResponses, + ListVaultItemsData, + ListVaultItemsError, + ListVaultItemsErrors, + ListVaultItemsResponse, + ListVaultItemsResponses, ListViewsData, ListViewsResponse, ListViewsResponses, @@ -3113,6 +3121,11 @@ export type { SearchTranscriptsResponses, SearchUpdate, SecretFlag, + SecretProvider, + SecretReference, + SecretsTestResult, + SecretsTry, + SecretsView, SegmentEdit, SegmentSplit, SegmentUpdate, @@ -3475,6 +3488,11 @@ export type { TestNotifyTargetErrors, TestNotifyTargetResponse, TestNotifyTargetResponses, + TestSecretsData, + TestSecretsError, + TestSecretsErrors, + TestSecretsResponse, + TestSecretsResponses, TestSlackData, TestSlackResponse, TestSlackResponses, @@ -3872,6 +3890,8 @@ export type { VariableCatalogResponses, VariableEntry, VaultAnswer, + VaultItem, + VaultItems, VaultOptionsData, VaultOptionsError, VaultOptionsErrors, diff --git a/nextjs-frontend/app/openapi-client/sdk.gen.ts b/nextjs-frontend/app/openapi-client/sdk.gen.ts index 01d25d4b5..f5368d49a 100644 --- a/nextjs-frontend/app/openapi-client/sdk.gen.ts +++ b/nextjs-frontend/app/openapi-client/sdk.gen.ts @@ -703,6 +703,8 @@ import type { GetRunResponses, GetSandboxGatewayData, GetSandboxGatewayResponses, + GetSecretsData, + GetSecretsResponses, GetSensorData, GetSensorErrors, GetSensorResponses, @@ -1076,6 +1078,9 @@ import type { ListUploadsResponses, ListUsersData, ListUsersResponses, + ListVaultItemsData, + ListVaultItemsErrors, + ListVaultItemsResponses, ListViewsData, ListViewsResponses, ListVoicesData, @@ -1657,6 +1662,9 @@ import type { TestNotifyTargetData, TestNotifyTargetErrors, TestNotifyTargetResponses, + TestSecretsData, + TestSecretsErrors, + TestSecretsResponses, TestSlackData, TestSlackResponses, TestSourceData, @@ -3583,6 +3591,56 @@ export class Admin { }); } + /** + * Get Secrets + * + * The password managers Lens can read secrets from and whether each is set up; every secret field in Settings, + * whether it's set, and for one that refers to a password manager its reference and whether this server could read + * its value (and when, or why not). + */ + public static getSecrets( + options?: Options, + ): RequestResult { + return (options?.client ?? client).get({ + url: "/api/v1/secrets", + ...options, + }); + } + + /** + * List Vault Items + * + * The items of a password manager Lens may refer to (those in its folder): each one's name, type, the fields that + * hold something and the reference to each. Never their values. 409 when the manager isn't set up or can't be + * reached, with why. + */ + public static listVaultItems( + options?: Options, + ): RequestResult { + return (options?.client ?? client).get({ + url: "/api/v1/secrets/items", + ...options, + }); + } + + /** + * Test Secrets + * + * Sign in to a password manager with the settings saved and count the items Lens may read, to check them. + */ + public static testSecrets( + options?: Options, + ): RequestResult { + return (options?.client ?? client).post({ + url: "/api/v1/secrets/test", + ...options, + headers: { + "Content-Type": "application/json", + ...options?.headers, + }, + }); + } + /** * List Components * diff --git a/nextjs-frontend/app/openapi-client/types.gen.ts b/nextjs-frontend/app/openapi-client/types.gen.ts index 51de46f24..cd2bd2ac2 100644 --- a/nextjs-frontend/app/openapi-client/types.gen.ts +++ b/nextjs-frontend/app/openapi-client/types.gen.ts @@ -20209,6 +20209,170 @@ export type SecretFlag = { [key: string]: unknown; }; +/** + * SecretProvider + */ +export type SecretProvider = { + /** + * Id + */ + id: string; + /** + * Label + */ + label: string; + /** + * Scheme + * + * how references to it start (vaultwarden://…); none for the built-in store + */ + scheme?: string | null; + /** + * Configured + * + * set up, so a reference to it can be read + */ + configured: boolean; + /** + * Scope + * + * what it reads, in words: never a secret + */ + scope?: string | null; + /** + * Url + */ + url?: string | null; + /** + * Email + */ + email?: string | null; + /** + * Folder + */ + folder?: string | null; + /** + * Installed + * + * its client is fetched (a component), when it needs one + */ + installed?: boolean | null; + [key: string]: unknown; +}; + +/** + * SecretReference + */ +export type SecretReference = { + /** + * Section + */ + section: string; + /** + * Key + */ + key: string; + /** + * Ref + * + * the reference the field holds, like vaultwarden://OpenAI/password + */ + ref: string; + /** + * Provider + */ + provider: string; + /** + * Read + * + * whether its value could be read by this server process; null before the first read + */ + read?: boolean | null; + /** + * Error + * + * why it couldn't be read, last time it was tried + */ + error?: string | null; + /** + * Read At + * + * when its value was last read + */ + read_at?: string | null; + /** + * Missing + * + * the password manager answered that the item or field isn't there + */ + missing?: boolean; + [key: string]: unknown; +}; + +/** + * SecretsTestResult + */ +export type SecretsTestResult = { + /** + * Ok + */ + ok: boolean; + /** + * Error + */ + error?: string | null; + /** + * Items + * + * how many items the folder Lens reads holds + */ + items?: number | null; + /** + * Folder + */ + folder?: string | null; + /** + * Account + */ + account?: string | null; + [key: string]: unknown; +}; + +/** + * SecretsTry + */ +export type SecretsTry = { + /** + * Provider + */ + provider?: string; +}; + +/** + * SecretsView + */ +export type SecretsView = { + /** + * Providers + */ + providers: Array; + /** + * References + * + * every secret field that refers to a password manager + */ + references: Array; + /** + * Fields + * + * every secret field in Settings: section, key, set, and its ref when it holds one + */ + fields: Array<{ + [key: string]: unknown; + }>; + [key: string]: unknown; +}; + /** * SegmentEdit */ @@ -24862,6 +25026,58 @@ export type VaultAnswer = { prf: string; }; +/** + * VaultItem + */ +export type VaultItem = { + /** + * Name + */ + name: string; + /** + * Type + * + * login, note, card, identity or other + */ + type: string; + /** + * Fields + * + * the names of the fields that hold something + */ + fields: Array; + /** + * Ref + * + * the reference to the item's default field (a login's password, a note's text) + */ + ref?: string | null; + /** + * Refs + * + * the reference to each field + */ + refs?: { + [key: string]: string; + }; + [key: string]: unknown; +}; + +/** + * VaultItems + */ +export type VaultItems = { + /** + * Provider + */ + provider: string; + /** + * Items + */ + items: Array; + [key: string]: unknown; +}; + /** * VaultPasskey * @@ -28506,6 +28722,92 @@ export type ReindexSearchResponses = { export type ReindexSearchResponse = ReindexSearchResponses[keyof ReindexSearchResponses]; +export type GetSecretsData = { + body?: never; + path?: never; + query?: never; + url: "/api/v1/secrets"; +}; + +export type GetSecretsResponses = { + /** + * Successful Response + */ + 200: SecretsView; +}; + +export type GetSecretsResponse = GetSecretsResponses[keyof GetSecretsResponses]; + +export type ListVaultItemsData = { + body?: never; + path?: never; + query?: { + /** + * Provider + * + * the password manager + */ + provider?: string; + /** + * Q + * + * part of an item's name + */ + q?: string | null; + /** + * Limit + */ + limit?: number; + }; + url: "/api/v1/secrets/items"; +}; + +export type ListVaultItemsErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type ListVaultItemsError = ListVaultItemsErrors[keyof ListVaultItemsErrors]; + +export type ListVaultItemsResponses = { + /** + * Successful Response + */ + 200: VaultItems; +}; + +export type ListVaultItemsResponse = ListVaultItemsResponses[keyof ListVaultItemsResponses]; + +export type TestSecretsData = { + /** + * Body + */ + body?: SecretsTry | null; + path?: never; + query?: never; + url: "/api/v1/secrets/test"; +}; + +export type TestSecretsErrors = { + /** + * Validation Error + */ + 422: HttpValidationError; +}; + +export type TestSecretsError = TestSecretsErrors[keyof TestSecretsErrors]; + +export type TestSecretsResponses = { + /** + * Successful Response + */ + 200: SecretsTestResult; +}; + +export type TestSecretsResponse = TestSecretsResponses[keyof TestSecretsResponses]; + export type RemoteBackupsData = { body?: never; path?: never; diff --git a/nextjs-frontend/openapi.json b/nextjs-frontend/openapi.json index dcfb349c5..06427ac17 100644 --- a/nextjs-frontend/openapi.json +++ b/nextjs-frontend/openapi.json @@ -3886,6 +3886,154 @@ } } }, + "/api/v1/secrets": { + "get": { + "tags": [ + "admin" + ], + "summary": "Get Secrets", + "description": "The password managers Lens can read secrets from and whether each is set up; every secret field in Settings,\nwhether it's set, and for one that refers to a password manager its reference and whether this server could read\nits value (and when, or why not).", + "operationId": "get_secrets", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SecretsView" + } + } + } + } + } + } + }, + "/api/v1/secrets/items": { + "get": { + "tags": [ + "admin" + ], + "summary": "List Vault Items", + "description": "The items of a password manager Lens may refer to (those in its folder): each one's name, type, the fields that\nhold something and the reference to each. Never their values. 409 when the manager isn't set up or can't be\nreached, with why.", + "operationId": "list_vault_items", + "parameters": [ + { + "name": "provider", + "in": "query", + "required": false, + "schema": { + "type": "string", + "description": "the password manager", + "default": "vaultwarden", + "title": "Provider" + }, + "description": "the password manager" + }, + { + "name": "q", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "maxLength": 200 + }, + { + "type": "null" + } + ], + "description": "part of an item's name", + "title": "Q" + }, + "description": "part of an item's name" + }, + { + "name": "limit", + "in": "query", + "required": false, + "schema": { + "type": "integer", + "maximum": 1000, + "minimum": 1, + "default": 200, + "title": "Limit" + } + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/VaultItems" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v1/secrets/test": { + "post": { + "tags": [ + "admin" + ], + "summary": "Test Secrets", + "description": "Sign in to a password manager with the settings saved and count the items Lens may read, to check them.", + "operationId": "test_secrets", + "requestBody": { + "content": { + "application/json": { + "schema": { + "anyOf": [ + { + "$ref": "#/components/schemas/SecretsTry" + }, + { + "type": "null" + } + ], + "title": "Body" + } + } + } + }, + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SecretsTestResult" + } + } + } + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/admin/backups/remote": { "get": { "tags": [ @@ -67348,6 +67496,279 @@ ], "title": "SecretFlag" }, + "SecretProvider": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "label": { + "type": "string", + "title": "Label" + }, + "scheme": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Scheme", + "description": "how references to it start (vaultwarden://\u2026); none for the built-in store" + }, + "configured": { + "type": "boolean", + "title": "Configured", + "description": "set up, so a reference to it can be read" + }, + "scope": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Scope", + "description": "what it reads, in words: never a secret" + }, + "url": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Url" + }, + "email": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Email" + }, + "folder": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Folder" + }, + "installed": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Installed", + "description": "its client is fetched (a component), when it needs one" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "id", + "label", + "configured" + ], + "title": "SecretProvider" + }, + "SecretReference": { + "properties": { + "section": { + "type": "string", + "title": "Section" + }, + "key": { + "type": "string", + "title": "Key" + }, + "ref": { + "type": "string", + "title": "Ref", + "description": "the reference the field holds, like vaultwarden://OpenAI/password" + }, + "provider": { + "type": "string", + "title": "Provider" + }, + "read": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Read", + "description": "whether its value could be read by this server process; null before the first read" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error", + "description": "why it couldn't be read, last time it was tried" + }, + "read_at": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Read At", + "description": "when its value was last read" + }, + "missing": { + "type": "boolean", + "title": "Missing", + "description": "the password manager answered that the item or field isn't there", + "default": false + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "section", + "key", + "ref", + "provider" + ], + "title": "SecretReference" + }, + "SecretsTestResult": { + "properties": { + "ok": { + "type": "boolean", + "title": "Ok" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + }, + "items": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Items", + "description": "how many items the folder Lens reads holds" + }, + "folder": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Folder" + }, + "account": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Account" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "ok" + ], + "title": "SecretsTestResult" + }, + "SecretsTry": { + "properties": { + "provider": { + "type": "string", + "title": "Provider", + "default": "vaultwarden" + } + }, + "additionalProperties": false, + "type": "object", + "title": "SecretsTry" + }, + "SecretsView": { + "properties": { + "providers": { + "items": { + "$ref": "#/components/schemas/SecretProvider" + }, + "type": "array", + "title": "Providers" + }, + "references": { + "items": { + "$ref": "#/components/schemas/SecretReference" + }, + "type": "array", + "title": "References", + "description": "every secret field that refers to a password manager" + }, + "fields": { + "items": { + "additionalProperties": true, + "type": "object" + }, + "type": "array", + "title": "Fields", + "description": "every secret field in Settings: section, key, set, and its ref when it holds one" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "providers", + "references", + "fields" + ], + "title": "SecretsView" + }, "SegmentEdit": { "properties": { "idx": { @@ -75276,6 +75697,77 @@ ], "title": "VaultAnswer" }, + "VaultItem": { + "properties": { + "name": { + "type": "string", + "title": "Name" + }, + "type": { + "type": "string", + "title": "Type", + "description": "login, note, card, identity or other" + }, + "fields": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Fields", + "description": "the names of the fields that hold something" + }, + "ref": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Ref", + "description": "the reference to the item's default field (a login's password, a note's text)" + }, + "refs": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Refs", + "description": "the reference to each field" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "name", + "type", + "fields" + ], + "title": "VaultItem" + }, + "VaultItems": { + "properties": { + "provider": { + "type": "string", + "title": "Provider" + }, + "items": { + "items": { + "$ref": "#/components/schemas/VaultItem" + }, + "type": "array", + "title": "Items" + } + }, + "additionalProperties": true, + "type": "object", + "required": [ + "provider", + "items" + ], + "title": "VaultItems" + }, "VaultPasskey": { "properties": { "id": { diff --git a/threat-model.json b/threat-model.json index 210450d6e..11550ecee 100644 --- a/threat-model.json +++ b/threat-model.json @@ -1276,6 +1276,14 @@ "provenance": "documented", "source": "docs/configuration.md \"The document goes only to the address set\"" }, + { + "property": "P-SECRET-REFERENCES", + "violation_symptom": "info-leak", + "severity_tier": "security", + "conditions": "a secret field that holds a reference to a password manager (vaultwarden://item/field) stores only the reference, sealed like any secret (P-SECRETS); its value is read into memory, never written to the database, disk or logs; the API, the assistant's and MCP tools and approval cards show the reference and whether it could be read, never the value, and listing the vault shows item and field names only; Lens reads one folder of one account (secrets.vaultwarden_folder), and an item deleted or moved out of it stops being used at the next read; a reference is checked against the vault before it's saved; the vault's own sign-in (API key and master password) is sealed and can't itself be a reference", + "provenance": "documented", + "source": "docs/secrets.md \"People and agents see the reference and whether it could be read, but never the value\"" + }, { "property": "P-TEMPLATE", "violation_symptom": "integrity-bypass", diff --git a/threat-model.md b/threat-model.md index cf84fb850..2e52d280e 100644 --- a/threat-model.md +++ b/threat-model.md @@ -176,7 +176,8 @@ flowchart LR (P-TIKA-SERVER) and the LLM prompt. - **B4, Lens to the outside.** Fetches whose URL a non-admin sets, and iCal and news feeds, go through `netguard`. Other admin-set URLs do not; the telemetry endpoint is only shape-checked (P-ADMIN-GUARD). The opt-in browser - container has no way out but `browser-proxy`, the same guard run on its own. + container has no way out but `browser-proxy`, the same guard run on its own. A password manager an admin set up + (Vaultwarden) is read for the secret fields that refer to it (P-SECRET-REFERENCES). - **B5, peers.** Workers, the API and the web app trust the database and each other. Workers hold the DB credential. - **B6, devices.** Sensor listeners take traffic from the LAN (or wider, per `sensors.bind`) once an admin enables them. @@ -300,6 +301,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | `POST /imports/web`, extension `http` tools, notification targets | URL | yes: editors (web, tools), owners (targets) | resource-name, x-network-destination | none | *(maintainer, 2026-10)* | | the assistant's `web_search` and `web_read` (off until an admin picks a provider, `web.search`) | a search query and a page address, both chosen by the model; the results and pages that come back | yes: anyone who may use the assistant, and indirectly whatever it read, through the model; results and pages: whoever runs the site | resource-name, x-network-destination, x-prompt-injection, size | none; queries go only to the admin's provider; pages are fetched through netguard like web import, 10 MB at most (P-SSRF-GUARD); once the answer has read outside text, a query or address neither the person nor a tool gave it asks first (P-INJECTION-EGRESS); results and pages reach the model marked as data; the ledger keeps the provider or host, never the query or page | *(documented, docs/assistant.md "a search or a page read sends words out of Lens")* | | admin settings: model, embeddings, speech, decisions, Fedora, bridge, Telegram bot, Slack app tokens, assistant mailbox, telemetry, IIIF import, rclone remotes, IMAP, Gmail (a Google refresh token or a Workspace service account key), iCal, news feeds, MQTT bridge, OIDC issuer, web search (a SearXNG address, a Brave Search or Tavily key) | URL, credentials, `api_key_env` | admin only; documented guards still apply (P-ADMIN-GUARD) | resource-name, x-credential | admin is trusted | *(maintainer, 2026-10)* | +| Secret references and the password manager's sign-in: `secrets.*`, a secret field set to `vaultwarden://…` (`PUT /settings/
`, MCP `set_secret_reference`, the assistant's `change_settings`), `GET /secrets`, `GET /secrets/items`, `POST /secrets/test`, MCP `list_secrets` and `list_vault_items`, the assistant's `list_secrets` and `vault_items` | a reference (provider, item, field); the vault's address, account, API key and master password; the items and values the vault sends back | admins only; the vault's answers: whoever can write to Lens's vault account or share items into its folder | resource-name, x-credential, x-network-destination | admin is trusted with the address; a reference is checked against the vault before it's saved, and the vault's own sign-in can't be one; only one folder is read; values go only where the field's own value would, never to the API, tools, approval cards or logs (P-SECRET-REFERENCES); the assistant's changes to the section always ask first | *(documented, docs/secrets.md "People and agents see the reference and whether it could be read, but never the value")* | | replies from remote servers | web page HTML and scripts, IIIF manifests, model replies, OIDC discovery, calendars and their redirects | yes, the remote | data, topology | treat as untrusted | *(inferred, Q17)* | | chat, `/graph/ask`, MCP tools | prompt text; model-chosen tool calls | yes; also indirectly through content | data, x-prompt-injection | none; what people say to Lens, in the web app or a chat app, reaches automations as outside text (`chat.message`, `lens.untrusted`) | *(documented, docs/assistant.md "limited to what the person can read")* | | Lens extension side panel | answers shown there (model text that can repeat third-party text), the sign-in coming back | yes, indirectly through content | data, x-prompt-injection | none; answers render as Markdown cleaned by DOMPurify (no scripts, images or forms; only http and https links), and the sign-in checks its `state` and PKCE | *(documented, extensions/browser/src/core/assistant.js answerHtml docstring "cleaned of anything that could run or fetch on its own")* | @@ -490,6 +492,7 @@ third-party source; dependencies come from `uv.lock`, the pnpm lockfile and cont | NVIDIA OpenShell gateway (opt-in; `sandbox.gateway`) and its Python SDK (`sandbox` extra, Apache-2.0) | running the assistant's code apart from Lens: an unprivileged user with no capabilities, Landlock on files, seccomp, and egress only as the sandbox's policy allows, enforced by its supervisor; the gateway holds the container runtime (the Docker socket with its Docker driver), so whoever it lets in can start containers | upstream for the sandbox's isolation and the gateway's own access control (mutual TLS); the policy Lens asks for (P-SANDBOX), and in the `sandbox` profile the gateway's container, its certificates and where its port is published (§1.6) in model; a gateway an admin runs elsewhere is theirs to keep on a network only Lens reaches | *(documented, docs/assistant.md "Code sandbox")* | | The browser (`mcr.microsoft.com/playwright/mcp`, pinned, opt-in Compose profile `browser`) and its proxy `browser-proxy` (`netguard.py` on its own) | running a headless Chromium the assistant drives over MCP away from Lens: a container with none of Lens's data, keys or database, every capability dropped, `no-new-privileges`, a read-only image, memory and process limits, its profile in memory only, a network shared only with the backend and worker and marked internal, and answering only to the `Host` `browser:8931`; its only way out is `browser-proxy`, which reaches public addresses on ports 80 and 443 only, with a byte budget per connection; its `browser_run_code_unsafe` tool starts switched off | upstream for Chromium and Playwright MCP (a pinned image, trusted input, §1.9); the container's isolation and the proxy's checks in model; one browser, with its cookies and open pages, is shared by everyone whose assistant uses it | *(documented, docs/assistant.md "The browser")* | | Build, lint and codegen toolchain (frontend dev dependencies, openapi-ts, linters) | building the images; in no running service | upstream; a report must show the code ships | *(inferred, Q45)* | +| python-vaultwarden (`vaultwarden` extra, Apache-2.0; fetched once `secrets.vaultwarden_url` is set) and the Vaultwarden or Bitwarden server an admin points it at | signing in with an API key, deriving the master key and decrypting the account's items the way Bitwarden's clients do; the master password itself is never sent | upstream for the client's crypto and the server's access control; which folder is read, what is kept in memory and what is shown, in model (P-SECRET-REFERENCES) | *(documented, docs/secrets.md "The client is python-vaultwarden")* | | OIDC providers, SMTP, model and speech providers | honest responses over TLS | the admin chose them; trusted input | *(maintainer, 2026-10)* | ## 1.10 Adversary model @@ -559,6 +562,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-CONVERT-TIMEOUT | Every document conversion and capture ends within `documents.convert_seconds` and leaves no process of its group running; ffmpeg decoding (6 h), antiword, catdoc and pdftotext reads (300 s) and ffmpeg encoding for a speech service (600 s) end at their timeouts. | hang | correctness-only | *(maintainer, 2026-10-06, Q20)* | Voided by: children that ffmpeg or a reader starts itself (only converters run in their own process group) — `domain/ingest.py:794`, `domain/speech.py:183`. | | P-CONVERSION-NODE | A conversion node (`documents.anytopdf_url`) gets the document and its token only at the address the admin set: http(s) with no user name or password, and no redirect is followed. Its job answers are read to 1 MB, its PDF to 2 GB, and a PDF that doesn't start like one is refused before Lens reads it. anytopdf converting here has no network, no plugins and no config file. | info-leak, bad-data-accepted | security-critical | *(documented, docs/configuration.md "the token never goes on to another server")* | Voided by: The node's own address is trusted input (N-ADMIN); a node on plain `http://` off loopback sends the token in clear (§1.13) — `domain/anytopdf.py:237-244`. The node converts what it is sent: what it reads from the document is the node's, not Lens's. | | P-TIKA-SERVER | A Tika server (`documents.tika_url`, off by default) gets the document only at the address the admin set: http(s) with no user name or password, no redirect is followed, and the document goes as `document.`, never by its own name. Its answer is read to 64 MB, only the Dublin Core terms Lens keeps are taken from it (bounded in length), and the text kept is capped at 20 million characters. A Tika server that can't be reached fails only a document nothing else can read. The compose service runs read-only, without privileges, reachable only inside Lens's network. | info-leak, bad-data-accepted | security-critical | *(documented, docs/configuration.md "The document goes only to the address set")* | Voided by: The server's address is trusted input (N-ADMIN); a Tika server off this machine on plain `http://` sees every document in clear, and Tika itself has no sign-in, so whoever reaches it can use it — `domain/tika.py`. What Tika reads from the document is Tika's, not Lens's. | +| P-SECRET-REFERENCES | A secret field that holds a reference to a password manager (`vaultwarden://item/field`) stores only the reference, sealed like any secret (P-SECRETS). Its value is read into memory, never written to the database, disk or logs. The API, the assistant's and MCP tools and approval cards show the reference and whether it could be read, never the value, and listing the vault shows item and field names only. Lens reads one folder of one account (`secrets.vaultwarden_folder`); an item deleted or moved out of it stops being used at the next read. A reference is checked against the vault before it's saved. The vault's own sign-in (API key and master password) is sealed and can't itself be a reference. | info-leak | security-critical | *(documented, docs/secrets.md "People and agents see the reference and whether it could be read, but never the value")* | Voided by: whoever holds `secret.key` unseals the vault's sign-in, and with it every item of Lens's vault account (Q14). The account's decrypted key and the values read sit in the memory of each process that uses them — `domain/vaultwarden_secrets.py`, `domain/secret_providers.py`. An admin who points a field's consumer elsewhere (a model URL, the telemetry endpoint) sends it the value, as with a typed secret (trusted). The vault's address is trusted input (N-ADMIN): a vault on plain http off this machine shows the API key and the encrypted items to the network. A value read isn't checked the way a typed one is until it's used. | | P-TEMPLATE | Templates render in a sandboxed Jinja environment. They cannot reach Python internals, output is capped, and reports escape HTML. | integrity-bypass | security-critical | *(documented, docs/processing.md "sandboxed Jinja environment")* | Voided by: Shipped report templates use an unsandboxed environment over server-built values — `domain/render.py:22`. `grep -rn 'Environment(\\|from_string' fastapi_backend/app` — hits in `domain/render.py` and `domain/templates.py`. | | P-ENCRYPTION | With `encryption.files` on, files are encrypted per namespace. A file cut short, edited or reordered fails to open. A vault has no recovery code, password or admin override, and separate workers never get its key. | info-leak, integrity-bypass | security-critical | *(documented, docs/encryption.md "There is no recovery code, password or admin override")* | Voided by: `encryption.files` off — `domain/keyring.py:545`; shipped default off (`domain/store.py:402`) until a new archive's first start or `lens encrypt`. Unlocked vault keys stay in the API for `encryption.vault_minutes`. `secret.key` placement (Q14). | | P-TELEMETRY | Telemetry is off unless turned on, has no built-in destination, and never sends content, names, addresses or error messages. | info-leak | security-critical | *(documented, docs/telemetry.md "Never sent")* | Voided by: none identified (not required for this symptom; no exhaustive search) | diff --git a/threat-model.yaml b/threat-model.yaml index 05ec43df2..6fc06869d 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -1735,6 +1735,14 @@ properties_claimed: violation_symptoms: [info-leak, bad-data-accepted] x-voided-by: "the server's address is trusted input (N-ADMIN); a Tika server off this machine on plain http sees every document in clear, and Tika has no sign-in of its own (domain/tika.py); what Tika reads from the document is Tika's" provenance: {kind: documented, source: 'docs/configuration.md "The document goes only to the address set"'} + - id: P-SECRET-REFERENCES + kind: confidentiality + components: [outbound, assistant] + tier: security-critical + conditions: "a secret field that holds a reference to a password manager (vaultwarden://item/field) stores only the reference, sealed like any secret (P-SECRETS); its value is read into memory, never written to the database, disk or logs; the API, the assistant's and MCP tools and approval cards show the reference and whether it could be read, never the value, and listing the vault shows item and field names only; Lens reads one folder of one account (secrets.vaultwarden_folder), and an item deleted or moved out of it stops being used at the next read; a reference is checked against the vault before it's saved; the vault's own sign-in (API key and master password) is sealed and can't itself be a reference" + violation_symptoms: [info-leak] + x-voided-by: "whoever holds secret.key unseals the vault's sign-in, and with it every item of Lens's vault account (Q14); the account's decrypted key and the values read sit in the memory of each process that uses them (domain/vaultwarden_secrets.py, domain/secret_providers.py); an admin who points a field's consumer elsewhere (a model URL, the telemetry endpoint) sends it the value, as with a typed secret (trusted); the vault's address is trusted input (N-ADMIN): a vault on plain http off this machine shows the API key and the encrypted items to the network; a value read isn't checked the way a typed one is until it's used" + provenance: {kind: documented, source: 'docs/secrets.md "People and agents see the reference and whether it could be read, but never the value"'} - id: P-TEMPLATE kind: integrity components: [ingest, assistant]