From 59ee8290f5b44c4aa256f81e1126e42148171666 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Wed, 14 Oct 2026 09:53:58 +0000 Subject: [PATCH] docs(threat-model): claim the prompt-injection egress controls P-INJECTION-EGRESS: once an answer has read text from outside Lens, outbound tools and setup changes ask first, and chat-room answers keep only the links people, the archive, tools or Lens wrote. Correctness-only, a bypass is VALID-HARDENING (Q26). The Assistant x-prompt-injection row is now claimed; N-PROMPT says marking and flagging make injection less likely and are not claimed; MCP results and the optional classifier are described; Q26 lists #302, #309, #329, #338, #343, #344 and #347. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed --- threat-model.json | 10 +++++++++- threat-model.md | 10 ++++++---- threat-model.yaml | 16 ++++++++++++---- 3 files changed, 27 insertions(+), 9 deletions(-) diff --git a/threat-model.json b/threat-model.json index 44e4af5ca..800feccac 100644 --- a/threat-model.json +++ b/threat-model.json @@ -959,6 +959,14 @@ "provenance": "documented", "source": "docs/assistant.md \"Approvals\"" }, + { + "property": "P-INJECTION-EGRESS", + "violation_symptom": "info-leak", + "severity_tier": "correctness", + "conditions": "Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no bridge.users, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own read web tools, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is VALID-HARDENING (Q26).", + "provenance": "documented", + "source": "docs/assistant.md \"it read out of Lens waits for your approval\"" + }, { "property": "P-EXT-SCOPE", "violation_symptom": "integrity-bypass", @@ -1336,7 +1344,7 @@ }, { "property": "N-PROMPT", - "reason": "resistance to prompt injection from transcripts, documents, OCR, web pages or tool replies; impact on words and notes only; crossing the person's reads, approvals or role is P-RETRIEVAL or P-APPROVAL; exfiltration through rendered answers is Q26", + "reason": "resistance to prompt injection from transcripts, documents, OCR, web pages or tool replies; impact on words and notes only; crossing the person's reads, approvals or role is P-RETRIEVAL or P-APPROVAL; carrying what it read out through tools or chat-room links is P-INJECTION-EGRESS. Marking third-party text as data, breaking up chat-template control tokens and flagging text that reads like instructions (ai.untrusted_marking, ai.injection_check) make injection less likely; they are not claimed", "false_friend": true, "provenance": "inferred", "source": "open question Q26" diff --git a/threat-model.md b/threat-model.md index 9496b468e..8f894564b 100644 --- a/threat-model.md +++ b/threat-model.md @@ -387,7 +387,7 @@ Every row is claimed, disclaimed, N/A with a reason, or unresolved. Family short | Assistant | Reference lifecycle | unresolved | saved answers keep text the person can no longer read | Q11 | *(inferred, Q11)* | | Assistant | Concurrency / reentrancy | N/A — one loop per conversation | tool calls run in order | none | *(inferred, Q34)* | | Assistant | Resource complexity | disclaimed | budgets are pre-run estimates, off by default | N-BUDGET | *(documented, docs/budgets.md "Budgets are off unless an admin sets one")* | -| Assistant | x-prompt-injection | unresolved | content steers the model within the person's tools, or into answers that carry text out | Q26 | *(inferred, Q26)* | +| Assistant | x-prompt-injection | claimed | steering the model within the person's tools is disclaimed (N-PROMPT); carrying what it read out through outbound tools, setup changes or chat-room links asks first or is broken up | P-INJECTION-EGRESS, N-PROMPT | *(documented, docs/assistant.md "it read out of Lens waits for your approval")* | | Assistant | x-template-reach | unresolved | any writer can use a `template` node in a custom node | Q23 | *(inferred, Q23)* | | Optional | Numeric domain | unresolved | payload 256 KB, 600 per minute per stream, syslog line 64 KB | Q29 | *(inferred, Q29)* | | Optional | Atomicity, recursion, serialization | N/A — flat, append-only readings stored as data | no multi-step state, nesting or object reconstruction | none | *(inferred, Q34)* | @@ -411,7 +411,7 @@ stored text for the consumer's format. | Embed player and `/s/` pages | `api/pages.py` | HTML built from DB values under a strict CSP; framable only per `server.embed_frame_ancestors` | that titles are inert outside this page | *(documented, docs/configuration.md "Content-Security-Policy")* | | Report pages | `api/pages.py` | links re-signed on read for every recording of the namespace in the page | that a report link opens only the report | *(inferred, Q10)* | | Assistant answers, summaries, LLM-written notes | Assistant | model output over untrusted content | that it is grounded or free of injected instructions | *(inferred, Q26)* | -| MCP tool results | `api/mcp_tools.py` | the person's readable data, as text | that transcript text is safe to follow as instructions | *(documented, docs/mcp.md "sees exactly what that person sees")* | +| MCP tool results | `api/mcp_tools.py` | the person's readable data, as text, with chat-template control tokens broken up and a note when text reads like instructions to an AI | that transcript text is safe to follow as instructions | *(documented, docs/mcp.md "sees exactly what that person sees")* | | Error text from remote servers | LLM, speech, rclone, IMAP callers | up to 300 characters of the remote reply and the URL as typed | that internal host names stay hidden | *(inferred, Q18)* | | Webhook bodies | `notify.py` | event data, HMAC-signed for the `webhook` kind only | that Slack or Discord messages are signed | *(documented, domain/notify.py docstring "signed the Standard Webhooks way")* | | Fedora export | `fedora.py` | every namespace's metadata and files | that the Fedora account is limited to one namespace | *(documented, domain/fedora.py docstring "every namespace, collection, recording")* | @@ -439,6 +439,7 @@ third-party source; dependencies come from `uv.lock`, the pnpm lockfile and cont | rclone, stdlib `urllib`, `imaplib`, `ssl` | protocol clients and TLS checks | upstream | *(inferred, Q45)* | | cloudflared, llama.cpp, `hf:` models, pip packages | integrity only as far as HTTPS and the publisher | upstream; tampered downloads are disclaimed (N-DOWNLOADS) | *(maintainer, 2026-10)* | | Catalog GGUF models | pinned revision and SHA-256 | in model (P-MODEL-CHECKSUM) | *(documented, docs/local-models.md "checked against their SHA-256")* | +| Prompt-injection classifier (ProtectAI deberta-v3-base-prompt-injection-v2, `injection` extra, opt-in) | a judgement on whether a tool's result reads like an injection; the model is fetched at a pinned revision and checked against its SHA-256 | upstream for the model's judgement: a miss or a false flag is N-PROMPT; a model file that passes unchecked is in model | *(documented, docs/assistant.md "fetched when you pick it")* | | Optional ML extras (whisper, pyannote, speechbrain, insightface, doctr) | model inference; some fetch weights on their own | upstream | *(inferred, Q19)* | | StyleTTS2 service (`styletts2/`, opt-in Compose profile) and macOS `say` | text to speech; StyleTTS2 fetches its weights from Hugging Face and GitHub on first use and listens only on the Compose network; `say` gets the text on stdin, never as arguments | upstream for the model and synthesis; the argument handling in model | *(maintainer, 2026-10)* | | Build, lint and codegen toolchain (frontend dev dependencies, openapi-ts, linters) | building the images; in no running service | upstream; a report must show the code ships | *(inferred, Q45)* | @@ -499,6 +500,7 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-QUERY | Cypher is read-only and runs on the caller's readable namespaces. A query stops within 3 million steps or 8 seconds; paths run up to 8 hops. Threshold: running past the budget is a bug; slow within it is not. | integrity-bypass (writes, scope); hang (budget) | security-critical (scope, writes); correctness-only (budget) | *(documented, docs/graph.md "3 million steps")* | Voided by: Admins see every namespace; write refusals — `domain/cypher.py:39`; budgets — `domain/cypher.py:35-38`. | | P-RETRIEVAL | Chat, assistant tools and MCP return only what the person can read, within the conversation's scope, and the tool loop stops at `ai.max_steps` and `ai.max_transcript_reads`. | info-leak (scope); hang (caps) | security-critical (scope); correctness-only (caps) | *(documented, docs/assistant.md "limited to what the person can read")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | | P-APPROVAL | Assistant tools that change data wait for the person's approval, except notes and an admin's setup conversation. | integrity-bypass | security-critical | *(documented, docs/assistant.md "Approvals")* | Voided by: Admins read everything; an admin setup conversation adds server tools and the `act` flag — `domain/ai_tools.py:287-289`; `ai.extensions: false` turns extensions off. | +| P-INJECTION-EGRESS | Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no `bridge.users`, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own `read` web tools, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is `VALID-HARDENING` (Q26). | info-leak | correctness-only | *(documented, docs/assistant.md "it read out of Lens waits for your approval")* | Voided by: `ai.untrusted_asks: false` — `domain/ai_tools.py:519`; `bridge.safe_links: false` — `domain/bridge.py:298`. Notes and memories are written at once: they stay in Lens. Hooks run as their owners set them (P-EXT-SCOPE). An address the read text wrote itself stays linkable in a room. The person's browser asks first as P-BROWSER-GUARD says. | | P-EXT-SCOPE | Your own hooks run in your conversations; another person's hooks run only when an admin shared them. | integrity-bypass | security-critical | *(documented, docs/assistant.md "other people's run only when an admin shared them")* | Voided by: An admin sharing the extension — `domain/extensions.py:595-598`. | | P-NO-CODE | No non-admin can run code on the host. This covers owners, editors, viewers, API tokens, MCP agents, OAuth apps and model tool calls acting for one. Python tools are written by admins only and run only while an admin owns them. | integrity-bypass | security-critical | *(maintainer, 2026-10)* | Voided by: Making someone admin; authoring gate — `domain/extensions.py:129-130`; run gate — `domain/extensions.py:600-601`. `grep -rnE "eval\(\|exec\(\|pickle\|yaml\.load\(\|__import__" fastapi_backend/app` — one `exec`, inside the code-tool child. | | P-UPLOAD | Uploads accept only the types in `uploads.extensions` and are size-capped. Nothing uploaded is executed. Files are served as audio, video or a download, and names carry no folders. | bad-data-accepted | security-critical | *(documented, docs/configuration.md "Uploads accept only the types")* | Voided by: Admin-added types; check — `domain/uploads.py:157-158`. The `attachments` role takes any type (never executed). Bytes are not checked against the extension. | @@ -564,7 +566,7 @@ Exported by phase 3.6 (backtest) and de-identified. Closing rows carry their sta | N-CHROMIUM | Chromium's own sandbox in containers or as root; the `netguard` proxy is what stops a page | Conversion and capture | security-critical | yes: "headless browser" is not isolation | *(documented, docs/deployment.md "what stops a page is Lens's proxy")* | | N-DECODERS | Sandboxing or memory-safety for ffmpeg, poppler, antiword and similar decoders | Memory bugs inside them route to §1.9. Does not cover Lens's own process control (timeouts, killing children): that is P-CONVERT-TIMEOUT | security-critical | no | *(inferred, Q20)* | | N-BOMBS | Bounds on decompressed size, nested emails or media duration | docx XML, email attachments, long media. Does not cover a conversion or capture outliving Lens's own timeout (P-CONVERT-TIMEOUT) | security-critical | no | *(inferred, Q22)* | -| N-PROMPT | Resistance to prompt injection from transcripts, documents, OCR, web pages or tool replies. "Use only the excerpts" is a prompt, not a control | Impact on words and notes only; crossing the person's reads, approvals or role is P-RETRIEVAL or P-APPROVAL. Exfiltration through rendered answers is Q26 | correctness-only | yes: a system prompt is not a filter | *(inferred, Q26)* | +| N-PROMPT | Resistance to prompt injection from transcripts, documents, OCR, web pages or tool replies. "Use only the excerpts" is a prompt, not a control | Impact on words and notes only; crossing the person's reads, approvals or role is P-RETRIEVAL or P-APPROVAL; carrying what it read out through tools or chat-room links is P-INJECTION-EGRESS. Marking third-party text as data, breaking up chat-template control tokens and flagging text that reads like instructions (`ai.untrusted_marking`, `ai.injection_check`) make injection less likely; they are not claimed | correctness-only | yes: a system prompt is not a filter | *(inferred, Q26)* | | N-MODEL-OUTPUT | Correct or grounded model output; answers can be wrong | Checking sources re-asks the same model | correctness-only | no | *(documented, docs/assistant.md "Check sources")* | | N-REMOTE | Trust in remote content: storage listings, mail, calendars, IIIF manifests, model replies | Treated as data. Does not cover Lens's own cache keys or names for remote content (P-REMOTE-NAMES, Q43) | security-critical | no | *(inferred, Q17)* | | N-ADMIN-IMPORT-BOUNDS | Resource bounds on admin-started imports and syncs; SPARQL time | IIIF media up to 4 GB in memory; rclone without byte cap | security-critical | no | *(inferred, Q16)* | @@ -744,7 +746,7 @@ the proposed answer, except where the table says otherwise. The questions stay b | Q20 | Claim P-CONVERT-TIMEOUT | decoders, document readers and speech encoding given timeouts (#202) | | Q21 | `VALID` under P-NO-INJECTION | ffmpeg and ffprobe get a demuxer allow-list (#196) | | Q22–Q25 | Proposed answers | Q24 checked: holds | -| Q26 | Proposed answer; render-time fetch is `VALID-HARDENING` | pasted Markdown images no longer fetch (#194) | +| Q26 | Proposed answer; render-time fetch is `VALID-HARDENING` | pasted Markdown images no longer fetch (#194); third-party text marked as data and flagged, outbound tools and setup changes ask first after reading it, chat-room links the model built broken up, MCP results neutralized (P-INJECTION-EGRESS; #302, #309, #329, #338, #343, #344, #347) | | Q27 | Claim P-EXT-SCOPE for tools | arguments never change the host (checked); another person's tool now asks first unless an admin shared it (#203) | | Q28 | Proposed answer | — | | Q29 | Proposed answer | checked: holds | diff --git a/threat-model.yaml b/threat-model.yaml index dc0979c46..114dcab8d 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:275791b79d131205c80bd3e1ea47b5c20ebbe620940f45a9cb3e2c4a1aba90db" +prose_version: "threat-model.md@sha256:c57648dcbdb780aa2b558aa874b44dcffd38506a5f9d2249a9d41c906990c71f" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -570,7 +570,7 @@ contract_dimensions: - {component: assistant, dimension: reference-lifecycle, status: unresolved, conditions: "saved answers keep text the person can no longer read", question_id: Q11, provenance: {kind: inferred, question_id: Q11}} - {component: assistant, dimension: concurrency-reentrancy, status: not-applicable, conditions: "one loop per conversation; tool calls run in order", provenance: {kind: inferred, question_id: Q34}} - {component: assistant, dimension: resource-complexity, status: disclaimed, property_id: N-BUDGET, conditions: "budgets are pre-run estimates, off by default", provenance: {kind: documented, source: 'docs/budgets.md "Budgets are off unless an admin sets one"'}} - - {component: assistant, dimension: x-prompt-injection, status: unresolved, conditions: "content steers the model within the person's tools, or into answers that carry text out", question_id: Q26, provenance: {kind: inferred, question_id: Q26}} + - {component: assistant, dimension: x-prompt-injection, status: claimed, property_id: P-INJECTION-EGRESS, conditions: "steering the model within the person's tools is disclaimed (N-PROMPT); carrying what it read out through outbound tools, setup changes or chat-room links asks first or is broken up", provenance: {kind: documented, source: 'docs/assistant.md "it read out of Lens waits for your approval"'}} - {component: assistant, dimension: x-template-reach, status: unresolved, conditions: "any writer can use a template node in a custom node", question_id: Q23, provenance: {kind: inferred, question_id: Q23}} # --- Optional - {component: optional, dimension: numeric-domain, status: unresolved, conditions: "payload 256 KB, 600 per minute per stream, syslog line 64 KB", question_id: Q29, provenance: {kind: inferred, question_id: Q29}} @@ -646,7 +646,7 @@ outputs: - component: authz channel: "MCP tool results (api/mcp_tools.py)" taint: same-as-input - x-taint-note: "the person's readable data, as text" + x-taint-note: "the person's readable data, as text, with chat-template control tokens broken up and a note when text reads like instructions to an AI" taint_provenance: {kind: documented, source: 'docs/mcp.md "sees exactly what that person sees"'} invariants: - {id: inv-mcp-person-reads, statement: "results contain only what the person can read", property_id: P-RETRIEVAL, provenance: {kind: documented, source: 'docs/mcp.md "sees exactly what that person sees"'}} @@ -1369,6 +1369,14 @@ properties_claimed: violation_symptoms: [integrity-bypass] x-voided-by: "admins read everything; an admin setup conversation adds server tools and the act flag (domain/ai_tools.py:287-289); ai.extensions: false turns extensions off" provenance: {kind: documented, source: 'docs/assistant.md "Approvals"'} + - id: P-INJECTION-EGRESS + kind: confidentiality + components: [assistant] + tier: correctness-only + conditions: "Once an assistant answer has read text from outside Lens (a tool result other than Lens's own, the page it was asked from, a namespace assistant's memories, a message in a chat room with no bridge.users, or an earlier answer in the conversation that read any of these), tools that could carry it out of Lens (the person's own read web tools, other servers' MCP tools, Python tools with the network) and an admin setup conversation's own changes wait for the person's approval. A chat room's answer keeps a web address linkable only when the person, the archive, a tool or Lens wrote it. A bypass is VALID-HARDENING (Q26)." + violation_symptoms: [info-leak] + x-voided-by: "ai.untrusted_asks: false (domain/ai_tools.py:519); bridge.safe_links: false (domain/bridge.py:298); notes and memories are written at once; hooks run as their owners set them (P-EXT-SCOPE); an address the read text wrote itself stays linkable in a room; the person's browser asks first as P-BROWSER-GUARD says" + provenance: {kind: documented, source: 'docs/assistant.md "it read out of Lens waits for your approval"'} - id: P-EXT-SCOPE kind: integrity components: [assistant] @@ -1586,7 +1594,7 @@ properties_disclaimed: - {id: N-CHROMIUM, components: [outbound, ingest], tier: security-critical, false_friend: true, conditions: "Chromium's own sandbox in containers or as root; the netguard proxy is what stops a page; conversion and capture", x-false-friend-note: "'headless browser' is not isolation", provenance: {kind: documented, source: 'docs/deployment.md "what stops a page is Lens''s proxy"'}} - {id: N-DECODERS, components: [ingest], tier: security-critical, false_friend: false, conditions: "sandboxing or memory-safety for ffmpeg, poppler, antiword and similar decoders; memory bugs inside them route to §1.9; does not cover Lens's own process control (timeouts, killing children): that is P-CONVERT-TIMEOUT", provenance: {kind: inferred, question_id: Q20}} - {id: N-BOMBS, components: [ingest], tier: security-critical, false_friend: false, conditions: "bounds on decompressed size, nested emails or media duration (docx XML, email attachments, long media); does not cover a conversion or capture outliving Lens's own timeout (P-CONVERT-TIMEOUT)", provenance: {kind: inferred, question_id: Q22}} - - {id: N-PROMPT, components: [assistant], tier: correctness-only, false_friend: true, conditions: "resistance to prompt injection from transcripts, documents, OCR, web pages or tool replies; impact on words and notes only; crossing the person's reads, approvals or role is P-RETRIEVAL or P-APPROVAL; exfiltration through rendered answers is Q26", x-false-friend-note: "a system prompt is not a filter", provenance: {kind: inferred, question_id: Q26}} + - {id: N-PROMPT, components: [assistant], tier: correctness-only, false_friend: true, conditions: "resistance to prompt injection from transcripts, documents, OCR, web pages or tool replies; impact on words and notes only; crossing the person's reads, approvals or role is P-RETRIEVAL or P-APPROVAL; carrying what it read out through tools or chat-room links is P-INJECTION-EGRESS. Marking third-party text as data, breaking up chat-template control tokens and flagging text that reads like instructions (ai.untrusted_marking, ai.injection_check) make injection less likely; they are not claimed", x-false-friend-note: "a system prompt is not a filter", provenance: {kind: inferred, question_id: Q26}} - {id: N-MODEL-OUTPUT, components: [assistant], tier: correctness-only, false_friend: false, conditions: "correct or grounded model output; answers can be wrong; checking sources re-asks the same model", provenance: {kind: documented, source: 'docs/assistant.md "Check sources"'}} - {id: N-REMOTE, components: [outbound], tier: security-critical, false_friend: false, conditions: "trust in remote content (storage listings, mail, calendars, IIIF manifests, model replies), treated as data; does not cover Lens's own cache keys or names for remote content (P-REMOTE-NAMES, Q43)", provenance: {kind: inferred, question_id: Q17}} - {id: N-ADMIN-IMPORT-BOUNDS, components: [outbound, query], tier: security-critical, false_friend: false, conditions: "resource bounds on admin-started imports and syncs, and SPARQL time; IIIF media up to 4 GB in memory; rclone without byte cap", provenance: {kind: inferred, question_id: Q16}}