From fff8f0bf554157d6299928f8a18b929f48989917 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Thu, 15 Oct 2026 22:38:13 +0000 Subject: [PATCH] feat(automations): call a web address, with credentials sealed in headers Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01CyjR2zEfmTXBewKAx9Taw5 --- docs/automations.md | 21 ++ docs/notifications.md | 2 +- fastapi_backend/app/domain/actions.py | 5 +- fastapi_backend/app/domain/actions_web.py | 226 ++++++++++++++++++ fastapi_backend/app/domain/notify.py | 14 +- fastapi_backend/tests/api/test_actions_web.py | 188 +++++++++++++++ threat-model.json | 4 +- threat-model.md | 6 +- threat-model.yaml | 8 +- 9 files changed, 460 insertions(+), 14 deletions(-) create mode 100644 fastapi_backend/app/domain/actions_web.py create mode 100644 fastapi_backend/tests/api/test_actions_web.py diff --git a/docs/automations.md b/docs/automations.md index 1f5cb7501..6970491e1 100644 --- a/docs/automations.md +++ b/docs/automations.md @@ -19,6 +19,7 @@ merged is called built. | The action registry with risk from each action's effect, and one action, `log_line` (`app/domain/actions.py`) | built | | Telling a person: the `notify` action and `channels.tell` (their own chat, a notification target, a Matterbridge room) | built | | The model and the agent: the `ask_model`, `call_tool` and `ask_agent` actions, and prompts with outside text marked | built | +| Calling a web address: the `http_request` action, with credentials in headers kept sealed | built | | Settings → Automations: how often followers look and how long the event log is kept | built | | Storing and running automations, schedules and timers, joins, the other actions, approvals, variables, the timeline page, the studio | planned | @@ -261,6 +262,7 @@ or removed does nothing. What it writes is by the automation itself, `automation | `ask_model` | asks the model a `prompt` (with your own `system` instructions and `model` if you like); with `json_schema` (a JSON Schema of type object) the reply is that object | read | `{text}`, or the object the schema describes | | `call_tool` | calls one of the assistant's tools (`tool`, with `args`) as the automation | read: a change becomes an approval | `{tool, result, summary, approvals}` | | `ask_agent` | hands what happened (`about`, by default the event) to the agent with your `instructions`, limited to some `tools` if you like, for up to `max_steps` rounds (6, at most 12) | read: changes become approvals | `{answer, chat, approvals, steps}` | +| `http_request` | sends a `method` (GET, POST or PUT) to a `url`, with `headers` and a `body` (text, or an object sent as JSON) | out | `{status, ok, body, json, content_type, truncated, method, url, headers}`: the answer, as outside text | #### Telling a person @@ -298,6 +300,22 @@ the run read from outside Lens before the model sees it: each activity's `lens.u and body) and everything earlier steps passed on (a model's reply, a web page). Your own words in the prompt stay as written. `ask_agent` passes the activity it is about the same way. +#### Calling a web address + +`http_request` reaches public addresses only, or networks an admin allowed for notifications (Settings → +Notifications), connecting to the address it checked and following no redirect (a 3xx is passed on as it is). It takes +10 seconds at most, sends a body of up to 64 KB, and passes on the status and the first 64 KB of the answer (`json` +too when the answer is JSON and whole). The answer is outside text: a later prompt reads it marked as data. It always +reaches outside Lens, so it asks first unless you set its node to run on its own; each call goes in the activity +ledger (`automation.http`, with the method, host and status). + +Credentials go in headers. A header whose name says it carries one (`Authorization`, cookies, API keys, tokens, +secrets, passwords, signatures) is never shown: what the action describes and passes on says `••••`. When an automation +is saved, `actions_web.seal_headers(db, cfg, space, automation, url, headers)` seals those values with the namespace's +key ([Encryption](encryption.md)), bound to the automation and to the scheme, host and port of its address, so a +sealed value can't be copied into another automation or sent to another host; it is opened only to send. A locked +vault stops the request until it is unlocked. + ## Refine later - More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`. @@ -314,3 +332,6 @@ written. `ask_agent` passes the activity it is about the same way. - The model and the agent: per-node model picks in the studio; streaming an agent's steps into the run's log as they happen; a namespace assistant's memory for automations (`remember` is left out today); approving an agent's proposals from the chat app it told you in; count the automation's own budget once budgets take `automation:`. +- Calling a web address: secrets in the address itself (a Slack or Discord webhook URL) sealed like headers; PATCH and + DELETE (DELETE would count as deleting); OAuth sign-in for an API; form-encoded bodies; retries with backoff; + networks allowed for automations apart from notifications. diff --git a/docs/notifications.md b/docs/notifications.md index 39d63dae7..eec091b12 100644 --- a/docs/notifications.md +++ b/docs/notifications.md @@ -118,7 +118,7 @@ Under Settings → Notifications (the `notifications` section, see [Configuratio | Setting | Default | | | --- | --- | --- | | `enabled` | `true` | Off: nothing is sent, and what happens meanwhile isn't sent later | -| `networks` | `[]` | Private networks targets may be in, like `192.168.1.0/24` or `172.16.0.0/12` (Docker) | +| `networks` | `[]` | Private networks targets may be in, like `192.168.1.0/24` or `172.16.0.0/12` (Docker); automations calling a web address ([`http_request`](automations.md#calling-a-web-address)) may reach them too | | `app_url` | `null` | Where links in messages point; `null` uses the server's `FRONTEND_URL` | | `poll_seconds` | `5` | How often the notifier looks | | `max_attempts` | `6` | How many times a message is tried | diff --git a/fastapi_backend/app/domain/actions.py b/fastapi_backend/app/domain/actions.py index 55cad0684..de8545977 100644 --- a/fastapi_backend/app/domain/actions.py +++ b/fastapi_backend/app/domain/actions.py @@ -13,7 +13,8 @@ (`person`), with their permissions as they are when it runs (`reach`), and what it writes is by the automation itself (`actor`). Params marked `"format": "prompt"` are prompts: the engine renders them with `render_prompt`, which wraps what the run read from outside Lens as data. `log_line` ships here; the other actions are registered by a module per -group, loaded with this one: actions_messages.py (notify), actions_ai.py (ask_model, call_tool, ask_agent). +group, loaded with this one: actions_messages.py (notify), actions_ai.py (ask_model, call_tool, ask_agent), +actions_web.py (http_request). """ from __future__ import annotations @@ -341,4 +342,4 @@ def _log_line(db, cfg, intent, user, dry): ) -from . import actions_ai, actions_messages # noqa: E402, F401 - they register their actions; loaded with the catalog +from . import actions_ai, actions_messages, actions_web # noqa: E402, F401 - they register their actions; loaded with the catalog diff --git a/fastapi_backend/app/domain/actions_web.py b/fastapi_backend/app/domain/actions_web.py new file mode 100644 index 000000000..223dea637 --- /dev/null +++ b/fastapi_backend/app/domain/actions_web.py @@ -0,0 +1,226 @@ +"""The automation action that calls a web address (docs/automations.md, Actions): `http_request`. + +It sends GET, POST or PUT to a public address (or a network an admin allowed for notifications), pinned to the address +that was checked and without following redirects (notify.request), waits at most 10 seconds, and passes on the status +and the first 64 KB of the answer as outside text (a later prompt reads it marked as data). It always reaches outside +Lens (effect `out`), so it asks first unless the person set that node to run on its own. + +Secrets go in headers, and are kept sealed with the namespace's key (keyring.py): `seal_headers` seals the values of +headers that carry credentials (Authorization, cookies, API keys and tokens) when an automation is saved, bound to the +automation and to the scheme, host and port they were saved for, so a sealed value can't be moved to another +automation or sent to another host. They are opened only to send, and never shown: what the action describes and +passes on names the header, not its value. +""" + +from __future__ import annotations + +import http.client +import json +import re +import urllib.parse + +from . import actions, activity, keyring, notify + +METHODS = ("GET", "POST", "PUT") +TIMEOUT = 10 # seconds the whole request may take +MAX_ANSWER = 64 * 1024 +MAX_BODY = 64 * 1024 +MAX_HEADERS = 30 +NAME_RX = re.compile(r"^[A-Za-z0-9!#$%&'*+.^_`|~-]{1,100}$") +SECRET_RX = re.compile(r"auth|token|secret|key|password|cookie|signature|session", re.I) +FIXED = {"host", "content-length", "transfer-encoding", "connection", "upgrade", "te", "expect", "keep-alive"} +HIDDEN = "••••" + + +def secret(name): + """Whether a header carries a credential, so its value is sealed and never shown.""" + return bool(SECRET_RX.search(str(name))) + + +def _origin(url): + u = urllib.parse.urlsplit(url) + return f"{u.scheme}://{(u.hostname or '').lower()}:{u.port or (443 if u.scheme == 'https' else 80)}" + + +def _context(automation, name, url): + return f"automation:{automation}.header:{name.lower()}@{_origin(url)}" + + +def _url(p): + try: + return notify.check_url(str(p.get("url") or ""), "webhook") + except ValueError as e: + raise ValueError(f"http_request: {e}") from None + + +def _headers(p): + h = p.get("headers") or {} + if not isinstance(h, dict) or len(h) > MAX_HEADERS: + raise ValueError(f"http_request: headers are an object of up to {MAX_HEADERS} names and values") + out = {} + for k, v in h.items(): + if not NAME_RX.match(str(k)) or str(k).lower() in FIXED: + raise ValueError(f"http_request: {k} isn't a header it can set") + if not isinstance(v, str) or "\r" in v or "\n" in v or len(v) > 8000: + raise ValueError(f"http_request: the {k} header is one line of text") + out[str(k)] = v + return out + + +def seal_headers(db, cfg, space, automation, url, headers): + """Headers as kept: the values of those that carry credentials sealed with the namespace's key, bound to the + automation and to where `url` points. Values already sealed are kept as they are.""" + out = {} + for k, v in _headers({"headers": headers}).items(): + if secret(k) and v and not keyring.is_sealed(v): + v = keyring.seal(db, cfg, int(space), v.encode(), _context(automation, k, _url({"url": url}))) + out[k] = v + return out + + +def _open(db, cfg, intent, url, headers): + """The headers to send: sealed values opened, when they were sealed for this automation, one of its namespaces + and this address.""" + out = {} + for k, v in headers.items(): + if keyring.is_sealed(v): + try: + sid = int(v.split(".")[1]) + except (IndexError, ValueError): + raise ValueError(f"http_request: the {k} header's sealed value is damaged; enter it again") from None + if intent["spaces"] and sid not in intent["spaces"]: + raise ValueError(f"http_request: the {k} header was sealed for another namespace; enter it again") + try: + v = keyring.unseal(db, cfg, v, _context(intent.get("automation"), k, url)).decode() + except keyring.Locked: + raise ValueError("http_request: the namespace's vault is locked; unlock it to send its secrets") from None + except (keyring.Damaged, KeyError, ValueError): + raise ValueError(f"http_request: the {k} header was sealed for another automation or address; enter it again") from None + out[k] = v + return out + + +def _shown(headers): + return {k: HIDDEN if secret(k) or keyring.is_sealed(v) else v for k, v in headers.items()} + + +def _body(method, body, headers): + """(bytes or None, headers with a Content-Type): an object is sent as JSON, text as it is.""" + if body is None or body == "": + return None, headers + if method == "GET": + raise ValueError("http_request: a GET sends no body") + has_type = any(k.lower() == "content-type" for k in headers) + if isinstance(body, (dict, list)): + data = json.dumps(body, ensure_ascii=False).encode() + headers = headers if has_type else {**headers, "Content-Type": "application/json"} + elif isinstance(body, str): + data = body.encode() + headers = headers if has_type else {**headers, "Content-Type": "text/plain; charset=utf-8"} + else: + raise ValueError("http_request: the body is text or a JSON object") + if len(data) > MAX_BODY: + raise ValueError(f"http_request: the body is over {MAX_BODY // 1024} KB") + return data, headers + + +def _method(p): + m = str(p.get("method") or "GET").upper() + if m not in METHODS: + raise ValueError(f"http_request: the method is {', '.join(METHODS)}") + return m + + +def _answer(status, data, ctype, more): + text = data.decode("utf-8", "replace") + out = {"status": status, "ok": 200 <= status < 300, "body": text, "content_type": ctype, "truncated": more, "untrusted": True} + if not more and "json" in ctype.lower(): + try: + out["json"] = json.loads(text) + except ValueError: + pass + return out + + +def _http_request(db, cfg, intent, user, dry): + p = intent["params"] + actions.person(db, intent, user) + method, url = _method(p), _url(p) + headers = _headers(p) + data, headers = _body(method, p.get("body"), headers) + sent = {"method": method, "url": url, "headers": _shown(headers)} + headers = _open(db, cfg, intent, url, headers) + if dry: + return {**sent, "status": None, "would": f"{method} {url}"} + host = urllib.parse.urlsplit(url).hostname + status, error = None, None + try: + status, got, ctype, more = notify.request(cfg, method, url, data, headers, TIMEOUT, TIMEOUT, MAX_ANSWER) + except TimeoutError: + error = f"{host} didn't answer within {TIMEOUT} seconds" + except ValueError as e: + error = str(e) + except (OSError, http.client.HTTPException) as e: + error = f"couldn't reach {host}: {type(e).__name__}: {e}"[:300] + activity.record( + "out", + "automation.http", + [f"space:{s}" for s in intent["spaces"]], + cfg, + db, + ok=error is None and status is not None and status < 400, + error=None if error is None else "unreachable", + detail={"method": method, "host": host, "status": status}, + ) + if error: + raise ValueError(f"http_request: {error}") + return {**sent, **_answer(status, got, ctype, more)} + + +def _describe(p): + m = str(p.get("method") or "GET").upper() + url = str(p.get("url") or "a web address") + return f"{m} {url if len(url) <= 120 else url[:119] + '…'}" + + +actions.register( + actions.ActionSpec( + name="http_request", + label="Call a web address", + group="Web", + description=( + "Sends a GET, POST or PUT to a public web address and passes on its status and answer (up to 64 KB); asks " + "first unless you set it to run on its own. Secrets go in headers, kept sealed." + ), + params={ + "type": "object", + "properties": { + "method": {"type": "string", "title": "Method", "enum": list(METHODS), "default": "GET"}, + "url": {"type": "string", "title": "Address", "maxLength": notify.URL_MAX}, + "headers": { + "type": "object", + "title": "Headers", + "default": {}, + "description": "credentials (Authorization, API keys, tokens) are kept sealed and never shown", + }, + "body": {"title": "Body", "description": "text, or an object sent as JSON (not with GET)"}, + }, + }, + required=("url",), + effect="out", + output={ + "type": "object", + "description": "the answer is outside text: later prompts read it marked as data", + "properties": { + "status": {"type": "integer"}, + "ok": {"type": "boolean"}, + "body": {"type": "string"}, + "json": {"description": "the answer parsed, when it is JSON"}, + "content_type": {"type": "string"}, + "truncated": {"type": "boolean"}, + }, + }, + perform=_http_request, + describe=_describe, + ) +) diff --git a/fastapi_backend/app/domain/notify.py b/fastapi_backend/app/domain/notify.py index 0c4ddf649..8c1580dbc 100644 --- a/fastapi_backend/app/domain/notify.py +++ b/fastapi_backend/app/domain/notify.py @@ -282,6 +282,15 @@ def connect(self): def post(cfg, url, body, headers, timeout=TIMEOUT, deadline=DEADLINE): """POST to a target: (status, the start of its answer), within `deadline` seconds in all. ValueError when the address isn't allowed; OSError and http.client errors when it can't be reached or doesn't answer in time.""" + status, data, _, _ = request(cfg, "POST", url, body, {"Content-Type": "application/json", **headers}, timeout, deadline, 2000) + return status, data.decode("utf-8", "replace") + + +def request(cfg, method, url, body, headers, timeout=TIMEOUT, deadline=DEADLINE, limit=2000): + """One HTTP request to a public address (or a network an admin allowed, `networks`), pinned to the address that was + checked, without following redirects: (status, the first `limit` bytes of the answer, its Content-Type, whether + there was more), within `deadline` seconds in all. ValueError when the address isn't allowed; OSError and + http.client errors when it can't be reached or doesn't answer in time.""" u = urllib.parse.urlsplit(url) port = u.port or (443 if u.scheme == "https" else 80) nets = networks(cfg) @@ -312,9 +321,10 @@ def cut(): # past the deadline, a target that trickles its answer is cut off timer.daemon = True timer.start() try: - conn.request("POST", path, body=body, headers={"User-Agent": UA, "Content-Type": "application/json", **headers}) + conn.request(method, path, body=body, headers={"User-Agent": UA, **headers}) r = conn.getresponse() - answer = r.status, r.read(2000).decode("utf-8", "replace") + data = r.read(limit + 1) + answer = r.status, data[:limit], r.getheader("Content-Type") or "", len(data) > limit except (OSError, http.client.HTTPException): if time.monotonic() - started >= deadline: raise TimeoutError(f"no full answer within {deadline} seconds") from None diff --git a/fastapi_backend/tests/api/test_actions_web.py b/fastapi_backend/tests/api/test_actions_web.py new file mode 100644 index 000000000..c56e8c231 --- /dev/null +++ b/fastapi_backend/tests/api/test_actions_web.py @@ -0,0 +1,188 @@ +"""The automation action that calls a web address (docs/automations.md, Actions): http_request reaches public addresses +(or networks an admin allowed) only, asks first unless the node runs on its own, waits 10 seconds at most, passes on +up to 64 KB of the answer as outside text, and keeps credentials in headers sealed to the automation and the host.""" + +from __future__ import annotations + +import http.server +import json +import threading +import time + +import pytest + +from app.domain import actions, actions_web, keyring, store +from tests import fake_llm +from tests.api._assist import start_llm +from tests.helpers import make_user, seed + + +class Site: + """A local web server: answers every request with `answer` (status, content type, body) and keeps what it got.""" + + def __init__(self): + self.got, self.answer, self.wait, self.location = [], (200, "application/json", b'{"temp": 21}'), 0, None + site = self + + class H(http.server.BaseHTTPRequestHandler): + def _any(self): + body = self.rfile.read(int(self.headers.get("Content-Length") or 0)) + site.got.append( + {"method": self.command, "path": self.path, "headers": {k.lower(): v for k, v in self.headers.items()}, "body": body} + ) + time.sleep(site.wait) + status, ctype, data = site.answer + self.send_response(status) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(data))) + if site.location: + self.send_header("Location", site.location) + self.end_headers() + self.wfile.write(data) + + do_GET = do_POST = do_PUT = _any # noqa: N815 - the stdlib's names + + def log_message(self, *a): + pass + + self.srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), H) + threading.Thread(target=self.srv.serve_forever, daemon=True).start() + self.url = f"http://127.0.0.1:{self.srv.server_address[1]}" + + +@pytest.fixture +def site(): + s = Site() + yield s + s.srv.shutdown() + + +@pytest.fixture +def app(cfg, db): + from app.main import create_app + + srv = start_llm(cfg) + yield create_app(cfg, db, background=False) + fake_llm.Handler.usage = None + srv.shutdown() + + +@pytest.fixture +def people(app, db, cfg, folder): + seed(db, cfg, folder) + cfg["notifications"] = {**(cfg.get("notifications") or {}), "networks": ["127.0.0.0/8"]} + return {"me": make_user(db, "me@x.io", "my password 1", roles={"pods": "editor"}), "pods": store.ns_id(db, "pods")} + + +def run(db, cfg, people, params, dry=False, **kw): + intent = { + "key": "51-n4", + "action": "http_request", + "params": params, + "effect": "out", + "risk": "ask", + "untrusted": False, + "spaces": [people["pods"]], + "run_as": people["me"], + "node": "n4", + "automation": 4, + "run": 51, + **kw, + } + return actions.perform(db, cfg, intent, None, dry) + + +def test_it_reaches_outside_lens_so_it_asks_unless_the_node_runs_on_its_own(): + spec = actions.get("http_request") + assert spec.group == "Web" and actions.effect_of(spec, {}) == "out" + assert actions.resolve_risk(spec, {"url": "https://x.example"}, {"node": "n4"}) == "ask" + assert actions.resolve_risk(spec, {"url": "https://x.example"}, {"node": "n4"}, {"n4": "auto"}) == "routine" + assert actions.check_params(spec, {"url": "https://x.example"}) == {"method": "GET", "url": "https://x.example", "headers": {}} + assert actions.describe("http_request", {"method": "post", "url": "https://x.example/hook"}) == "POST https://x.example/hook" + + +def test_it_calls_an_address_and_passes_on_the_answer_as_outside_text(db, cfg, people, site): + got = run(db, cfg, people, {"url": site.url + "/weather?city=Oslo", "headers": {"Accept": "application/json"}}) + assert got["status"] == 200 and got["ok"] and got["json"] == {"temp": 21} and got["body"] == '{"temp": 21}' + assert got["untrusted"] and not got["truncated"] and got["headers"] == {"Accept": "application/json"} + assert site.got[-1]["method"] == "GET" and site.got[-1]["path"] == "/weather?city=Oslo" and site.got[-1]["headers"]["user-agent"] + # an object is sent as JSON, text as it is + run(db, cfg, people, {"method": "POST", "url": site.url + "/hook", "body": {"subject": "Trip on Friday"}}) + assert site.got[-1]["headers"]["content-type"] == "application/json" and json.loads(site.got[-1]["body"]) == { + "subject": "Trip on Friday" + } + site.answer = (201, "text/plain", b"made") + got = run(db, cfg, people, {"method": "PUT", "url": site.url + "/x", "body": "plain words"}) + assert ( + site.got[-1]["method"] == "PUT" + and site.got[-1]["body"] == b"plain words" + and site.got[-1]["headers"]["content-type"].startswith("text/plain") + ) + assert got["status"] == 201 and got["body"] == "made" and "json" not in got + row = db.one("SELECT * OMIT id FROM activity WHERE action = 'automation.http' ORDER BY at DESC LIMIT 1") + assert row["kind"] == "out" and row["detail"] == {"method": "PUT", "host": "127.0.0.1", "status": 201} and row["ok"] + # a dry run sends nothing + sent = len(site.got) + got = run(db, cfg, people, {"method": "POST", "url": site.url + "/hook", "body": "x"}, dry=True) + assert got["would"] == f"POST {site.url}/hook" and got["status"] is None and len(site.got) == sent + + +def test_the_answer_is_capped_redirects_are_not_followed_and_it_waits_ten_seconds_at_most(db, cfg, people, site, monkeypatch): + site.answer = (200, "application/json", b"[" + b"1," * 40000 + b"1]") + got = run(db, cfg, people, {"url": site.url}) + assert len(got["body"]) == 64 * 1024 and got["truncated"] and "json" not in got + site.answer, site.location = (302, "text/plain", b""), "http://169.254.169.254/latest/meta-data/" + got = run(db, cfg, people, {"url": site.url}) + assert got["status"] == 302 and not got["ok"] and len(site.got) == 2 # the redirect isn't followed + site.answer, site.location, site.wait = (200, "text/plain", b"late"), None, 1.5 + monkeypatch.setattr(actions_web, "TIMEOUT", 1) + with pytest.raises(ValueError, match="127.0.0.1 didn't answer within 1 seconds"): + run(db, cfg, people, {"url": site.url}) + assert ( + actions_web.TIMEOUT == 1 + and not db.one("SELECT ok, at FROM activity WHERE action = 'automation.http' ORDER BY at DESC LIMIT 1")["ok"] + ) + + +def test_only_public_addresses_unless_an_admin_allowed_the_network(db, cfg, people, site): + cfg["notifications"]["networks"] = [] + with pytest.raises(ValueError, match="127.0.0.1 isn't a public address"): + run(db, cfg, people, {"url": site.url}) + assert not site.got + for params, said in ( + ({"url": "ftp://x.example"}, "an http or https URL"), + ({"url": "https://me:pw@x.example"}, "leave the user name and password out"), + ({"url": "https://x.example", "method": "DELETE"}, "method is one of GET, POST, PUT"), + ({"url": "https://x.example", "body": "x"}, "a GET sends no body"), + ({"url": "https://x.example", "headers": {"Host": "intranet"}}, "Host isn't a header it can set"), + ({"url": "https://x.example", "headers": {"X-A": "a\r\nX-B: b"}}, "the X-A header is one line"), + ({"url": "https://x.example", "method": "POST", "body": "x" * (65 * 1024)}, "the body is over 64 KB"), + ): + with pytest.raises(ValueError, match=said): + run(db, cfg, people, params, dry=True) + + +def test_credentials_in_headers_are_sealed_to_the_automation_and_the_host_and_never_shown(db, cfg, people, site): + kept = actions_web.seal_headers( + db, cfg, people["pods"], 4, site.url + "/hook", {"Authorization": "Bearer s3cret", "X-Api-Key": "k-1", "Accept": "text/plain"} + ) + assert keyring.is_sealed(kept["Authorization"]) and keyring.is_sealed(kept["X-Api-Key"]) and kept["Accept"] == "text/plain" + assert "s3cret" not in json.dumps(kept) + assert actions_web.seal_headers(db, cfg, people["pods"], 4, site.url, kept) == kept # sealed once + got = run(db, cfg, people, {"method": "POST", "url": site.url + "/other-path", "headers": kept, "body": {"a": 1}}) + assert site.got[-1]["headers"]["authorization"] == "Bearer s3cret" and site.got[-1]["headers"]["x-api-key"] == "k-1" + assert got["headers"] == {"Authorization": "••••", "X-Api-Key": "••••", "Accept": "text/plain", "Content-Type": "application/json"} + assert "s3cret" not in json.dumps(got) and "s3cret" not in actions.describe("http_request", {"url": site.url, "headers": kept}) + # typed in plain, a credential is sent but still not shown + got = run(db, cfg, people, {"url": site.url, "headers": {"Authorization": "Bearer typed"}}) + assert got["headers"] == {"Authorization": "••••"} and site.got[-1]["headers"]["authorization"] == "Bearer typed" + # another automation, another host or another namespace can't use them + sent = len(site.got) + with pytest.raises(ValueError, match="sealed for another automation or address"): + run(db, cfg, people, {"url": site.url, "headers": kept}, automation=5) + with pytest.raises(ValueError, match="sealed for another automation or address"): + run(db, cfg, people, {"url": site.url.replace("127.0.0.1", "localhost"), "headers": kept}) + calls = store.ns_id(db, "calls") + with pytest.raises(ValueError, match="sealed for another namespace"): + run(db, cfg, people, {"url": site.url, "headers": kept}, spaces=[calls]) + assert len(site.got) == sent diff --git a/threat-model.json b/threat-model.json index 9885e8eff..7bf82cc45 100644 --- a/threat-model.json +++ b/threat-model.json @@ -973,7 +973,7 @@ "property": "P-SECRETS", "violation_symptom": "info-leak", "severity_tier": "security", - "conditions": "stored secrets (source credentials, notification secrets, model API keys, iCal addresses) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server", + "conditions": "stored secrets (source credentials, notification secrets, model API keys, iCal addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server", "provenance": "documented", "source": "SECURITY.md \"stored credentials for sources and model servers leaking\"" }, @@ -981,7 +981,7 @@ "property": "P-SSRF-GUARD", "violation_symptom": "info-leak", "severity_tier": "security", - "conditions": "URLs a non-admin sets (web import and extension http tools by editors; notification targets by owners) reach public addresses only, or networks the operator listed; web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address", + "conditions": "URLs a non-admin sets (web import and extension http tools by editors; notification targets by owners; an automation's http_request by its person, within the notification networks) reach public addresses only, or networks the operator listed; web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address; an automation's http_request follows no redirect, takes 10 seconds and reads 64 KB at most, asks first unless the person set its node to run on its own, and passes the answer on as outside text", "provenance": "documented", "source": "maintainer ruling, 2026-10" }, diff --git a/threat-model.md b/threat-model.md index 8c8e929b5..faae1de06 100644 --- a/threat-model.md +++ b/threat-model.md @@ -361,7 +361,7 @@ Every row is claimed, disclaimed, N/A with a reason, or unresolved. Family short | Outbound | Numeric domain | claimed | web paths and iCal: ports 80 and 443 | P-SSRF-GUARD, P-ADMIN-GUARD | *(documented, docs/api.md "Only public addresses are reached, on ports 80 and 443")* | | Outbound | Numeric domain: notification ports | unresolved | notification targets may use any port; the guard checks addresses only | Q15 | *(inferred, Q15)* | | Outbound | Failure atomicity | claimed | each notification event claimed once across processes, then retried | P-NOTIFY | *(documented, docs/notifications.md "claimed once")* | -| Outbound | Recursive / cyclic topology (redirects) | claimed | http and https only, each hop re-checked; credentials never follow a redirect to another server; notifications follow none | P-SSRF-GUARD, P-SECRETS | *(documented, docs/processing.md "never sent on to another server")* | +| Outbound | Recursive / cyclic topology (redirects) | claimed | http and https only, each hop re-checked; credentials never follow a redirect to another server; notifications and an automation's `http_request` follow none | P-SSRF-GUARD, P-SECRETS | *(documented, docs/processing.md "never sent on to another server")* | | Outbound | Callback / collaborator execution | disclaimed | Chromium may lack its sandbox; the proxy is the barrier | N-CHROMIUM | *(documented, docs/deployment.md "what stops a page is Lens's proxy")* | | Outbound | Serialization / reconstruction | claimed | sealed secrets are write-only through the API | P-SECRETS | *(documented, docs/configuration.md "the API reports whether one is set, never its value")* | | Outbound | x-admin-operand-guards | claimed | iCal public-address guard; telemetry endpoint without user:password or link-local address | P-ADMIN-GUARD | *(documented, docs/processing.md "fetched the way web pages are captured")* | @@ -508,8 +508,8 @@ Paths in the Voided by column are relative to `fastapi_backend/app/`. Searches c | P-HOST | The `Host` header must be a served host. This stops DNS rebinding. | bad-data-accepted | correctness-only | *(documented, docs/configuration.md "stops DNS rebinding")* | Voided by: `*` in `server.allowed_hosts` — `core/middleware.py:70`; `ARCHIVE_ALLOWED_HOSTS` — `domain/settings.py:129`. | | P-FRAME | The API sends a strict CSP. Only the player (`/embed/`, `/s/`) can be framed, and only by `server.embed_frame_ancestors`. | integrity-bypass | security-critical | *(documented, docs/configuration.md "can be framed")* | Voided by: `server.embed_frame_ancestors` — `core/middleware.py:72` (default `'self'`, `domain/store.py:185`). | | P-HARDENED | In the compose files the database, API and Fedora ports bind to 127.0.0.1, containers run non-root with `no-new-privileges`, and the web app sends `nosniff`, a referrer policy, a permissions policy and a frame policy (HSTS through Cloudflare). In `docker-compose.prod.yml` the DB password is random and the web app listens on this machine unless `LENS_BIND` is set. | x-missing-hardening | correctness-only | *(documented, docs/deployment.md "What's hardened already")* | Voided by: `LENS_BIND=0.0.0.0` — `docker-compose.prod.yml:153`, which an admin can also ask for from Settings (`PUT /api/v1/admin/install`); dev web port — `docker-compose.yml:95` (§1.3). | -| P-SECRETS | Stored secrets (source credentials, notification secrets, model API keys, the Hugging Face token, iCal addresses) are encrypted and write-only. The API reports whether one is set, never its value. A key from the environment is never shown. They do not leak through the API, logs, exports, or a redirect to another server. | info-leak | security-critical | *(documented, SECURITY.md "stored credentials for sources and model servers leaking")* | Voided by: `secret.key` or `ARCHIVE_SECRET_KEY` holders — `domain/settings.py:196-199` (Q14). An admin moving a model URL sends the key there — `domain/llm.py:30-32` (trusted). LLM errors name the URL as typed — `domain/llm.py:41` (Q18). | -| P-SSRF-GUARD | URLs a non-admin sets (web import and extension `http` tools by editors; notification targets by owners) reach public addresses only, or networks the operator or an admin listed (an admin, in Settings, never loopback or link-local ones). Web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address. | info-leak | security-critical | *(maintainer, 2026-10)* | Voided by: Operator networks — `domain/store.py:475-476`, `domain/netguard.py:40-41`; admin web networks — `domain/settings.py` `_web_networks`; admin notification networks — `domain/notify.py:286-288`; Chromium without sandbox — `domain/convert.py:273`. Search: `grep -rlE 'public_ip\|netguard\.(resolve\|Guard)\|web_networks' fastapi_backend/app` — `netguard.py`, `webcapture.py`, `feeds.py`, `extensions.py`, `notify.py`, `convert.py`, `store.py`, `routes/imports.py`; none disables the check. | +| P-SECRETS | Stored secrets (source credentials, notification secrets, model API keys, the Hugging Face token, iCal addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only. The API reports whether one is set, never its value. A key from the environment is never shown. They do not leak through the API, logs, exports, or a redirect to another server. | info-leak | security-critical | *(documented, SECURITY.md "stored credentials for sources and model servers leaking")* | Voided by: `secret.key` or `ARCHIVE_SECRET_KEY` holders — `domain/settings.py:196-199` (Q14). An admin moving a model URL sends the key there — `domain/llm.py:30-32` (trusted). LLM errors name the URL as typed — `domain/llm.py:41` (Q18). | +| P-SSRF-GUARD | URLs a non-admin sets (web import and extension `http` tools by editors; notification targets by owners; an automation's `http_request` by its person, within the notification networks) reach public addresses only, or networks the operator or an admin listed (an admin, in Settings, never loopback or link-local ones). Web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address. An automation's `http_request` follows no redirect, takes 10 seconds and reads 64 KB at most, asks first unless the person set its node to run on its own, and passes the answer on as outside text. | info-leak | security-critical | *(maintainer, 2026-10)* | Voided by: Operator networks — `domain/store.py:475-476`, `domain/netguard.py:40-41`; admin web networks — `domain/settings.py` `_web_networks`; admin notification networks — `domain/notify.py:286-288`; Chromium without sandbox — `domain/convert.py:273`. Search: `grep -rlE 'public_ip\|netguard\.(resolve\|Guard)\|web_networks' fastapi_backend/app` — `netguard.py`, `webcapture.py`, `feeds.py`, `extensions.py`, `notify.py`, `convert.py`, `store.py`, `routes/imports.py`; none disables the check. | | P-ADMIN-GUARD | Guards Lens documents on admin-set operands hold. iCal feeds are fetched like web captures: public addresses only (and `documents.web_networks`), ports 80 and 443, and the password never goes on to a server the calendar redirects to *(documented, docs/processing.md "fetched the way web pages are captured")*. The telemetry endpoint cannot carry a user name or password or be a link-local address *(documented, domain/settings.py validation message "can't be a link-local address")*. | info-leak | security-critical | *(documented, docs/processing.md "fetched the way web pages are captured")* | Voided by: `documents.web_networks` widens the iCal guard — `domain/feeds.py:377`; telemetry checks — `domain/settings.py:751`, `domain/settings.py:757`. | | P-NO-FETCH | SPARQL refuses SERVICE and FROM and never loads a named graph. RDF import fetches nothing, refuses remote JSON-LD contexts and does not read RDF/XML. | info-leak | security-critical | *(documented, rdf_import.py docstring "Nothing is fetched while reading")* | Voided by: Process-global rdflib flag, set per call — `domain/rdf.py:576`; SERVICE refused — `domain/rdf.py:551`; remote contexts — `domain/rdf_import.py:54-56`. | | P-QUERY | Cypher is read-only and runs on the caller's readable namespaces. A query stops within 3 million steps or 8 seconds; paths run up to 8 hops. Threshold: running past the budget is a bug; slow within it is not. | integrity-bypass (writes, scope); hang (budget) | security-critical (scope, writes); correctness-only (budget) | *(documented, docs/graph.md "3 million steps")* | Voided by: Admins see every namespace; write refusals — `domain/cypher.py:39`; budgets — `domain/cypher.py:35-38`. | diff --git a/threat-model.yaml b/threat-model.yaml index b911b0b9b..b418d582e 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:6957510fc090d277933c6b0708ea72227197a1e9b2fa267b40b883caf70bfe0e" +prose_version: "threat-model.md@sha256:6f2e5e104620679f5a4d2f0e232a9a4cfdfab594ce55758f4d186bcae9a9492f" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -572,7 +572,7 @@ contract_dimensions: - {component: outbound, dimension: numeric-domain, status: claimed, property_id: P-SSRF-GUARD, conditions: "web paths and iCal: ports 80 and 443", x-routes-to: "P-SSRF-GUARD, P-ADMIN-GUARD", provenance: {kind: documented, source: 'docs/api.md "Only public addresses are reached, on ports 80 and 443"'}} - {component: outbound, dimension: x-numeric-domain-notification-ports, status: unresolved, conditions: "notification targets may use any port; the guard checks addresses only", question_id: Q15, provenance: {kind: inferred, question_id: Q15}} - {component: outbound, dimension: failure-atomicity, status: claimed, property_id: P-NOTIFY, conditions: "each notification event claimed once across processes, then retried", provenance: {kind: documented, source: 'docs/notifications.md "claimed once"'}} - - {component: outbound, dimension: recursive-cyclic-topology, status: claimed, property_id: P-SSRF-GUARD, conditions: "redirects: http and https only, each hop re-checked; credentials never follow a redirect to another server; notifications follow none", x-routes-to: "P-SSRF-GUARD, P-SECRETS", provenance: {kind: documented, source: 'docs/processing.md "never sent on to another server"'}} + - {component: outbound, dimension: recursive-cyclic-topology, status: claimed, property_id: P-SSRF-GUARD, conditions: "redirects: http and https only, each hop re-checked; credentials never follow a redirect to another server; notifications and an automation's http_request follow none", x-routes-to: "P-SSRF-GUARD, P-SECRETS", provenance: {kind: documented, source: 'docs/processing.md "never sent on to another server"'}} - {component: outbound, dimension: callback-execution, status: disclaimed, property_id: N-CHROMIUM, conditions: "Chromium may lack its sandbox; the proxy is the barrier", provenance: {kind: documented, source: 'docs/deployment.md "what stops a page is Lens''s proxy"'}} - {component: outbound, dimension: serialization-reconstruction, status: claimed, property_id: P-SECRETS, conditions: "sealed secrets are write-only through the API", provenance: {kind: documented, source: 'docs/configuration.md "the API reports whether one is set, never its value"'}} - {component: outbound, dimension: x-admin-operand-guards, status: claimed, property_id: P-ADMIN-GUARD, conditions: "iCal public-address guard; telemetry endpoint without user:password or link-local address", provenance: {kind: documented, source: 'docs/processing.md "fetched the way web pages are captured"'}} @@ -1401,7 +1401,7 @@ properties_claimed: kind: confidentiality components: [outbound] tier: security-critical - conditions: "stored secrets (source credentials, notification secrets, model API keys, iCal addresses) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server" + conditions: "stored secrets (source credentials, notification secrets, model API keys, iCal addresses, an automation's credential headers, sealed to that automation and the address they were saved for) are encrypted and write-only; the API reports whether one is set, never its value; a key from the environment is never shown; they do not leak through the API, logs, exports, or a redirect to another server" violation_symptoms: [info-leak] x-voided-by: "secret.key or ARCHIVE_SECRET_KEY holders (domain/settings.py:196-199, Q14); an admin moving a model URL sends the key there (domain/llm.py:30-32, trusted); LLM errors name the URL as typed (domain/llm.py:41, Q18)" provenance: {kind: documented, source: 'SECURITY.md "stored credentials for sources and model servers leaking"'} @@ -1409,7 +1409,7 @@ properties_claimed: kind: confidentiality components: [outbound, assistant] tier: security-critical - conditions: "URLs a non-admin sets (web import and extension http tools by editors; notification targets by owners) reach public addresses only, or networks the operator listed; web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address" + conditions: "URLs a non-admin sets (web import and extension http tools by editors; notification targets by owners; an automation's http_request by its person, within the notification networks) reach public addresses only, or networks the operator listed; web paths use ports 80 and 443, follow redirects only to http and https, re-check every redirect and sub-request, and connect to the checked address; an automation's http_request follows no redirect, takes 10 seconds and reads 64 KB at most, asks first unless the person set its node to run on its own, and passes the answer on as outside text" violation_symptoms: [info-leak] x-voided-by: "operator networks (domain/store.py:475-476, domain/netguard.py:40-41); admin notification networks (domain/notify.py:286-288); Chromium without sandbox (domain/convert.py:273)" provenance: {kind: maintainer, date: "2026-10"}