From 032ee38308c65fb82f63d93425f6e57c1784dca2 Mon Sep 17 00:00:00 2001 From: Adeel Ahmad Date: Thu, 15 Oct 2026 23:35:20 +0000 Subject: [PATCH] feat(automations): sensor readings, alert patterns and new devices as activities sensor.reading (noisy) reads the readings a sensor keeps, one sensor at a time over the (sensor, at) index and only the sensors a follower's hint names; sensor.alert is raised by the intake once per pattern per hour for patterns labelled alert or notable; sensor.joined reads sensors discovery made. The timeline breaks published ties by each source's own position. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01QewNs6jxSB2vRoag2EvdyX --- docs/automations.md | 50 +- docs/sensors.md | 4 + fastapi_backend/app/domain/events.py | 4 +- fastapi_backend/app/domain/events_streams.py | 476 +++++++++++++++++- fastapi_backend/app/domain/sensor_patterns.py | 5 +- fastapi_backend/app/domain/sensors.py | 34 ++ fastapi_backend/app/domain/store.py | 3 + .../tests/domain/test_events_streams.py | 193 +++++++ threat-model.json | 2 +- threat-model.md | 2 +- threat-model.yaml | 6 +- 11 files changed, 759 insertions(+), 20 deletions(-) create mode 100644 fastapi_backend/tests/domain/test_events_streams.py diff --git a/docs/automations.md b/docs/automations.md index 9e120cfce..5b34b303a 100644 --- a/docs/automations.md +++ b/docs/automations.md @@ -15,7 +15,8 @@ merged is called built. | New assets by kind: emails received, calendar events, files from storage, uploads, pages saved from the web, imports | built | | Runs that finished or failed, deletions and calendar events that moved | built | | Comments, notes written and saved, and changes to the entity graph | built | -| More kinds: sensors, chat, system events | planned | +| Sensors: readings a sensor keeps, alert and notable log patterns, devices that report for the first time ([Sensors](#sensors-as-activities)) | built | +| More kinds: chat, system events | planned | | The timeline API (`GET /api/v1/activities`), the MCP tools `timeline` and `event_kinds`, the assistant's `timeline` | built | | Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built | | The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built | @@ -102,6 +103,9 @@ and the paths holding outside text are in `GET /api/v1/activities/kinds` (`event | `note.created` | | `note_pages` | a `Page`: `summary`, `about` (what it's the page of), `place`, `author` (person or assistant) | Assistant wrote the note “St Mary's School” | | `note.saved` | | `note_versions` | as `note.created`, with `version` (the page as it was before) and `why` | Sam edited the note “School trip plans” | | `entity.changed` | | `graph` | `op` (created, renamed, merged, split, linked, unlinked, deleted, rolled back, changed), `change` (the graph's own op), `entityType`, `before` (a renamed entity's name), `version`, `count`; tagged with its entities | Sam renamed “St Marys” to “St Mary's School” | +| `sensor.reading` | | `sensor_readings` | `{type: Note, id: lens:sensor_reading/, name: ": ", url: /sensors/}`, `stream`, `value`, `text`, `fields` (a JSON reading's numbers), `level` (syslog), `pattern`, `label` (the line's pattern and its label); noisy | Kitchen sensor temperature: 31.2 | +| `sensor.alert` | | `log` | `{type: Note, id: lens:sensor_pattern/}`, `label` (alert or notable), `pattern`, `example` (a line it fits), `count` (lines in all), `lines` (in the batch that raised it), `hour` | 192.168.1.1 logged an alert: wlan: authentication failed for | +| `sensor.joined` | | `sensors` | `{type: Service, id: lens:sensor/, url: /sensors/}`, `sensor_type` (mqtt, syslog, ...), `device`, `status` | A new MQTT device reported: zigbee2mqtt/kitchen | | `webhook.received` | | `log` | `{type: Note, hook: the webhook's id, payload: what was sent (JSON, a form's fields or text), content_type, query}`; the actor is the webhook (`Service`, `lens:automation_hook/`, "Webhook for ") | Webhook for Front door was called | A new asset is one activity, of the most specific kind its row tells, decided by its content type and where it came @@ -135,6 +139,25 @@ its text once it has been read (`recording.email`). Emails and events imported b fields their rows had. Outside text (titles, subjects, addresses, snippets, places, file names, page addresses) is listed in `lens.untrusted`. +#### Sensors as activities + +Built (`app/domain/events_streams.py`). A sensor's actor is the sensor (`Service`, `lens:sensor/`, its name); a +sensor with no namespace makes activities for admins, as its page is. Its names, streams, text and patterns are +outside text. + +- `sensor.reading` is a reading the sensor keeps, as its handling stores them: `all`, `changes` (a change, and once + an hour while a value stays the same), none for `summary` or `none`, and none while it is paused or ignored. To + trigger on every change, set the sensor to store changes. It is noisy: the timeline leaves it out unless asked for. + Readings are read one sensor at a time over the index on `(sensor, at)`, and a follower whose hint names sensors + (`{"actors": ["lens:sensor/5"]}`, which an automation's trigger on one sensor gives) reads only theirs, so a busy + sensor nobody listens to is never read. +- `sensor.alert` is a log pattern ([Sensors](sensors.md#log-patterns-and-triage)) labelled alert or notable that new + lines fit, raised by the intake as it writes them: once per pattern per hour (the event log's key is the pattern + and the hour, so two processes raise it once), whether the lines are kept or dropped. A label the decision model + wasn't sure of raises none until a person settles it, and a paused or ignored sensor raises none. +- `sensor.joined` is a device or host that made itself a sensor by reporting for the first time (MQTT, syslog), + read from its `storage_source` row by `created_at`; a sensor added by hand isn't news. + ### Sources and the event log Built. A source (`events.Source`, added with `events.register_source`) reads rows Lens already keeps as activities, @@ -148,7 +171,7 @@ so nothing is stored twice: - `get(db, cfg, activity_id)`: one activity by its id, or null; `now(db)`, `back(db, seconds)`: positions of now and of `seconds` ago; `key(a)`: the position `page` orders by. -A kind belongs to the source named in its `source`, and a source may list more in `kinds`. Two are built: +A kind belongs to the source named in its `source`, and a source may list more in `kinds`. These are built: - `recordings` (`app/domain/events_content.py`) reads `recording` by `created_at`, the id breaking ties, as `asset.added` and its children (Kinds, above). The actor is whoever added it: an automation (its first job was @@ -164,6 +187,9 @@ A kind belongs to the source named in its `source`, and a source may list more i - `comments` reads `comment` by `created_at`; `note_pages` reads `note_page` by `created_at`; `note_versions` reads `note_version` by `at` (to the millisecond) and keeps one version for each page and window; `graph` reads `graph_event` by `at`, the event number breaking ties (numbers are given out in order, so the two agree). +- `sensor_readings` (`app/domain/events_streams.py`) reads `sensor_reading` by `at` (microseconds), the id breaking + ties, one sensor at a time, as `sensor.reading`; `sensors` reads the `storage_source` rows discovery made by + `created_at`, as `sensor.joined`. - `log` reads `event_log`: activities no row of their own records (a deletion, a webhook call), written by `events.emit(db, kind, *, actor, obj, space, summary, target=None, tag=None, key=None, published=None, spaces=None, audience=None, owner=None, untrusted=None)`, which gives the id (`log-`). Emitting a `key` again changes @@ -195,10 +221,11 @@ where the next look starts (`event_cursor:`, a position per source). ### Timeline reads Built. `events.page(db, cfg, viewer, before=None, flt=None, limit=50)` reads the stream newest first for a viewer -(`events.viewer(account, spaces, admin)`, spaces null for every one): `{"items": [...], "next": ...}`, where `next` -(an opaque cursor, null at the end) goes in `before` for the page after. It merges every source's `page`, applies -who may see what, and when filters thin a page out it reads on, up to five pages from each source. -`events.get(db, cfg, id, viewer)` reads one activity, null when there is none or it isn't the viewer's to see. +(`events.viewer(account, spaces, admin)`, spaces null for every one): `{"items": [...], "next": ...}`, where `next` (an +opaque cursor, null at the end) goes in `before` for the page after. It merges every source's `page` by `published` +(each source's own position breaking ties), applies who may see what, and when filters thin a page out it reads on, up +to five pages from each source. `events.get(db, cfg, id, viewer)` reads one activity, null when there is none or it +isn't the viewer's to see. - Who sees what: `lens.audience` `owner` is its owner alone; `admins` is admins; `space` is everyone who reads one of its namespaces (`space` or `lens.spaces`); admins read every namespace, and an activity of no namespace is for @@ -287,10 +314,10 @@ phone shortcut, another service. No webhook exists until an automation with such - What was sent becomes a `webhook.received` activity in the event log, with the payload, its content type and the query as outside text. Its namespace is the automation's (admins see it while the automation isn't stored). - Signed calls: a caller that signs the Standard Webhooks way (`webhook-id`, `webhook-timestamp` in seconds, - `webhook-signature: v1,..">` keyed with the secret, as Lens signs its - own [notification webhooks](notifications.md#webhooks)) has the signature checked, and the time must be within five minutes. - A hook made with `signed=True` refuses calls that aren't signed. A call naming its message (`webhook-id`, or an - `Idempotency-Key` header) is logged once however often the sender retries it. + `webhook-signature: v1,..">` keyed with the secret, as Lens signs its own + [notification webhooks](notifications.md#webhooks)) has the signature checked, and the time must be within five + minutes. A hook made with `signed=True` refuses calls that aren't signed. A call naming its message (`webhook-id`, or + an `Idempotency-Key` header) is logged once however often the sender retries it. - In Docker the API runs no background work, so the worker's automations see the call on their next look (5 seconds, `events.poll_seconds`). @@ -493,3 +520,6 @@ the agent can draft an event as a note. - Writing calendar events (a new Google consent, so it changes existing installs). - Webhook trigger: signed answers to the caller (what the run did); a rate limit shared by every API process (today each counts its own); showing a hook's last call in the studio. +- Sensors: a threshold per reading in the intake (only readings over a value become activities, for sensors that + keep everything); a reading's unit, which sensors don't record yet; an alert as soon as a pattern is labelled, for + the lines already seen that hour. diff --git a/docs/sensors.md b/docs/sensors.md index eb74ecfd5..809c4a41c 100644 --- a/docs/sensors.md +++ b/docs/sensors.md @@ -98,6 +98,10 @@ the language model when there's no Jev key) to label the busiest unlabelled patt pattern, never per line. A label it wasn't sure of (below `decisions.act_above`) is kept with `sure: false`, for you to settle. +Automations see sensors too ([Automations](automations.md#sensors-as-activities)): the readings a sensor keeps as +`sensor.reading`, a pattern labelled alert or notable as `sensor.alert` (once an hour while lines fit it), and a device +that reports for the first time as `sensor.joined`. + ## Daily digests With `digest` on (and a namespace), each full day (UTC) of a sensor becomes a document in its namespace: each stream's diff --git a/fastapi_backend/app/domain/events.py b/fastapi_backend/app/domain/events.py index 77ba89acc..02080aaf5 100644 --- a/fastapi_backend/app/domain/events.py +++ b/fastapi_backend/app/domain/events.py @@ -1033,8 +1033,8 @@ def fits(a): def ok(a): return fits(a) and (keep is None or a["id"] in kept) - def k(s, a): - return (a["published"], s.name, _ikey(s.key(a)[1])) + def k(s, a): # the source's own position breaks ties, so rows a millisecond apart keep its order + return (a["published"], s.name, pos_key(s.key(a))) def fetch(s): rows = s.page(db, cfg, at[s.name], who, sflt, limit)[:limit] diff --git a/fastapi_backend/app/domain/events_streams.py b/fastapi_backend/app/domain/events_streams.py index 6b63b5359..26b311859 100644 --- a/fastapi_backend/app/domain/events_streams.py +++ b/fastapi_backend/app/domain/events_streams.py @@ -2,13 +2,32 @@ file. Every kind here has its source named in the catalog; those read from the event log (`log`) are emitted where the thing happens. +- `sensor.reading` (noisy; source `sensor_readings`): the readings a stream sensor keeps (`sensor_reading`, as its + handling stores them: all, changes, or none for summary), read by (at, id) one sensor at a time over the index on + (sensor, at), and only the sensors a follower's hint names (`{"actors": ["lens:sensor/5"]}`), so a busy sensor + nobody listens to is never read. Their text and the sensor's names are outside text. +- `sensor.alert` (log): a log line whose pattern is labelled alert or notable, emitted by the intake (sensors.py) + once per pattern per hour, whatever the sensor keeps of its lines. +- `sensor.joined` (source `sensors`): a device or host that made itself a sensor by reporting (`storage_source` rows + discovery made), by `created_at`. - `webhook.received`: a call to an automation's webhook address (automation_hooks.py), emitted by the route that takes it. Its payload and query are outside text. + +A sensor with no namespace makes activities for admins (`lens.audience`), as its page is. """ from __future__ import annotations -from . import events +import datetime as dt +import re + +from . import events, events_content, sensors, store + +R = store.R +SENSOR_RX = re.compile(r"^lens:sensor/(\d+)$") +READING_RX = re.compile(r"^reading-([a-z0-9]{1,100})$") +JOINED_RX = re.compile(r"^joined-sensor-(\d+)$") +TEXT_MOST = 1000 # characters of a reading's text an activity carries def _s(title, typ="string", **more): @@ -59,3 +78,458 @@ def _s(title, typ="string", **more): }, }, ) + + +# ---------- sensors ---------- +def _stamp(t): + """A time as sensor readings write theirs (sensors._ts: microseconds, +00:00), so they compare as text.""" + return sensors._ts(events._parse(t)) + + +def _hinted(hint): + """The sensor ids a follower's hint names, or None when it names none (then every sensor is read).""" + ids = {int(m.group(1)) for m in (SENSOR_RX.match(str(a)) for a in (hint or {}).get("actors") or []) if m} + return ids or None + + +def _sensors(db, ids=None): + """Stream sensors: {id: {name, space, type}}, those in `ids` when given.""" + rows = db.rows("SELECT record::id(id) AS id, name, space, type FROM storage_source WHERE type IN $t", t=sorted(sensors.STREAM_TYPES)) + return {r["id"]: r for r in rows if ids is None or r["id"] in ids} + + +def _actor(sid, row): + return { + "type": "Service", + "id": f"lens:sensor/{sid}", + "name": " ".join(str((row or {}).get("name") or "").split())[:200] or f"Sensor {sid}", + } + + +def _where(space): + """`space` and `audience` for a sensor's activities: its namespace, else admins only.""" + return ( + {"space": space, "audience": None} + if isinstance(space, int) and not isinstance(space, bool) + else {"space": None, "audience": "admins"} + ) + + +def _number(v): + return f"{v:g}" if isinstance(v, (int, float)) and not isinstance(v, bool) else None + + +READING_FIELDS = "id, record::id(id) AS n, sensor, stream, at, value, text, fields, level, pattern" + + +def _readings(db, rows, known=None): + """sensor_reading rows as sensor.reading activities (a row whose sensor is gone, or whose time isn't one, is + skipped).""" + rows = [r for r in rows if r and isinstance(r.get("n"), str)] + if not rows: + return [] + known = known if known is not None else _sensors(db, {r.get("sensor") for r in rows}) + streams = events_content._lookup(db, "sensor_stream", {r.get("stream") for r in rows if r.get("stream")}, "name") + patterns = events_content._lookup(db, "sensor_pattern", {r.get("pattern") for r in rows if r.get("pattern")}, "template, label") + out = [] + for r in rows: + sid = r.get("sensor") + if sid not in known: + continue + try: + published = events.stamp(r["at"]) + except (TypeError, ValueError): + continue + sensor = known[sid] + stream = str((streams.get(r.get("stream")) or {}).get("name") or "default")[:255] + text = " ".join(str(r.get("text") or "").split())[:TEXT_MOST] or None + value = r.get("value") if isinstance(r.get("value"), (int, float)) and not isinstance(r.get("value"), bool) else None + pat = patterns.get(r.get("pattern")) or {} + said = _number(value) if value is not None and not text else text or _number(value) or "a reading" + obj = store.clean( + { + "type": "Note", + "id": f"lens:sensor_reading/{r['n']}", + "name": f"{stream}: {said}"[:300], + "url": f"/sensors/{sid}", + "stream": stream, + "value": value, + "text": text, + "fields": r.get("fields") if isinstance(r.get("fields"), dict) and r["fields"] else None, + "level": r.get("level") if isinstance(r.get("level"), int) else None, + "pattern": pat.get("template"), + "label": pat.get("label"), + } + ) + actor = _actor(sid, sensor) + out.append( + events.build( + f"reading-{r['n']}", + "sensor.reading", + actor=actor, + obj=obj, + summary=f"{actor['name']} {stream}: {said}", + source="sensor_readings", + pos=[r["at"], r["n"]], + published=published, + **_where(sensor.get("space")), + ) + ) + return out + + +def _key(a): + return (a["lens"]["pos"][0], a["lens"]["pos"][1]) + + +class ReadingsSource(events.Source): + """sensor_reading rows as sensor.reading, read one sensor at a time over the (sensor, at) index: after a position + by `at`, then those at its time with a later id, so the index's range and order do the work.""" + + name = "sensor_readings" + + def now(self, db): + return [sensors._ts(), None] + + def back(self, db, seconds): + return [sensors._ts(dt.datetime.now(dt.timezone.utc) - dt.timedelta(seconds=seconds)), None] + + def _one(self, db, sid, pos, op, limit, since=None, until=None): + """One sensor's readings past `pos` ([at, id]; op ">" oldest first, "<" newest first), at most `limit`.""" + order = "at, id" if op == ">" else "at DESC, id DESC" + bounds, p = "", {"s": sid} + if since: + bounds += " AND at > $since" + p["since"] = _stamp(since) + if until: + bounds += " AND at <= $until" + p["until"] = _stamp(until) + rows = [] + if pos is not None and pos[1] is not None: + rows += db.rows( + f"SELECT {READING_FIELDS} FROM sensor_reading WHERE sensor = $s AND at = $pt AND id {op} $pi{bounds} " + f"ORDER BY {order} LIMIT {int(limit)}", + pt=pos[0], + pi=R("sensor_reading", pos[1]), + **p, + ) + if pos is None: + where = "true" + else: + where = f"at {op} $pt" + p["pt"] = pos[0] + rows += db.rows( + f"SELECT {READING_FIELDS} FROM sensor_reading WHERE sensor = $s AND {where}{bounds} ORDER BY {order} LIMIT {int(limit)}", **p + ) + return rows + + def read(self, db, cfg, pos, spaces, limit, hint=None): + known = {sid: s for sid, s in _sensors(db, _hinted(hint)).items() if spaces is None or s.get("space") in spaces} + rows = [r for sid in sorted(known) for r in self._one(db, sid, pos, ">", limit)] + rows.sort(key=lambda r: (r["at"], r["n"])) + return _readings(db, rows[: int(limit)], known) + + def page(self, db, cfg, before, who, flt, limit): + ids = _hinted({"actors": flt["actor"]}) if flt.get("actor") else None + if flt.get("actor") and not ids: + return [] # the actors asked for aren't sensors + known = _sensors(db, ids) + if not who.get("admin"): + readable = who.get("spaces") or set() + known = {sid: s for sid, s in known.items() if s.get("space") in readable} + if flt.get("spaces") is not None: + known = {sid: s for sid, s in known.items() if s.get("space") in flt["spaces"]} + rows = [r for sid in sorted(known) for r in self._one(db, sid, before, "<", limit, flt.get("since"), flt.get("until"))] + rows.sort(key=lambda r: (r["at"], r["n"]), reverse=True) + return _readings(db, rows[: int(limit)], known) + + def get(self, db, cfg, activity_id): + got = self.get_many(db, cfg, [activity_id]) + return got[0] if got else None + + def get_many(self, db, cfg, ids): + ns = [m.group(1) for m in (READING_RX.match(i) for i in ids) if m] + if not ns: + return [] + return _readings( + db, db.rows(f"SELECT {READING_FIELDS} FROM sensor_reading WHERE id IN $rs", rs=[R("sensor_reading", n) for n in ns]) + ) + + +events.register_source(ReadingsSource()) + +JOINED_FIELDS = "id AS rid, record::id(id) AS id, name, type, space, device, status, created_at" + + +def _joined(db, rows): + out = [] + for r in rows: + if not r or r.get("type") not in sensors.STREAM_TYPES: + continue + try: + published = events.stamp(r["created_at"]) + except (TypeError, ValueError, KeyError): + continue + actor = _actor(r["id"], r) + label = sensors.STREAM_TYPES[r["type"]]["label"] + label = label if label[1:2].isupper() else label[:1].lower() + label[1:] # "MQTT device", "syslog sender" + obj = store.clean( + { + "type": "Service", + "id": f"lens:sensor/{r['id']}", + "name": actor["name"], + "url": f"/sensors/{r['id']}", + "sensor_type": r["type"], + "device": " ".join(str(r.get("device") or "").split())[:200] or None, + "status": r.get("status"), + } + ) + out.append( + events.build( + f"joined-sensor-{r['id']}", + "sensor.joined", + actor=actor, + obj=obj, + summary=f"A new {label} reported: {actor['name']}", + source="sensors", + pos=[r["created_at"], r["id"]], + published=published, + **_where(r.get("space")), + ) + ) + return out + + +class JoinedSource(events.Source): + """storage_source rows discovery made (a device or host reporting for the first time) as sensor.joined, read by + (created_at, id).""" + + name = "sensors" + + def read(self, db, cfg, pos, spaces, limit, hint=None): + p: dict = {"t": sorted(sensors.STREAM_TYPES)} + where = f"created_by = 'discovered' AND type IN $t AND {events.after_sql('created_at', pos, p, table='storage_source')}" + rows = db.rows( + f"SELECT {JOINED_FIELDS} FROM storage_source WHERE {where} AND {events.space_sql(spaces, p)} ORDER BY created_at, rid LIMIT {int(limit)}", + **p, + ) + return _joined(db, rows) + + def page(self, db, cfg, before, who, flt, limit): + p: dict = {"t": sorted(sensors.STREAM_TYPES)} + where = ["created_by = 'discovered'", "type IN $t", "created_at != NONE"] + if not who.get("admin"): + where.append(events.space_sql(who.get("spaces") or set(), p)) + if before is not None: + where.append(events.after_sql("created_at", before, p, "<", table="storage_source")) + if flt.get("spaces") is not None: + p["fs"] = sorted(flt["spaces"]) + where.append("space IN $fs") + # bounds that can't be an index range: newest first over one hits SurrealDB's range-order bug (docs/automations.md) + if flt.get("since"): + p["since"] = events_content._sec(flt["since"]) + where.append("(created_at > $since OR created_at = NONE)") + if flt.get("until"): + p["until"] = events_content._sec(flt["until"]) + where.append("(created_at <= $until OR created_at = NONE)") + rows = db.rows( + f"SELECT {JOINED_FIELDS} FROM storage_source WHERE {' AND '.join(where)} ORDER BY created_at DESC, rid DESC LIMIT {int(limit)}", + **p, + ) + return _joined(db, rows) + + def get(self, db, cfg, activity_id): + m = JOINED_RX.match(activity_id) + if not m: + return None + r = db.one(f"SELECT {JOINED_FIELDS} FROM $r WHERE created_by = 'discovered'", r=R("storage_source", int(m.group(1)))) + got = _joined(db, [r]) if r else [] + return got[0] if got else None + + +events.register_source(JoinedSource()) + + +def alert(db, pid, pt, sensor, count): + """A sensor.alert for a pattern labelled alert or notable that new lines fit (`pt`: the intake's tally for this + pattern since its last write, with `label`); once per pattern per hour, whichever process sees it first. Its id.""" + hour = str(pt["first"])[:13] + actor = _actor(pt["sensor"], sensor) + template = str(pt.get("template") or "")[:300] + label = pt.get("label") or "alert" + obj = store.clean( + { + "type": "Note", + "id": f"lens:sensor_pattern/{pid}", + "name": template or "a log line", + "url": f"/sensors/{pt['sensor']}", + "label": label, + "pattern": template, + "example": " ".join(str(pt.get("example") or "").split())[:500] or None, + "count": count if isinstance(count, int) else None, + "lines": pt.get("n"), + "hour": hour, + } + ) + where = _where((sensor or {}).get("space")) + return events.emit( + db, + "sensor.alert", + actor=actor, + obj=obj, + space=where["space"], + audience=where["audience"], + summary=f"{actor['name']} logged {'an alert' if label == 'alert' else 'something notable'}: {template or 'a log line'}", + published=pt["first"], + key=f"sensor.alert:{pid}:{hour}", + ) + + +_SENSOR_OBJECT = { + "stream": _s("Stream"), + "value": _s("Value", "number"), + "text": _s("Text"), + "fields": {"type": "object", "title": "Fields", "additionalProperties": {"type": "number"}}, + "level": _s("Syslog level (0 emergency to 7 debug)", "integer"), + "pattern": _s("Log pattern"), + "label": _s("Pattern's label", enum=["routine", "notable", "alert"]), +} + + +def _sample(aid, kind, as_type, actor, obj, summary, source, pos, untrusted, space=None): + return { + "id": aid, + "type": as_type, + "kind": kind, + "published": "2026-10-15T16:21:58.000Z", + "actor": actor, + "object": obj, + "target": None, + "tag": [], + "summary": summary, + "space": space, + "lens": {"source": source, "pos": pos, "untrusted": list(untrusted), "cause": None, **({} if space else {"audience": "admins"})}, + } + + +KITCHEN = {"type": "Service", "id": "lens:sensor/5", "name": "Kitchen sensor"} +ROUTER = {"type": "Service", "id": "lens:sensor/8", "name": "192.168.1.1"} +READING_UNTRUSTED = ("actor.name", "object.name", "object.stream", "object.text", "object.fields", "object.pattern") +events.kind( + "sensor.reading", + label="Sensor reading", + source="sensor_readings", + as_type="Update", + object_type="Note", + noisy=True, + fields={ + "object": { + "type": "object", + "description": "Only the readings a sensor keeps are read: set its handling to store changes (or all) to " + "trigger on every change; a sensor keeping only a summary has none.", + "properties": _SENSOR_OBJECT, + } + }, + untrusted=READING_UNTRUSTED, + sample=_sample( + "reading-k3v9q2m7x1c8z5b4n6d0", + "sensor.reading", + "Update", + KITCHEN, + { + "type": "Note", + "id": "lens:sensor_reading/k3v9q2m7x1c8z5b4n6d0", + "name": "temperature: 31.2", + "url": "/sensors/5", + "stream": "temperature", + "value": 31.2, + }, + "Kitchen sensor temperature: 31.2", + "sensor_readings", + ["2026-10-15T16:21:58.120391+00:00", "k3v9q2m7x1c8z5b4n6d0"], + READING_UNTRUSTED, + space=3, + ), +) +ALERT_UNTRUSTED = ("actor.name", "object.name", "object.pattern", "object.example") +events.kind( + "sensor.alert", + label="Sensor alert", + source="log", + as_type="Announce", + object_type="Note", + fields={ + "object": { + "type": "object", + "properties": { + "label": _s("Label", enum=["alert", "notable"]), + "pattern": _s("Log pattern"), + "example": _s("A line it fits"), + "count": _s("Lines seen in all", "integer"), + "lines": _s("Lines when it was raised", "integer"), + "hour": _s("Hour (UTC)"), + }, + } + }, + untrusted=ALERT_UNTRUSTED, + sample=_sample( + "log-40", + "sensor.alert", + "Announce", + ROUTER, + { + "type": "Note", + "id": "lens:sensor_pattern/8-3f2a9c1e0b7d4a6c-1a2b3c4d5e6f", + "name": "wlan: authentication failed for ", + "url": "/sensors/8", + "label": "alert", + "pattern": "wlan: authentication failed for ", + "example": "wlan0: authentication failed for 3c:22:fb:10:aa:01", + "count": 412, + "lines": 3, + "hour": "2026-10-15T16", + }, + "192.168.1.1 logged an alert: wlan: authentication failed for ", + "log", + ["2026-10-15T16:21:58+00:00", 40], + ALERT_UNTRUSTED, + ), +) +JOINED_UNTRUSTED = ("actor.name", "object.name", "object.device") +events.kind( + "sensor.joined", + label="New sensor", + source="sensors", + as_type="Join", + object_type="Service", + fields={ + "object": { + "type": "object", + "properties": { + "sensor_type": _s("Kind of sensor", enum=sorted(sensors.STREAM_TYPES)), + "device": _s("Device"), + "status": _s("Status", enum=list(sensors.STATUSES)), + }, + } + }, + untrusted=JOINED_UNTRUSTED, + sample=_sample( + "joined-sensor-5", + "sensor.joined", + "Join", + KITCHEN, + { + "type": "Service", + "id": "lens:sensor/5", + "name": "Kitchen sensor", + "url": "/sensors/5", + "sensor_type": "mqtt", + "device": "zigbee2mqtt/kitchen", + "status": "new", + }, + "A new MQTT device reported: Kitchen sensor", + "sensors", + ["2026-10-15T16:21:58+00:00", 5], + JOINED_UNTRUSTED, + ), +) diff --git a/fastapi_backend/app/domain/sensor_patterns.py b/fastapi_backend/app/domain/sensor_patterns.py index c76ef3f78..2ff819554 100644 --- a/fastapi_backend/app/domain/sensor_patterns.py +++ b/fastapi_backend/app/domain/sensor_patterns.py @@ -62,8 +62,8 @@ def update(db, pid, label=None, action=None, user=None, clear_label=False): db.q("UPDATE $r MERGE $d", r=R("sensor_pattern", pid), d=store.clean(sets)) if clear_label: db.q("UPDATE $r SET label = NONE, label_by = NONE, sure = NONE, confidence = NONE", r=R("sensor_pattern", pid)) - if action is not None: - sensors._bump(db) # the intake drops (or keeps) its lines from now on + if action is not None or label is not None or clear_label: + sensors._bump(db) # the intake drops (or keeps) its lines, and raises sensor.alert for them, from now on def triage(db, cfg, ids=None, limit=TRIAGE_PER_RUN, say=None): @@ -104,5 +104,6 @@ def triage(db, cfg, ids=None, limit=TRIAGE_PER_RUN, say=None): ) done += 1 if done: + sensors._bump(db) # alert and notable patterns raise sensor.alert from now on say(f"labelled {done} log patterns") return done diff --git a/fastapi_backend/app/domain/sensors.py b/fastapi_backend/app/domain/sensors.py index 4a51d53fc..6920d84e7 100644 --- a/fastapi_backend/app/domain/sensors.py +++ b/fastapi_backend/app/domain/sensors.py @@ -61,6 +61,7 @@ }, } STATUSES = ("new", "active", "paused", "ignored") +ALERT_LABELS = ("alert", "notable") # a log pattern labelled so raises sensor.alert (events_streams.py) STORE_MODES = ("all", "changes", "summary", "none") HANDLING = ("store", "raw_days", "rollup_days", "important_days", "max_per_minute", "triage", "digest") KINDS = ("number", "boolean", "json", "text", "log") @@ -672,6 +673,9 @@ def __init__(self, db, cfg_fn, log=None): self.rollups: dict = {} self.patterns: dict = {} self.dropping: set = set() # patterns whose lines are counted but not kept + self.alerting: dict = {} # patterns labelled alert or notable: {pattern: label} + self.alerts: dict = {} # alerting patterns seen since the last write: {pattern: (label, the sensor's name and space)} + self.alerted: set = set() # (pattern, hour) this process raised sensor.alert for self.known: dict = {} # stream -> the patterns it has self.stats: dict = {} self.seen: dict = {} @@ -703,6 +707,11 @@ def _fresh(self): rows = self.db.rows("SELECT key, params FROM storage_source WHERE type = 'mqtt'") self.prefixes = sorted(((r["params"] or {}).get("prefix") or "" for r in rows), key=len, reverse=True) self.dropping = set(self.db.values("SELECT VALUE record::id(id) FROM sensor_pattern WHERE action = 'drop'")) + self.alerting = { + pid: label + for label in ALERT_LABELS + for pid in self.db.values("SELECT VALUE record::id(id) FROM sensor_pattern WHERE label = $l AND sure != false", l=label) + } self.known.clear() # retention may have removed some def sensor(self, typ, key, name, device, params, space=None): @@ -779,6 +788,8 @@ def put(self, sensor, stream, payload, at=None, level=None, log=False): ) pt["n"] += 1 pt["last"] = when + if pid in self.alerting and (sensor.get("status") or "active") not in ("paused", "ignored"): + self.alerts[pid] = (self.alerting[pid], {"name": sensor.get("name"), "space": sensor.get("space")}) st = self.stats.setdefault( k, {"sensor": sid, "name": stream, "count": 0, "stored": 0, "dropped": 0, "fields": set(), "first": when} ) @@ -844,6 +855,7 @@ def flush(self): rows, self.rows = self.rows, [] rollups, self.rollups = self.rollups, {} patterns, self.patterns = self.patterns, {} + alerts, self.alerts = self.alerts, {} stats, self.stats = self.stats, {} seen, self.seen = self.seen, {} db = self.db @@ -883,8 +895,30 @@ def flush(self): db.q("UPDATE $r SET last_seen_at = $t", r=R("storage_source", sid), t=when) except Exception as e: # noqa: BLE001 - lose this batch rather than stop taking readings self.log(f"sensors: couldn't save {len(rows)} readings: {type(e).__name__}: {e}") + if alerts: + self._alert(patterns, alerts) return len(rows) + def _alert(self, patterns, alerts): + """sensor.alert for the alert and notable patterns new lines fit: once per pattern per hour (the event log's + key makes it once across processes too).""" + from . import events_streams # events_streams reads sensors + + for pid, (label, sensor) in alerts.items(): + pt = patterns.get(pid) + hour = str((pt or {}).get("first") or "")[:13] + if not pt or (pid, hour) in self.alerted: + continue + try: + count = self.db.values("SELECT VALUE count FROM $r", r=R("sensor_pattern", pid)) + events_streams.alert(self.db, pid, {**pt, "label": label}, sensor, count[0] if count else None) + except Exception as e: # noqa: BLE001 - an alert that couldn't be logged never stops the readings + self.log(f"sensors: couldn't log an alert for {pid}: {type(e).__name__}: {e}") + continue + if len(self.alerted) > 10000: + self.alerted.clear() + self.alerted.add((pid, hour)) + _INTAKES: dict = {} _IL = threading.Lock() diff --git a/fastapi_backend/app/domain/store.py b/fastapi_backend/app/domain/store.py index 7bde98726..05e3858f3 100644 --- a/fastapi_backend/app/domain/store.py +++ b/fastapi_backend/app/domain/store.py @@ -1293,6 +1293,8 @@ def close(self): # hourly rollups (sensor_rollup:--), hub logins and the processes running the hub "DEFINE INDEX IF NOT EXISTS storage_source_key ON storage_source FIELDS key", "DEFINE INDEX IF NOT EXISTS storage_source_push ON storage_source FIELDS push_hash", + # sensors discovery made, newest first, as sensor.joined (events_streams.py) + "DEFINE INDEX IF NOT EXISTS storage_source_created ON storage_source FIELDS created_at", "DEFINE TABLE IF NOT EXISTS sensor_stream SCHEMALESS", "DEFINE INDEX IF NOT EXISTS sensor_stream_sensor ON sensor_stream FIELDS sensor", "DEFINE TABLE IF NOT EXISTS sensor_reading SCHEMALESS", @@ -1307,6 +1309,7 @@ def close(self): "DEFINE INDEX IF NOT EXISTS sensor_pattern_sensor ON sensor_pattern FIELDS sensor, stream", "DEFINE INDEX IF NOT EXISTS sensor_pattern_stream ON sensor_pattern FIELDS stream", "DEFINE INDEX IF NOT EXISTS sensor_pattern_action ON sensor_pattern FIELDS action", + "DEFINE INDEX IF NOT EXISTS sensor_pattern_label ON sensor_pattern FIELDS label", "DEFINE INDEX IF NOT EXISTS sensor_reading_pattern ON sensor_reading FIELDS pattern", "DEFINE TABLE IF NOT EXISTS sensor_login SCHEMALESS", "DEFINE INDEX IF NOT EXISTS sensor_login_name ON sensor_login FIELDS username UNIQUE", diff --git a/fastapi_backend/tests/domain/test_events_streams.py b/fastapi_backend/tests/domain/test_events_streams.py new file mode 100644 index 000000000..c5d27119f --- /dev/null +++ b/fastapi_backend/tests/domain/test_events_streams.py @@ -0,0 +1,193 @@ +"""Sensors as activities (docs/automations.md, Kinds): the readings a sensor keeps as `sensor.reading`, read only for +the sensors a follower names; an alert or notable log pattern as `sensor.alert`, once per pattern per hour; a device +that reported for the first time as `sensor.joined`.""" + +from __future__ import annotations + +import time + +import pytest + +from app.domain import events, sensor_patterns, sensors, store + +R = store.R +ADMIN = events.viewer(1, None, admin=True) + + +@pytest.fixture(autouse=True) +def fresh(): + events._seen.clear() + events._claimed.clear() + yield + + +def _cfg_fn(cfg): + return lambda: cfg + + +def _next_second(): + t = store.now() + while store.now() == t: + time.sleep(0.05) + + +def _alerts(db): + return db.rows("SELECT record::id(id) AS id, activity FROM event_log WHERE kind = 'sensor.alert' ORDER BY id") + + +def test_readings_a_sensor_keeps_are_sensor_reading_activities(db, cfg): + pods = store.ns_id(db, "pods") + it = sensors.Intake(db, _cfg_fn(cfg)) + kitchen, _ = sensors.create(db, cfg, "mqtt", "Kitchen sensor", {"prefix": "kitchen"}, space=pods) + plug, _ = sensors.create(db, cfg, "mqtt", "Plug", {"prefix": "plug"}, handling_={"store": "changes"}) + quiet, _ = sensors.create(db, cfg, "mqtt", "Meter", {"prefix": "meter"}, handling_={"store": "summary"}) + assert events.follow(db, cfg, "test.readings", kinds=["sensor.reading"])[0] == [] + assert events.follow(db, cfg, "test.kitchen", kinds=["sensor.reading"], hint={"actors": [f"lens:sensor/{kitchen}"]})[0] == [] + + k, p, m = (sensors.get(db, s) for s in (kitchen, plug, quiet)) + it.put(k, "temperature", "21.5") + for v in ("1", "1", "2"): + it.put(p, "power", v) + it.put(k, "air", '{"temperature": 22.1, "humidity": 40}') + it.put(m, "watts", "300") + it.put(k, "door", "Front door opened") + it.flush() + + got, commit = events.follow(db, cfg, "test.readings", kinds=["sensor.reading"]) + commit() + for a in got: + events.check_activity(a) + assert [(a["actor"]["name"], a["object"]["name"]) for a in got] == [ + ("Kitchen sensor", "temperature: 21.5"), + ("Plug", "power: 1"), + ("Plug", "power: 2"), # changes only + ("Kitchen sensor", 'air: {"temperature": 22.1, "humidity": 40}'), + ("Kitchen sensor", "door: Front door opened"), + ] # the meter keeps a summary, so no readings + first = got[0] + assert first["kind"] == "sensor.reading" and first["type"] == "Update" and first["space"] == pods + assert first["actor"] == {"type": "Service", "id": f"lens:sensor/{kitchen}", "name": "Kitchen sensor"} + assert ( + first["object"]["stream"] == "temperature" and first["object"]["value"] == 21.5 and first["object"]["url"] == f"/sensors/{kitchen}" + ) + assert first["summary"] == "Kitchen sensor temperature: 21.5" and first["id"].startswith("reading-") + assert "object.text" in first["lens"]["untrusted"] and "audience" not in first["lens"] + assert got[1]["space"] is None and got[1]["lens"]["audience"] == "admins" # no namespace: admins only + assert got[3]["object"]["fields"] == {"temperature": 22.1, "humidity": 40.0} + assert events.follow(db, cfg, "test.readings", kinds=["sensor.reading"])[0] == [] # read once + + # a follower naming its sensors reads only theirs + mine, _ = events.follow(db, cfg, "test.kitchen", kinds=["sensor.reading"], hint={"actors": [f"lens:sensor/{kitchen}"]}) + assert [a["object"]["stream"] for a in mine] == ["temperature", "air", "door"] + + # left out of the timeline unless asked for; then newest first, for those who may see them + assert not [a for a in events.page(db, cfg, ADMIN)["items"] if a["kind"] == "sensor.reading"] + page = events.page(db, cfg, ADMIN, flt={"kinds": ["sensor.reading"]})["items"] + assert [a["id"] for a in page] == [a["id"] for a in reversed(got)] + reader = events.viewer(2, [pods]) + assert [a["id"] for a in events.page(db, cfg, reader, flt={"kinds": ["sensor.reading"]})["items"]] == [ + got[4]["id"], + got[3]["id"], + got[0]["id"], + ] + only = events.page(db, cfg, ADMIN, flt={"kinds": ["sensor.reading"], "actor": f"lens:sensor/{plug}"})["items"] + assert [a["object"]["name"] for a in only] == ["power: 2", "power: 1"] + assert events.page(db, cfg, ADMIN, flt={"kinds": ["sensor.reading"], "actor": "lens:user/1"})["items"] == [] + two = events.page(db, cfg, ADMIN, flt={"kinds": ["sensor.reading"]}, limit=2) + rest = events.page(db, cfg, ADMIN, before=two["next"], flt={"kinds": ["sensor.reading"]}, limit=10) + assert [a["id"] for a in two["items"] + rest["items"]] == [a["id"] for a in page] + + assert events.get(db, cfg, got[0]["id"], ADMIN)["object"] == first["object"] + assert events.get(db, cfg, got[1]["id"], reader) is None + assert events.get(db, cfg, "reading-nothere", ADMIN) is None + + +def test_an_alert_pattern_is_a_sensor_alert_once_an_hour(db, cfg): + it = sensors.Intake(db, _cfg_fn(cfg)) + sid, _ = sensors.create(db, cfg, "syslog", None, {"address": "192.168.1.1"}) + s = sensors.get(db, sid) + line = "wlan0: authentication failed for 3c:22:fb:10:aa:{:02x}" + it.put(s, "hostapd", line.format(1), at="2026-10-15T16:01:00.000000+00:00", level=4, log=True) + it.flush() + assert not _alerts(db) # not labelled yet + pid = sensor_patterns.list_patterns(db, sid)[0]["id"] + sensor_patterns.update(db, pid, label="alert") + it.checked = 0 # the intake looks for changes at most once a second + + for i in range(3): + it.put(s, "hostapd", line.format(i + 2), at=f"2026-10-15T16:2{i}:00.000000+00:00", level=4, log=True) + it.flush() + it.put(s, "hostapd", line.format(9), at="2026-10-15T16:40:00.000000+00:00", level=4, log=True) + it.flush() + other = sensors.Intake(db, _cfg_fn(cfg)) # another process sees the same hour + other.put(s, "hostapd", line.format(10), at="2026-10-15T16:50:00.000000+00:00", level=4, log=True) + other.flush() + [row] = _alerts(db) + a = events.get(db, cfg, f"log-{row['id']}", ADMIN) + events.check_activity(a) + assert a["kind"] == "sensor.alert" and a["type"] == "Announce" and a["space"] is None and a["lens"]["audience"] == "admins" + assert a["actor"] == {"type": "Service", "id": f"lens:sensor/{sid}", "name": "192.168.1.1"} + assert a["object"]["pattern"] == "wlan: authentication failed for " and a["object"]["label"] == "alert" + assert a["object"]["lines"] == 3 and a["object"]["count"] == 4 and a["object"]["hour"] == "2026-10-15T16" + assert a["object"]["example"] == line.format(2) and a["published"].startswith("2026-10-15T16:20:00") + assert a["summary"] == "192.168.1.1 logged an alert: wlan: authentication failed for " + + it.put(s, "hostapd", line.format(11), at="2026-10-15T17:05:00.000000+00:00", level=4, log=True) + it.flush() + assert len(_alerts(db)) == 2 # the next hour + + # notable says so; a paused sensor, or a label the model wasn't sure of, raises none + sensor_patterns.update(db, pid, label="notable") + it.checked = 0 + it.put(s, "hostapd", line.format(12), at="2026-10-15T18:00:00.000000+00:00", level=4, log=True) + it.flush() + assert events.get(db, cfg, f"log-{_alerts(db)[-1]['id']}", ADMIN)["summary"].startswith("192.168.1.1 logged something notable:") + sensors.update(db, sid, status="paused") + it.checked = 0 + it.put(sensors.get(db, sid), "hostapd", line.format(13), at="2026-10-15T19:00:00.000000+00:00", level=4, log=True) + it.flush() + sensors.update(db, sid, status="active") + db.q("UPDATE $r SET label = 'alert', sure = false", r=R("sensor_pattern", pid)) + sensors._bump(db) + it.checked = 0 + it.put(sensors.get(db, sid), "hostapd", line.format(14), at="2026-10-15T20:00:00.000000+00:00", level=4, log=True) + it.flush() + assert len(_alerts(db)) == 3 + + +def test_a_device_reporting_first_is_sensor_joined(db, cfg): + pods = store.ns_id(db, "pods") + it = sensors.Intake(db, _cfg_fn(cfg)) + assert events.follow(db, cfg, "test.joined", kinds=["sensor.joined"])[0] == [] + _next_second() + + s = it.mqtt_sensor("zigbee2mqtt/kitchen/temperature") + sensors.create(db, cfg, "mqtt", "By hand", {"prefix": "byhand"}) # added by a person: not news + it.sensor("syslog", "syslog:10.0.0.9", "10.0.0.9", "10.0.0.9", {"address": "10.0.0.9"}, space=pods) + got, commit = events.follow(db, cfg, "test.joined", kinds=["sensor.joined"]) + commit() + for a in got: + events.check_activity(a) + assert [a["summary"] for a in got] == [ + f"A new MQTT device reported: {s['name']}", + "A new syslog sender reported: 10.0.0.9", + ] + a = got[0] + assert a["id"] == f"joined-sensor-{s['id']}" and a["type"] == "Join" and a["lens"]["audience"] == "admins" + assert a["object"] == { + "type": "Service", + "id": f"lens:sensor/{s['id']}", + "name": s["name"], + "url": f"/sensors/{s['id']}", + "sensor_type": "mqtt", + "device": s["device"], + "status": "new", + } + assert got[1]["space"] == pods + assert events.follow(db, cfg, "test.joined", kinds=["sensor.joined"])[0] == [] + + page = events.page(db, cfg, ADMIN, flt={"kinds": ["sensor.joined"]})["items"] + assert [x["id"] for x in page] == [got[1]["id"], got[0]["id"]] + assert [x["id"] for x in events.page(db, cfg, events.viewer(2, [pods]), flt={"kinds": ["sensor.joined"]})["items"]] == [got[1]["id"]] + assert events.get(db, cfg, a["id"], ADMIN)["object"]["status"] == "new" + assert events.get(db, cfg, a["id"], events.viewer(2, [pods])) is None diff --git a/threat-model.json b/threat-model.json index 7b08e31f8..aa45d1b96 100644 --- a/threat-model.json +++ b/threat-model.json @@ -502,7 +502,7 @@ "parameter": "payloads, hub logins, token in the path", "attacker_controllable": "yes", "component": "optional", - "caller_must_enforce": "admin enables; operator firewalls", + "caller_must_enforce": "admin enables; operator firewalls; readings, alert patterns and new devices reach automations as outside text (lens.untrusted)", "provenance": "documented", "source": "docs/sensors.md \"Sensors are off until an admin turns them on\"" }, diff --git a/threat-model.md b/threat-model.md index 731845dd0..72b762014 100644 --- a/threat-model.md +++ b/threat-model.md @@ -294,7 +294,7 @@ classified by gate only *(inferred, Q8)*; unread modules are listed in Q34. | Lens extension side panel | answers shown there (model text that can repeat third-party text), the sign-in coming back | yes, indirectly through content | data, x-prompt-injection | none; answers render as Markdown cleaned by DOMPurify (no scripts, images or forms; only http and https links), and the sign-in checks its `state` and PKCE | *(documented, extensions/browser/src/core/assistant.js answerHtml docstring "cleaned of anything that could run or fetch on its own")* | | extensions (prompt, graph and `http` tools, hooks), custom nodes including `template` nodes (Q23) | tool spec, graph JSON, template text | yes, any writer for their own; sharing is gated | callback/code (declarative) | person enables only extensions they trust | *(documented, docs/assistant.md "other people's run only when an admin shared them")* | | Python code tools, shipped and admin-created templates, workflows, pipelines, routines | Python, Jinja source, regexes | admin only | callback/code | admin is trusted | *(maintainer, 2026-10)* | -| MQTT 1883, syslog 5514, `POST /sensors/push/{token}/{stream}` | payloads, hub logins, token in the path | yes, once enabled | data, size/rate, x-credential | admin enables; operator firewalls | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | +| MQTT 1883, syslog 5514, `POST /sensors/push/{token}/{stream}` | payloads, hub logins, token in the path | yes, once enabled | data, size/rate, x-credential, x-prompt-injection | admin enables; operator firewalls; readings, alert patterns and new devices reach automations as outside text (`lens.untrusted`) | *(documented, docs/sensors.md "Sensors are off until an admin turns them on")* | | `POST /automations/hooks/{token}` (an automation's webhook) | body (JSON, a form's text fields or text), query, Standard Webhooks headers, token in the path | yes, anyone given the address | data, size/rate, x-credential, x-prompt-injection | no hook exists until an automation with a webhook trigger is saved; the token is 256 random bits kept only as a hash, and revoked with its trigger; 64 KB and 60 calls a minute per hook; a signature, when sent (or required by a `signed` hook), must match the hook's sealed secret within five minutes; a retried message id is logged once; what was sent is outside text (`lens.untrusted`), wrapped as data in prompts | *(documented, docs/automations.md "No webhook exists until an automation with such a trigger is saved")* | | NFS share (port 12049 on the host, 2049 in the worker) | NFS and MOUNT RPCs, client address | yes, once enabled, from `nfs.allowed_networks` | data, x-network-source | admin enables and picks the account; operator firewalls | *(documented, docs/file-shares.md "NFS has no login")* | | SMB share (port 1445 on the host, 4445 in the worker) | SMB2 requests, NTLM sign-in with a file access key, client address | yes, once enabled, from `smb.allowed_networks` | data, x-credential, x-network-source | admin enables; operator firewalls | *(documented, docs/smb.md "What keeps it safe")* | diff --git a/threat-model.yaml b/threat-model.yaml index 5a09778c2..63f88e7f1 100644 --- a/threat-model.yaml +++ b/threat-model.yaml @@ -2,7 +2,7 @@ # Authority order: prose > yaml > json. Regenerate whenever the prose changes. schema: threat-model-sidecar/v2 project: lens -prose_version: "threat-model.md@sha256:4b1a5dd44394f216d19f1ac4d513d101f44165451df7f87070936ccbac0717dc" +prose_version: "threat-model.md@sha256:bcdebb916f791169f475437f97fea2d166bfca909704eba3c4f48b3b5786c2f1" model_status: under-review triage_policy: strict confidence: {documented: 201, maintainer: 46, inferred: 91, assumption: 8} @@ -433,9 +433,9 @@ entry_points: - name: "payloads, hub logins, token in the path" attacker_controllable: true x-condition: "once enabled" - control_kinds: [data, size, rate, x-credential] + control_kinds: [data, size, rate, x-credential, x-prompt-injection] obligation_id: enable-and-firewall-sensors - caller_must_enforce: "admin enables; operator firewalls" + caller_must_enforce: "admin enables; operator firewalls; readings, alert patterns and new devices reach automations as outside text (lens.untrusted)" provenance: {kind: documented, source: 'docs/sensors.md "Sensors are off until an admin turns them on"'} - id: "NFS share (port 12049 on the host, 2049 in the worker)" component: optional